Post-conflict Impact Assessment in Cambodia
by Ruth Bottomley and Phuong Ponha [ MAG Cambodia ]
This article presents a pilot study that tested the Post-conflict Impact Assessment methodology conducted by MAG Cambodia (Mines Advisory Group Cambodia) in the first quarter of 2009. Funded by the United Kingdom Department for International Development, the pilot was implemented to gauge the suitability and effectiveness of the PCIA methodology to collect impact-assessment data related to MAG’s humanitarian mine-action activities. This article describes the methodology, outlines the process of the pilot study and concludes by discussing the findings that emerged from the pilot, which are currently helping to inform the development of a MAG impact-assessment tool.
MAG (Mines Advisory Group) defines impact assessment as “the systematic analysis of the lasting or significant changes—positive or negative, intended or not—in people’s lives brought about by MAG’s action and/or a series of actions.”1 MAG believes that mine action is not just about removing landmines and explosive remnants of war, but also enabling positive change in the lives of affected communities, contributing to the alleviation of poverty and promoting socioeconomic development. However, effectively measuring the impact of mine-action interventions in post-conflict environments presents a considerable challenge. MAG already documents changes in the lives of selected beneficiaries and projects through case studies, and the majority of MAG programs include simple post-clearance assessments to ensure that cleared land is being used as anticipated and by the intended beneficiaries. However, the challenge still remains to develop a tool that can systematically assess the long-term impact of MAG’s work in local communities and that can be used consistently in a variety of programmatic and operational contexts.
Community liaison staff interview household members in a village in Pailin during the PCIA pilot field test.
All graphics courtesy of Ruth Bottomley/MAG
The PCIA methodology was developed by MAG and draws on two existing models: the Department for International Development Sustainable Livelihoods Framework and the Livelihood Assets Status Tracking System developed by the University of Manchester.2,3 The SLF allows analysis around the five capital livelihood assets (human, natural, physical, financial and social), which together form a household’s asset base, demonstrating that the ability or inability to access these assets impacts the livelihood outcomes of the poor.2 The more access a household has to a range of assets, the greater its asset base, and the less vulnerable it is to shocks and disasters, and vice versa. The access that households have to the capital on which they base their livelihood strategies may in turn be restricted or enhanced by societal structures and laws, and the degree to which these are enforced. In her policy briefing Humanitarian Action in Conflict: Implementing a Political Economy Approach,4 Sarah Collinson has also noted how, in conflict and post-conflict situations, the associated instability and control of resources and assets by political and military factions can increase vulnerabilities.
(Click image to enlarge)
Table 1: An example of a section of the Word Picture matrix for assets. *HH means household.
The Livelihood Asset Status Tracking System, which draws on SLF, was developed to assess whether integrated livelihood improvement projects were having an impact on the livelihoods of beneficiaries. LAST is a monitoring system intended to “track the ongoing dynamics of the five capital assets essential to household livelihoods as a proxy for impact.”3 The methodology comprises rapid, repeat assessments of large numbers of beneficiary households to detect emerging changes in their livelihood platform.
The main tools used for the PCIA pilot were frameworks called Word Pictures, adapted from tools used in the Livelihood Asset Status Tracking System. One Word Picture represents household assets (see Table 1), and another Word Picture represents household risk (see Table 2). The asset Word Picture comprises a matrix with the five capital assets listed vertically on the left-hand axis, and four columns along the horizontal axis, headed poorest to richer, with a numerical scale along the top. Within the columns of the matrix, various household situations are recorded, describing the worst situation to the best situation, using definitions and information that local people believe are relevant to their livelihoods. For example, under “natural assets,” the worst situation could include a lack of land and no access to clean water or forest resources. The best situation would be the reverse of this one, with ownership of productive agricultural land, the use or ownership of pump wells and easy access to well-managed forest resources.
The risk Word Picture describes risk situations under each of the five capital assets, with the four columns of the horizontal axis headed high risk to no risk. For example, the worst scenarios include households that have no understanding of mine risk and enter mined areas to farm, collect forest products, or graze animals, and the best scenarios show households well-versed in risk education and conducting all livelihood activities in safe areas. During discussions with household members, each household is marked on the Word Pictures using a large cross on the descriptions that best reflect the household situation. A score is then given for each capital asset by finding the balance point between the crosses on the Word Picture and marking this point on the numeric scale. In this way, the Word Pictures help to convert a qualitative understanding of a range of locally relevant household situations into a quantifiable number5 and should indicate changes in household assets and risk over time. The Word Pictures can be used at the household level in combination with qualitative tools, such as semi-structured interviews, focus-group discussions and observation. Ideally the PCIA tools would be used sequentially in the same village before an intervention and then at intervals during and after the intervention to gauge how the situation changes for households. However, in situations for which there is no baseline, the Word Pictures have to be used for both the current situation and retrospectively, based on the recall of the households.
Word Pictures can only be developed for areas of reasonable homogeneity in terms of social, cultural, agricultural and economic practice,3 so that the descriptions of household situations are relevant within all the areas where the Word Pictures will be used. Once developed, the Word Pictures have to remain the same for each consecutive assessment, as changes in the descriptions would affect the scores and not accurately reflect the real changes in household capitals or risk taking. For these reasons, the Word Pictures had to be developed carefully for the pilot study, using a two-stage process. The descriptions within each of the Word Pictures were initially developed by the national MAG community liaison teams, based on their knowledge of the target area for the pilot. The draft Word Pictures were then tested over three days in villages in Battambang province to ensure the descriptions were appropriate and reflected the range of possible household situations. The Word Pictures were revised based on the field test and these final versions were used during the pilot.
The Pilot Study
The PCIA pilot project in Cambodia was conducted during the first quarter of 2009.6 The purpose of the pilot study was not to collect impact-assessment data for analysis, but rather to test the feasibility of the proposed methodology for the MAG mine-action impact assessment. Key study questions included:
- How well will the MAG CL teams cope with the PCIA methodology?
- What does the data look like?
- What skills are required to analyze the data?
- How well does the data capture impact and supply the program with information to improve
- How practical is the PCIA methodology for adoption within MAG programs in terms of cost, time required to implement the assessment, quality of information collected, and the ease with which the methodology can be adapted to other contexts and operations?
The field work was conducted in the northwestern provinces of Battambang, Banteay Meanchey and Pailin, all of which are relatively homogenous in terms of key characteristics: proximity to the Thai-Cambodia border; heavy landmine contamination; an ethnic Khmer population comprising both long-term residents (often former Khmer Rouge) and newcomers; and economic activities that rely on labor opportunities both in Cambodia and Thailand, paddy rice cultivation, and, increasingly, cash-crop production. MAG has worked in Battambang since 1992, in Pailin since 1995 and in Banteay Meanchey since 2006.
The PCIA pilot study was conducted specifically around MAG clearance tasks and among those intended to benefit directly from MAG clearance. Twelve villages were included in the pilot and a total of 235 households were interviewed. The villages were selected according to three main criteria: villages awaiting an area to be cleared by MAG; villages where clearance had been completed approximately 12–18 months earlier; and villages where clearance had been completed three to five years earlier. As the time frame for the pilot was short, in all post-clearance sites the pre-clearance data had to be collected retrospectively and then compared with the post-clearance situation. Word Pictures were the main data-collection tool used, in combination with observation of the household and clearance sites, and unstructured interviews with selected households.
(Click image to enlarge)
Figure 1: A bar chart with households categorized into poorest, poor, medium and richer according to the household scores. The chart shows how some households, following MAG clearance, have been able to move from the poor category under some of the assets to the medium and richer categories.
Eighteen MAG Cambodia community liaison staff (Cambodian nationals) worked on the project under the supervision of the CL Manager for Southeast Asia and the Cambodia CL Standards Officer. The CL staff worked in pairs, with each pair responsible for interviewing one household. Both team members independently placed the household on the Word Pictures during the interview and provided a score. The placements and scores were then cross-checked immediately following the interviews to ensure consistency and to solve any anomalies. The teams were encouraged to closely observe the household situation and surroundings and to take photographs of the clearance sites. Following the field work, the household scores were entered into a spreadsheet and some initial analysis was conducted, with the data presented in bar and radar graphs. While the bar graphs (see Figure 1) were able to show the aggregate scores of the target households, the radar graphs (see Figure 2) were able to visually present the asset pentagon for each household and represent how assets or risks within the capital assets had accumulated or declined.
The final stage of the pilot was a three-day workshop to allow the teams to provide feedback on their experiences using the PCIA methodology, discuss the interpretation of the graphs and consider how well the emerging changes could be attributed to MAG activities.
(Click image to enlarge)
Figure 2: A radar graph shows the status of a household regarding levels of mine/ERW risk before and after clearance.
Findings and Discussion
The PCIA methodology has a great deal in common with other quantitative data-collection systems. As a rapid-assessment tool, the Word Pictures allow data collection on a large scale. The conversion of the qualitative judgments into quantitative data allows for the aggregation, analysis and presentation of the data in charts and graphs, which can visually illustrate major changes and trends. The methodology assesses the impact throughout the project cycle rather than as an isolated singular event at a specific end point, thus providing baselines against which any subsequent data can be measured. The analysis of the quantitative data can either provide basic information showing general improvements in overall risk reduction and the accumulation of capital assets, or provide more in-depth learning to improve program design, prioritization and follow-on development interventions. For example, mapping the progress of beneficiary households can examine:
- Whether those beneficiaries who show improvement on the risk or asset scale of the Word Picture matrix tend to start with a particular combination of assets; for example, strong human assets (labor and good health) may correlate with households being better able to reduce risk activities while also using
- Whether there is a link between an improvement in the asset base and a reduction in risk, and vice versa
- Whether some economic groups are able to maximize the benefits of clearance better than others
However, despite the obvious benefits, the pilot also found limitations with the methodology that would need to be considered before implementing it widely. First, the Word Pictures are a difficult tool to develop and use, particularly in a country with limited literacy and in areas where livelihoods are multifaceted. Writing concise, descriptive sentences for household situations from the worst to best case scenario was, in practice, a difficult exercise.
During the pilot it was decided to keep the indicators concise, with the intention that this would maintain the clarity of the Word Pictures. However, this practice resulted in the CL teams feeling uncomfortable about making judgments on where to place a household if the situation did not exactly match the indicator on the Word Picture. The teams also found scoring difficult because they had to approximate the center point between the marks on the Word Picture, and there were inconsistencies in both the household placement and scores that persisted throughout the field test.
In situations for which there was no baseline data, the Word Pictures were used with a retrospective element, which consisted of asking households what the situation was like before clearance. Inevitably, the reliance on recall resulted in less reliable data, and asking for information retrospectively also increased the time needed to conduct the household assessments. Another interesting issue that emerged from the pilot was that as the PCIA methodology focuses at the household level, it does not necessarily account for the beneficiary households that have moved from or into the area, land that is abandoned, or changes in ownership.
The Word Pictures can provide a baseline and indicate change over time, but they preclude the in-depth information and thorough description that can assist with attributing the changes to particular interventions. In areas where there had been mine-action and development support provided by a wide range of actors, the data collected through the Word Pictures struggled to reveal the impact of specific interventions. Word Pictures have to be combined with qualitative data collection and doing so increases both the time and the resources needed to conduct the impact assessment and the final analysis.
The pilot found that the PCIA methodology is relatively resource-intensive in terms of staffing, training, implementation and analysis, and is best applied in situations in which the required inputs are matched by the outputs in terms of the quality of data and the information learned that can improve the program. For MAG, the methodology would be best-suited to assessing the impact of large-scale integrated demining and development projects in which MAG and a partner carry out multiple clearance tasks and development activities in a number of communities. This course of action would assume that the changes brought about by the interventions would be more wide-ranging and would be better captured through the PCIA methodology.
The involvement of development partners in conducting impact assessments can also allow for better attribution of the changes in the household status, as community development workers tend to be in villages on a longer-term basis post-clearance than mine-action or CL teams. Working on impact assessment with development partners also signals a more collaborative approach allowing for joint learning and planning for the improvement of project implementation.
Ruth Bottomley is MAG’s Community Liaison Manager for Southeast Asia. She has worked in the Southeast Asia region since 1994 and in mine action since 2000, with a specific focus on the inclusion of affected communities in mine-action processes.
Phuong Ponha is the Standards Officer for Community Liaison in the MAG
Cambodia program. He has worked for MAG since 1997, initially within the mine-risk-education program, and joining the community liaison staff in 2000.
- MAG, 2008. “What Difference Does MAG Make? Outcome Reference Document.” Internal document. Manchester: MAG
- DFID Sustainable Livelihoods Framework: DFID (2007) Sustainable Livelihoods Guidance Sheets. http://nssd.net/pdf/sectiono.pdf. Accessed 25 March 2010.
- Bond, R. and Mukherjee, N. (2002). Livelihood Asset Tracking Tool: An Impact Monitoring Tool. Journal of International Development, Vol. 14, pp. 805–815.
- Collinson, S. (2003). Humanitarian Action in Conflict: Implementing a Political Economy Approach, Humanitarian Policy Group Briefing, No. 8. http://www.odi.org.uk/resources/download/276.pdf. Accessed 25 March 2010.
- Bond, R., P.H. Kapondamgaga, B. Mwenebanda, R.P.S. Yadav, A. Rizvi, (2007). Monitoring the Livelihood Platform: Reflections on the Operation of the L.A.S.T. Method from India and Malawi. Impact Assessment and Project Appraisal, 25(4), pp. 301–315.
- Cambodia was chosen for the pilot study because, as a country at the development end of the post-conflict continuum with a mature mine-action program, it was predicted that impacts from past mine-action interventions would be more visible and possible to capture with the PCIA methodology. The Royal Government of Cambodia, within the Cambodian Millennium Development Goals, also has a strong commitment to eliminate the negative humanitarian and socioeconomic impacts of landmines and ERW and was supportive of MAG conducting the pilot within Cambodia.
CLM Southeast Asia
P.O. Box 1111
Phnom Penh / Cambodia
Tel: +855 89 327 304
Web site: www.maginternational.org
Standards Officer for Community Liaison