MAG's Questionnaire to Assess the Impact of Clearance in Lao PDR
by Jo Durham [ Curtin University of Technology ] and Xaya Xayavong and Boulphan Inthaxay [ MAG ]
In an effort to assess the impact of clearing unexploded ordnance, MAG (Mines Advisory Group) developed a questionnaire to analyze local perspectives on clearance operations and to ensure positive livelihood development.
The continuing shift toward evidence-based practice in mine action means demonstrating post-clearance outcomes and impacts is becoming more important. However, mine-action programs tend to operate in complex and dynamic environments, and choosing appropriate assessment methods is challenging.
MAG is addressing this challenge by piloting a number of initiatives to develop an effective and practical impact-assessment methodology. The overall aim of this process is to enable MAG to better measure and report on post-clearance impacts on livelihoods, as well as improve program delivery. MAG’s definition of impact is “significant or lasting changes in people’s lives brought about by a given action or series of actions.” This definition comes with the consideration that “programs can make an important difference to people’s lives even if that change is not sustained over time.”1 Ultimately, MAG aims to better answer the question “Who benefits from removal of the explosive remnants of conflict, in what ways and in what contexts?”
The purpose of this article is to outline the process of and lessons learned in developing a structured questionnaire to assess the impact of unexploded ordnance clearance on household livelihoods in the Khammoaune-based MAG program. Khammouane province has the fourth highest UXO-contamination level in the Lao People’s Democratic Republic.2 Most of the population are subsistence rice farmers who supplement their income by fishing, small trade, foraging for scrap metal, rearing livestock, and working as non-skilled laborers in various industries. MAG aims to contribute to the development in this area by clearing UXO from land suitable for agriculture and by working closely with development partners to facilitate the implementation of rural development projects.
Evolution of the Questionnaire
The questionnaire was created in two steps. First, a qualitative study of the program recipients’ understanding of impact was conducted. Second, the qualitative data was analyzed and then used to develop and test a questionnaire that would later be administered to a representative population sample. The Sustainable Livelihoods Framework informed the development of the questionnaire.3 This Framework is a way of analyzing the vulnerability of the poor, and it proposes that people require a range of assets to pursue different professions and achieve positive livelihood outcomes. Five capital assets are seen as key to livelihood development:
- Natural resources (land, forests, wildlife, water and air quality)
- Physical (infrastructure, tools and equipment)
- Human (knowledge, skills, health and the ability to work)
- Financial (financial resources such as savings, remittances, access to credit)
- Social (membership in groups, networks and connectedness, relationships of trust, reciprocity and exchanges)
The first step in developing the questionnaire was qualitative data collection using semi-structured interviews with key informants and focus group discussions. A total of 18 focus group interviews (nine male, nine female) and 16 individual interviews (11 male, five female) were conducted. Qualitative data was analyzed by the research team, looking for examples of outcomes and impacts under the categories of the five capital assets of the Sustainable Livelihoods Framework.3 An example of this analysis is illustrated in Table 1.
Constructing the Questionnaire
The survey development team devised questions based on the qualitative analysis provided by the focus group interviews. These questions were then formatted and the language was reviewed with a small group for correct grammar and the appropriate use of language. Questions were written in the local language using familiar terms and phrases. Due to a lack of baseline data, MAG used a five-point scale as a way of assessing household perceptions of change. Examples from the questionnaire are displayed in Table 2.
The questionnaire was then tested with 30 randomly selected households. Repeated testing allowed for an identification of questions with high non-response rates or poor reliability (some of the questions did not receive the same response despite being re-taken by the same people). Based on these statistical tests and feedback from the enumerators administering the questionnaires, a revised questionnaire was drafted and re-checked for appropriateness of language.
The revised questionnaire was pilot-tested and was again tested with 30 randomly selected households, resulting in a pre-final questionnaire. The pre-final questionnaire was tested and retested 10 days later with the same population. Further statistical tests were performed, and based on the results, the five-point scales were reduced to three-point scales (e.g., better than before/no change/worse than before). Further feedback from the test enumerators and a statistician also led to additional changes to the layout and wording of the questionnaire. For example, questions with multiple-choice answers were reworded as illustrated in Table 3.
The process of developing the questionnaire is illustrated in Figure 1.
The final version of the questionnaire begins with a statement outlining the objectives of the questionnaire and includes declarations of confidentiality and consent. The remainder of the questionnaire is divided into eight sections:
- Basic information about the village and clearance site completed prior to the interview
- A composite of household wealth indicators to give an indication of access to assets
- Livelihood strategies
- Pre- and post-UXO clearance land use
- Participation in UXO-clearance process
- Impact of clearance on creation or depletion of assets (as organized under the five assets in the asset pentagon) and on beneficiaries
- Impact of clearance on cash income
- An analysis of the current UXO situation. At the beginning of each section, instructions are provided for enumerators.
There are also five short-response, open-ended questions about significant change, positive/negative change and reasons for change. Impact questions are arranged under the five assets of the Sustainable Livelihood Framework. An aggregate score for each asset and for overall impact can be calculated, giving a comprehensive household impact score. Furthermore, a score for each village, district, asset and statistical associations between factors can also be calculated; for example, the clearance and building of a road and increased access to a market. The questionnaire takes 20–30 minutes to finish and can be completed by the participants themselves or can be administered during a face-to-face or telephone interview.
Our research team learned a number of practical lessons:
- Short, clearly-worded questions in the local language garner the best results. Care should be taken to maintain local terms if the questionnaire is translated because translators may use a more formal linguistic style that is less accessible to local populations.
- Responses requiring a ticked box, rather than open-ended questions, are easier to administer and analyze.
- “Skip questions” should be avoided where possible (i.e., questions that direct the participant to skip over one or more questions.)
- Direct questions about changes in income or material well-being were difficult for respondents conceptually and culturally to answer, with many different terms being used to describe weights, time and distance. The test/retest process also suggested the questions are subject to recall bias, or an inaccurate result due to an inability to remember. This data may be better collected through indirect questions or at the community level.
- Involving local staff and enumerators in every stage of developing the questionnaire increases understanding and improves data quality.
- A “conversational” style when administering questionnaires face to face works best and is also likely to be more culturally appropriate.
The article discusses an overview of the process of developing a structured questionnaire to assess the impact of the removal of UXO on household livelihoods. The aim of the questionnaire was to develop a tool that does not require sophisticated analysis and can be applied efficiently across different contexts by local staff. The questionnaire has been validated and tested for reliability. Steps to take in the future will be to administer and cross-validate the questionnaire by comparing the statistical analysis from a similar questionnaire given in Iraq to the statistical analysis from Lao PDR.
The authors would like to thank all the program recipients who took time to answer our questions. We would also like to thank all the Community Liaison staff who helped in the testing of the questionnaire and the MAG team in Khammouane who provided fantastic support. We would also like to thank Dr. BK Tan for her advice in the early planning stages of this project. Finally, we would like to thank the National Regulatory Authority for the UXO sector in Lao PDR for their support. This project was partly funded by the Department for International Development of the United Kingdom.
Jo Durham, previously MAG’s Country Programme Manager in Lao PDR, is an Associate Lecturer and Ph.D. candidate in International Health at the Centre for International Health, Curtin University of Technology in Perth, Western Australia, and was the lead researcher on this project.
Xaya Xayavong has worked with MAG since 2006, first as Community Liaison Officer and now Project Officer. In his current position, Xayavong is working with the local authorities in Khammouane, Lao PDR, to prioritize tasks and undertake post-clearance assessments. Xaya was the co-researcher on this project.
Boulaphan Inthaxay is MAG Lao’s Community Liaison Coordinator and has extensive experience in coordinating community-based surveys. She helped with analyzing the data, developing the questionnaire and providing support and training to the enumerators for this project.
- Roche, C., Impact Assessment for Development Agencies: Learning to Value Change. 1999, Oxford: OXFAM.
- Handicap International, Living with UXO, Final Report, National Survey on the Socio-Economic Impact of UXO in the Lao PDR. 1997: Vientiane.
- Livelihoods Connect. Sustainable Livelihoods Guidance Sheets. eldis. 2007 (cited 2008 15th July). http://www.eldis.org/go/livelihoods/. Accessed 22 March 2010.
Centre for International Health
Curtin University of Technology
Perth / Australia
E-mail: firstname.lastname@example.org, email@example.com
MAG Lao, Ban PhonSaArt,
Khammouane / Lao PDR
Tel: +856 51 251 051
Mob: +856 20 216 7220
Fax: +856 51 251 051
Web site: www.maginternational.org
Community Liaison Coordinator
PO Box 4660
Ban Sithan Neu / Lao PDR
Tel: + 856 252 004/251 489
Mobile: + 856 20 677 7744
Fax: + 856 217 505