Non-technical Survey: A Model for Evidence-based Assessment

by Aron Larsson and Love Ekenberg [ Stockholm University ] and Åsa Wessel and Håvard Bach [ GICHD ]

In an ongoing effort to improve the Non-technical Survey, the Geneva International Centre for Humanitarian Demining teamed with Stockholm University to create an enhanced version of the Cambodia Mine Action Centre’s Evidence Assessment Model. The aim of the project was to make the existing model more user-friendly and modify the current standards for assessment of mine-affected land. CMAC is testing a revised model to ensure that it meets the needs of their Non-technical Survey teams.

The Geneva International Centre for Humanitarian Demining has developed an Evidence Assessment Model that may form part of a wider Non-technical Survey and enable decisions about when it is appropriate to release land by Non-technical Survey and when/how much Technical Survey is required. The first model was created in collaboration with Norwegian People’s Aid to enhance NPA’s land-release approach in Angola. A second, similar model was developed in support of the Cambodian Mine Action Centre’s land-release approach in Cambodia.

Although these models are in use and working fairly well, the GICHD wanted to test the quality of the model to ensure the validity of its logic procedures and develop an improved interface. The primary objective was to devise a credible, practical and user-friendly model for Non-technical Survey by August 2009. The project was a joint effort between the Department of Computer and Systems Sciences of Stockholm University and the GICHD, and it was partially funded by the Swedish Program for Information and Communication Technology in Developing Regions. GICHD asked the decision-analysis experts of the DECIDE Research Group at Stockholm University to assist with the project.

The project was initiated in March 2009, and the first phase was completed in September 2009,with the delivery of a revised CMAC Evidence Assessment Model to be used in pilot cases in Cambodia.

(Click image to enlarge)
Figure 1: Excerpt from the user sheet. A 'Y' in the cell is an added statement.

The Context and Work Process

The model is designed as a complementary tool in the existing process in which a team of field operators collects and analyzes information about an area suspected to be contaminated by landmines. Traditionally, the decision on whether an area can be released from suspicion of mines without any further mine-action support has been made by the field operator, based on personal experience and conviction. This method has often caused conservative decisions because it has been far easier and less risky for the survey teams to classify land as mine-suspected areas as opposed to “mine free” areas. A credible evidence-assessment model that shifts liability from the operator to the model, or the underlying concept, will encourage more appropriate decisions.

The model described in this article rates the importance, or value, of each individual piece of evidence about the mine threat provided by various informants. The model further contemplates the degree of trust in, or credibility of each source of information. If the credibility of an informant is low, the evidence weight will be reduced and will consequently contribute less to the final survey conclusion.

When organizations use the model, the burden of making the final decision rests less on the experience of the individual field operator and more on the embodied model assessment and recommendation. Using the model further ensures that every step of the survey process is thoroughly analyzed, evaluated and documented. A clear order trail is crucial and will further enable appropriate quality assurance and other follow-up if required.

Prerequisites and model requirements. An evidence-assessment tool needs to fulfill a number of requirements:

The properties of the method should be defined, including limitations and underlying assumptions. The method should therefore conform to an established set of evidence-assessment principles since doing so will enhance the credibility of the model, facilitate communication about it with other mine-action specialists, and facilitate future reviews and modifications.
The rules that regulate the final output should reflect an intuitive behavior to represent the view of experienced survey teams and ensure a high credibility among users.
The method should support means for sensitivity analyses and “what-if” questions.
It should be possible to adapt the model to different conditions, as the significance of different information sources may vary between different regions or countries. For instance, landmine records showing the type of mines and their specific location are sometimes available after a conflict, but the quality of these records may vary considerably between countries. Identical input statements from different areas should not necessarily generate the same recommendations.

The challenge is thus to select a suitable approach for the assessment. T. Denoeux discusses one approach that employs methods from the area of pattern classification in “Analysis of Evidence-Theoretic Decision Rules for Pattern Classification.”¹ It does not currently seem to be applicable in the context of operational mine action, although it would be theoretically appealing to explore it. We argue that a more applicable method should enable the elicitation of knowledge and experience from the local population, the military, land users and mine-action experts, and should allow user-friendly adaptations. A user-friendly model with a user sheet that accommodates individual evidence from informants has thus been developed in Microsoft Excel, which is easily available and will only require basic computer skills.

The proposed model. The proposed Evidence Assessment Model is based on traditional input- output assessment and classification, employing a formalized “recommendation rule” that proposes the next minimum mine-action requirement from a set of predefined recommendations. The model thus uses a well- established and easy-to-understand methodology. GICHD, Stockholm University and CMAC agreed to employ methods from multi-attribute decision theory, since the current approach for Non-technical Survey conformed to the use of numerical weights representing the relative importance of different sources of information. Typical for multi-attribute decision-making is the use of several attribute-specific values being aggregated using additive attribute weights.²

In the CMAC model, the aim has been to associate each sector S_i with a sector value v(S_i) used for the classification of sectors.³ We propose a method for the classification of sectors that are defined in an attribute-value space, where the sector value should be the result of an aggregation of values on evidence attributes. This method would enable simple and useful means for sensitivity analysis and model adaptations while maintaining the look and feel to which CMAC has been accustomed, with some improvements where needed. The simplicity of the model will thus be maintained, but the transparency and usefulness would improve. The model has also been designed to accommodate more advanced future methods for coping with uncertain information and vague assessments and risks.

Input statements. Input statements are entered into the model by field operators or by personnel who receive the reports from the field. Each statement is the result of obtained external information and information from field interviews with the population, police, military or other sources. A statement will either support the conclusion that an area is mined (hazardous) or mine-free (not hazardous), i.e., a statement will either belong to the statement set “pro-mines” M+ (supporting the presence of mines) or the statement set “con-mines” M- (supporting the absence of mines). Each statement corresponds to a certain sector of interest, an evidence attribute and a confidence assessment.

An evidence attribute is associated with a numerical weight, reflecting the strength (or importance) of this evidence, such as to what extent evidence supports the presence of mines in the area or not. The value of the different weights depends on whether the statement belongs to M+ or M-. Mine maps showing the presence of mines could, for example, be regarded more important (evidence of presence of mines) than mine maps showing no mines in certain areas (evidence of absence of mines).

An evidence attribute can be adjusted by the field operator based on the perceived credibility of the information source. One statement does, for example, address “mine maps/records from military or police.” If the investigator has access to maps that indicate an area is mined and the maps are considered accurate and reliable, this evidence attribute is either marked as “high” credibility or “low” credibility—the latter if the map is inaccurate but yet exists. In summary, each statement (s_j):

Belongs to the set M+ (pro-mines) or the set M- (con-mines).
Has an associated numerical weight w_ij assigned to an evidence attribute (i_i) and each pair (i_i, s_j). In the current model, weights are assuming a value within the interval [0, 10], reflecting the importance of this piece of evidence relative to other information, for the classification of a sector.
Is associated with a level of source credibility (c_j). In the current model, source confidence may assume a value within the interval [0, 10]. However, the user is restricted to assign the type of information as “high” or “low” (credibility of informant/information). The values are predefined. “High” is always given a predefined value of 10,⁴ while the value for “low” may vary between 1 and 9 depending on to what degree the reduced credibility of the informant/information is considered to influence the value of the evidence, represented as source confidence. The lower the predefined value is for a “low” credibility statement, the more impact it has on the overall confidence rating from a particular source.
If an evidence attribute has no statement, it is assigned a confidence value of 0.

Sector value and model output. The aggregation of information that provides a sector value is straightforward. Each pair (i_i, s_j) is assigned a weight w_ij [0, 10]. The weights are then subject to normalization so that

or the sum of all w_ij will add up to 1. The aggregated value (V(S)) of a sector (S) is obtained from the difference between the weighted sum of confidence values belonging to M+ and M- respectively. It follows that V(S) is within the interval [-10, 10].

(Click image to enlarge)
Figure 2: Threshholds used in current model.

The output is ultimately a recommendation for the next step required in the land-release process, typically a level of Technical Survey or clearance. In the current model, the recommendation rule is based on a set of thresholds for different intervals, each representing conclusions from the Non-technical Survey Evidence Assessment. The current interval thresholds are shown in Figure 2.

The recommendation may also be dependent on other criteria besides the sector value alone. For instance, a recommendation stemming from a high-confidence conclusion may be dependent on whether the field operator has entered a sufficient number of high-confidence statements in the model. In other words, a recommendation cannot be based upon a high-confidence conclusion if there are too few high-confidence statements entered into the model by the field operator.

(Click image to enlarge)

Figure 3: Example of a CHA with proposed classified sectors.

Conclusion

The model is currently being subjected to pilot testing by CMAC. If the pilot proves successful, the underlying idea of employing formal methods for making more appropriate Non-technical and Technical Survey decisions will be further refined. CMAC is one of several organizations that currently conducts a “Base Line Survey” with the aim of resurveying all areas suspected to be mined in Cambodia for better allocation of demining assets. The model is now used by all 13 CMAC Non-technical Survey teams as an integrated part of the BLS. Pending the results from Cambodia, the concept may well be introduced in other mine-affected countries.

There are, however, some challenges that need to be addressed, including the following issues and questions:

Weights, thresholds and confidence values are predefined. They should be reviewed and possibly revised for each individual case (country).
Which parameters are most important when conducting sensitivity analyses and should the analyses be mandatory?
Are there prominent dependencies between evidence attributes? If so, what is the impact of these dependencies and how may they be assessed by the field operator and addressed in the model?
Are there any uncertainties and inaccuracies in the input statements?

Preliminary results from the pilots have highlighted a need to ensure that:

Each Confirmed Hazard Area is divided into sectors properly. If this division is not done well, the model may not work as intended.
The questioning technique of the survey team is robust enough to gain all the available information. Too little information will just result in a default “normal” Technical Survey.
In addition to seeking evidence to confirm an area is a minefield, the survey team now seeks
counter-evidence to disprove this notion. This is a big mindset change because a survey team will now be inclined to take into account “evidence” supporting the statement that the area is not a minefield, rather than only putting focus on searching for evidence that an area is mined.

Biographies

Aron Larsson is a researcher at Stockholm University and Mid Sweden University. He has a Ph.D. in computer and systems sciences and a Master of Business Administration. Larsson has been working on the development and application of methods and tools for evaluation of decisions under multiple objectives, uncertainties and risk. He has been developing the DecideIT decision tool and has experience applying decision theories in both public and private sectors such as environmental planning and investment decision-making.

Love Ekenberg is a Professor at Stockholm University, the Swedish Royal Institute of Technology and Mid Sweden University. He has a Ph.D. in computer and systems sciences and a Ph.D. in mathematics. Ekenberg has been working with risk and decision analysis for more than 15 years. He has also worked with logic verification of complex industrial systems for several years at Swedish nuclear power plants.

Asa Wessel Åsa Wessel is the Land Release Research Officer at the GICHD. She previously worked as a Quality Assurance Officer at the Mine Action Coordination Centre in South Lebanon with the
United Nations Office for Project Services. Wessel worked as Explosives Ordnance Disposal Coordinator at the Regional Mine Action Coordination Centre in Sudan, employed by the Swedish Rescue Services Agency, and as Instructor at the Swedish Armed Forces EOD and De-mining Centre. She holds a Bachelor of Science in mechanical engineering

Havard Bach Håvard Bach heads the Operational Support Unit for the GICHD. He manages projects that currently include studies on mine-detection dogs, mechanical mine clearance, manual mine clearance and risk management. Bach also worked as a Norwegian Military Engineering Officer before being employed by Norwegian People’s Aid and managing several mine-action programs worldwide.

Endnotes

Denoeux, T. 1997. “Analysis of Evidence-Theoretic Decision Rules for Pattern Classification.” Pattern Recognition 30(7), pp. 1095–1107.
For a comprehensive treatment of this theory, see Chankong, Vera and Haimes, Yacov Y. Multiobjective Decision Making: Theory and Methodology 1983. North-Holland, Amsterdam or Keeney, R.L, and Raiffa, L. Decisions with Multiple Objectives: Preferences and Value Tradeoffs. Wiley. New York. http://www.amazon.com/Decisions-Multiple-Objectives-Preferences-Trade-Offs/dp/0521438837. Accessed 10 March 2010.
Minefields are partitioned into disjoint sub-areas known as sectors.
This may be interpreted as the default value if there is no particular reason to discredit the credibility of an informant a piece of information.

Contact Information

Aron Larsson
Dept. of Information Technology and Media
Mid Sweden University
SE-851 70 Sundsvall / Sweden
Tel: +46 60 148616
E-mail: aron.larsson@miun.se
Web site: http://www.miun.se/personal/aron.larsson

Love Ekenberg
Dept. of Computer and Systems Sciences
Stockholm University
Forum 100
SE-164 40 Kista / Sweden
Tel: +46 8 161679
E-mail: lovek@dsv.su.se
Web site: http://www.dsv.su.se/~lovek/

Åsa Wessel
Land Release Research Officer
Geneva International Center for
Humanitarian Demining
7 bis, avenue de la Paix
P.O. Box 1300
CH-1211 Geneva 1 / Switzerland
Tel: +41 7982 88752
E-mail: a.wessel@gichd.org
Web site: http://www.gichd.org