Measuring and evaluating the impact of development aid is a perennial challenge. Various approaches have been tried. Since many have been aimed at donor needs and objectives, whether or not they adequately reflect improvement in the daily lives of the target population is open to question.
Measuring and evaluating the impact of development aid is a perennial challenge. Various approaches have been tried. Since many have been aimed at donor needs and objectives, whether or not they adequately reflect improvement in the daily lives of the target population is open to question. The UK’s Overseas Development Institute (ODI) recently held a workshop on measuring and valuing wellbeing in health and development, in part to explore how experience in the health sector might help guide the creation of measurements for development efforts.
As the basis for discussion, OHE's Nancy Devlin co-authored a paper with Claire Melamed of the ODI and John Appleby of the King’s Fund, now available on the ODI’s website: 'Valuing development': Could approaches to measuring outcomes in health help make development more accountable? 'The challenge,' according to the authors, 'and the research question, is to establish whether we can devise a way of measuring outcomes, based on poor people's own values, that strikes the right balance between reasonable representation of reality and usability by policy makers'.
In the health sector, it is increasingly possible to express actual or expected outcomes using summary numeric measures based on patients’ experiences and/or public preferences — for example through QALYs and patient reported outcomes measures, such as the EQ-5D. The authors explain how these methods have been devised, tested, tweaked and used as the basis of resource allocation decisions. They note that many of the principles that apply in the appropriate design and use of health measures could be applicable to efforts to devise better measures for the effects of development aid.
Challenges in development measurement are likely to be somewhat different in kind and/or degree than those for health. For example, the objective of health care quite clearly is a long life lived in as good health as possible. Measuring the 'right' thing in the case of development requires understanding what poor people in developing countries value, which will be more than a single outcome. The authors note, however, that 'it is striking that participatory exercises of different types reveal a number of strong similarities in the way that people understand poverty and development in different countries and contexts'. Highly valued are physical health and strength, physical safety and security, economic security, and good community relationships. The relative value given to each of these, of course, will vary across and within countries, across subpopulations, among sectors, and over time. Whether people become used to sustained deprivation and adapt their preferences accordingly also is a matter of some debate. Devising a single measure, then, that can allow comparison across countries, as well as within countries, will be a serious methodological challenge. The kind of systematic, iterative approach used in health, however, can help work through the critical methodological issues.
Assuming that the challenges are met successfully, the authors identify the advantages of using a generic outcome measure for development aid as including the following.
- Consistent, comparable, reliable measurement can offer a method for comparing the impacts of projects across different sectors — or even within the same sector for projects with slightly different objectives. This could help identify what works and what doesn't, much as a randomised clinical trial does in medicine.
- Outcome measures could be used to compare changes in the whole population or in sectors over time, identifying when aid has been most, and least, successful.
- In practical application in decision-making, measurement can help donors and others 'ensure that the views and preferences of poor people feed directly into assessments of value for money and development spending'.
- A standard set of development outcomes and a methodology for exploring the weights attached to them could allow systematic comparisons of priorities across different groups. Understanding the different weights attached to various outcomes by, for example, population subgroups, local communities, donors, and government officials, could help ensure that dialogue about the value of efforts are based on real evidence about what 'value' means for different groups and stakeholders.
Of course, the ultimate goal of learning from the experiences of measurement in health care is creating a useful tool for making choices about development aid. A straightforward metric is important in helping policy makers bridge the gap between the complex realities faced by the poor and the need for simplicity in decision-making.
Download: Melamed, C., Devlin, N. and Appleby, J. (2012) 'Valuing development': Could approaches to measuring outcomes in health help make development more accountable? Research reports and studies. London: Overseas Development Institute.
