Counting what counts

Donors and partner-governments alike need indicators to measure the benefit of development programmes. In education or health projects, data are readily available to measure and confirm outcomes. Rising literacy rates can be attributed to community education projects. Declining rates of infectious diseases can prove the usefulness of mass vaccination. But effectively monitoring and evaluating investment in agriculture and rural poverty alleviation is chronically handicapped by the lack of an agreed set of performance measures and a scarcity of good data. Now, this may be about to change. Nwanze Okidegbe, Rural Strategy Adviser for Agriculture and Rural Development at the World Bank, outlines a unique Platform initiative to rally donors and partner-countries around a new set of core indicators, speaking with Timothy Nater in Washington.

Nater: Why are indicators important?

Okidegbe: More and more citizens in both developed and developing countries are demanding to see results from development assistance. Developing country governments are increasingly feeling this pressure. They recognise they have to improve their monitoring and evaluation (M&E) systems in order to track progress towards achieving the Millennium Development Goals (MDGs). But they know that their M&E system will only be as good as the quality of the underlying data.

Measuring progress towards achieving the MDGs won’t be effective or yield generally accepted results without a reliable, commonly agreed standard of indicators. So our core indicators initiative couldn’t be more topical and timely. It’s about measuring aid effectiveness on reducing rural poverty, donor harmonisation and alignment, and implementing the Paris Declaration. The Platform offers an ideal forum for donors and partner-countries to work out common policies and practices on this issue.

There’s a saying attributed to Albert Einstein that goes, “Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.” What will you be counting?

The conventional measurements of concrete inputs and outputs are useful, but not sufficient. What counts is the outcome. The focus of M&E is moving more and more from performance-based outputs to ultimate results. For example, when you’re evaluating a programme to improve rural services, you need indicators that measure how satisfied the beneficiaries are with the services they’re receiving. How does a given service or product resulting from investment in this programme affect the lives of men, women and children? This new approach means going to them directly and asking them.

Data drawn from outputs alone have in the past occasionally given a distorted picture. It’s sometimes taken four or five years to realise that a given intervention failed to meet its development objectives. By contrast, this new results-based approach demands speedy results, collecting and acting on data in far less time than before.

What’s the background?

This initiative springs from OECD Development Advisory Committee’s Joint Venture on Managing for Development Results. The Joint Venture operates under the auspices of the OECD/DAC Working Party on Aid Effectiveness and Donor Practices, which stems, in turn, from the Paris Declaration. Among other things, its purpose is to hold donors to their commitments under the Paris Declaration.

Those donor commitments include linking country programming and funding to country results, relying on partner-country systems to do the results-oriented reporting and monitoring, and harmonising donors’ individual monitoring and reporting requirements. And those donor commitments are crucial because partner-countries have long complained about the time they waste in complying with a proliferation of different donor requirements. In return, donors can expect partner-countries to show results through greater efficiency in implementing development assistance, including improvements in their governance.

Nater: What’s wrong with current data quality?

Okidegbe: As I said, an indicator is only as good as its underlying data. Many developing countries spend significant financial and human resources to collect data that are not always reliable or measurable, and that, in some cases, they may never use. In fact, there’s emerging consensus in many partner-countries that they have to change course and improve their data collection, analysis, interpretation and use. The Platform wants to help streamline and demystify the process. We’re defining an approach for selecting a core set of indicators in agriculture and rural development that most development partners would endorse and that partner-countries’ national systems can produce, adapt and use.

The indicators must work at both the project level, and, when aggregated, they should be able to tell a story at country and global levels, as well. They’ll have to be measurable, feasible and appropriate to country context, and that means involving partner-countries in validating the data. That’s why we’re field-testing them in Cambodia, Nicaragua, Tanzania, Senegal and Nigeria, to enhance partner-countries’ commitment and ownership, and getting them peer-reviewed by experts. We aim also to have an advanced draft reviewed and critiqued during the Fourth International Conference on Agricultural Statistics (ICAS IV) in Beijing this coming October.

How have you structured the research?

We’re working with experts from partner-countries, from the Platform’s member organisations, especially OECD/DAC, the European Commission, and the FAO and focusing, to start with, only on agriculture. First, we carried out an extensive literature review of indicators and documented how indicators have been developed and used under unfavourable conditions. We then identified a core set of indicators that should be collected in each of the sub-sectors of agriculture, including how they should be collected and by whom. We grouped the core indicators into three categories, from project component outputs through project development outcomes to final country outcomes. The set of indicators was subjected to rigorous review by experts and revised accordingly.

What will the finished product look like? Who’s it for and how will it help?

It’ll be a M&E toolkit. We aim to publish it by the end of 2007, and disseminate it widely. It will be periodically updated, both electronically and in print. Of course, we recognise that for the tools to work, they can’t be too complicated or too costly, or partner-countries won’t use them. The toolkit should help practitioners in the field select and use core M&E indicators for their projects and programmes. It should also be of use to policymakers and others working at the country level to assess how their programmes and strategies are contributing towards agricultural growth and rural poverty reduction.

The benefits should include better policy monitoring and refinement, more efficient investment in projects and programmes and more efficient, functioning markets. And overall, the toolkit should provide us with a better set of markers on progress towards achieving the MDGs in rural areas of partner-countries.

Changing customary practice is hard. Do you expect resistance?

Obviously, practitioners who are used to certain ways of doing things are not likely to accept new concepts without lots of questions. Such reactions are healthy. We recognise that there has to be buy-in before the core indicators are widely accepted and systematically used. Our task is to come up with the best-possible product and demonstrate to potential skeptics that the new approach both adds value and is cost-effective. This is why the in-country validation exercise is essential because it will help ensure the proposed core indicators are useful and relevant at the field level.

This Platform initiative is an opportunity for donors and partner-countries to come together over a single set of indicators for agriculture and show that we’re serious about both harmonisation and measuring results. It’s an opportunity we can’t afford to miss.

Timothy Nater. 3 Jul 2007