Ok, that was a clickbait title. But what really happened is not so far off.
In my recent course on “Evaluation” my grad students and I worked on a systematic review of development aid to Afghanistan. Every year I team up with a partner/client and then study with my students a real-life problem, which is of importance to our partner. In this case, we worked with the Afghanistan desk of the German Federal Ministry of Economic Cooperation and Development.
Our job was to produce the first-ever systematic review of development aid for Afghanistan. With the help of our brilliant research librarian, Patrick Labelle, we systematically searched for evaluation reports about aid to Afghanistan published since 2008. Based on a set of pre-defined, transparent and replicable criteria (one of which was a threshold for methodological robustness), we included 148 studies in our sample. Of these, 32 were rigours quantitative impact evaluations. Other types of evaluation reports were performance audits, formative assessment, and bilateral country-level evaluations.
One hundred forty-eight good-quality studies is a lot of evidence. So, what did we learn?
By and large, aid was not very often effective. Donors overestimated the capacity of the Afghan government and underestimated the detrimental effect of insecurity. They usually designed programs which were overambitious, ignored cultural norms, and assumed demand for programs where there was none. Donors were aware of the challenging context, but there is little evidence that they changed their approach over time. Instead, they repeated the mantra that “development needs time” and carried on as before.
A major lesson is that ambitious programs, aimed to be transformative, usually failed. Aid only had a fair chance of being effective when plans were modest, aimed at direct results (such as, for example, building small infrastructure and providing access to water and electricity), did not assume unrealistic partner capacities, were aware of the cultural context, did not spend aid money too fast, and did not devote aid money in highly insecure regions.
At the level of sectors, a clear pattern emerges:
- Interventions in education and health were reasonably effective. Very tangible progress was made with regard to access to education for boys and girls and to access to basic health care, which massively reduced child and maternal mortality.
- Rural development programs, often implemented through newly created community-level organizations, helped to build a large amount of small necessary infrastructure for water, energy, irrigation and transportation. This contributed to improved livelihoods and strengthened coping mechanisms (but has not led to sustainable economic growth that translated into jobs or income opportunities).
- Programs supporting macro-economic development, macroeconomic policies, and financial management achieved some progress in the early days but then lost momentum. Interventions aimed at promoting the private sector were rarely effective.
- Likewise, interventions in the governance sector (which includes capacity building, public sector and regulatory policy reform, democracy promotion, election support, anti-corruption programs and rule of law etc.) were not often effective.
- Programs aimed at improving capacities for the Afghan central administration rarely succeeded. In the few instances where progress was made, it remained confined to small silos that did not translate to overall state capacity, and/or it was “borrowed” from the so-called “second civil service” consisting of well-paid Afghan returnees or international consultants. Also rarely effective were programs aimed at increasing capacity for sub-national administration and those meant to build up capabilities for managing relations between the centre and provinces to provide meaningful decentralization.
- Finally, aid was least effective in the sectors of gender and stabilization. There is no evidence that the (extremely costly) stabilization interventions led to more security. On the contrary: aid often exacerbated inter-group tensions, fueled corruption and attracted violence.
- Concerning gender, the evaluation reports suggest that effectiveness of gender programming was very low. This is especially true for larger, more ambitious projects aimed directly at changing gender norms and relations. A lack of demand and an adverse cultural context made progress elusive. Not surprisingly, the few pockets of success were programs aimed at more modest objectives, such as rural literacy, increased access to health and education, and better livelihoods in women-specific activities within agriculture.
“But”, you ask, “that’s all interesting, but what’s with that clickbait title of yours”? Why do you think that University of Ottawa grad students may just have raised US $37 million of aid for Afghanistan?”
Here is why. Assume that donors will read our review. (I know they will). Furthermore, assume that donors are to some extent capable of adjusting their strategies based on evidence and that they will shift at least some of their aid from sectors where aid is demonstrably ineffective to sectors where aid is demonstrably effective. Finally, assume, very conservatively, that donors will shift just 1% of their aid from ineffective to effective sectors. Afghanistan received last year US$ 3.79 billion in official development aid. If donors shifted 1% of their aid from ineffective sectors to more effective sectors, then there is, magically, an additional US $37 million of aid available for Afghanistan.
I hope that donors will revise and modify their allocation of aid, based on the evidence which we presented. We owe it to the people of Afghanistan that we allocate aid in the most effective way possible. The work my students and I did can help us to get a bit closer to that goal.
PS: You are not convinced that the findings of our review are solid? See for yourself. We made every effort to be as rigorous and transparent as possible.
The summary report (which also details the methodology) is available here: Zürcher, Christoph. 2020. Meta-Review of Evaluations of Development Assistance to Afghanistan, 2008 – 2018. Summary Paper.
And the five underlying reports are here:
Zürcher, Christoph, with Andrew Coon, Marissa de la Torre Ugarte, Patrick Labelle, Binxi Li, Razan Masad, Hassina Popal, Reem Saraya Maryam Shah, Michael Swenson, Ella Sylvester, Anna Vanderkooy, and Mengrou Wang. 2020. Systematic Review of Impact Evaluations of Development Aid in Afghanistan, 2008 – 2018.
Musharaf Shah, Maryam, Ella Sylvester and Christoph Zürcher. 2020. Summary Report of Eleven Bilateral Country-Level Evaluations.
Popal, Hassina, and Christoph Zürcher. 2020. Summary of Selected SIGAR Reports, Afghanistan, 2008 – 2018.
Saraya, Reem and Christoph Zürcher. 2020. Summary Report of Evaluation Reports by the Asian Development Bank, 2008 – 2018.
Saraya, Reem and Christoph Zürcher. 2020. Summary Report of Selected Evaluation Reports by Multilateral Organizations and NGO, 2008 – 2018.
Or download all reports in a zip-file here.
Recommended: Gaza’s Ongoing and Potential Calamity