Indiana Jones and the Missing Baseline
I’ve conducted several reviews of international development organisations, looking across their portfolio of programmes to assess and compare impact. Each review has run into a key problem – nobody could tell us about any finished programmes. We were given enthusiastic reports about current and upcoming projects, but all the completed ones had been forgotten. It makes me feel a bit like Bill Murray in Groundhog Day, stuck in a permanent and repeating present.
This has really highlighted for me the problems of institutional memory in our sector. All too often, programmes are forgotten about straight after they’re finished. If there is an evaluation, it’s conducted, filed, and then lost. This has two obvious negative consequences. Firstly, we never really learn from the past, as ‘lessons learnt’ are swiftly forgotten. Secondly, there are few considerations of sustainability (or incentives to consider it.) Evaluations are almost all short-term, and nobody revisits the programme any later – because nobody knows that there is a programme to visit.
This isn’t an original critique. (In fact, Aidleap has made it in a previous post, though our proposed solution there was pretty unpopular.) But it is curiously unaddressed, both in the development and (as far as I know) the academic world. So I propose a reasonably simple solution. We undertake some evaluation archaeology; we find and evaluate an old and forgotten programme .
First we need to find a programme that finished ten or so years ago. Obviously records will be hard to come by, especially as this was in the early days of the internet, but hopefully there will be something etched on stone tablets in the depths of DFID’s archive. And then, like Bill Murray breaking out of his time–loop, let’s go and evaluate it. We could see if anyone still remembers the programme, whether any farmers use whatever technology it introduced, or whether it has any impact on the workings of government and other systems today.
Obviously there are technical challenges in assessing impact. There is unlikely to be any baseline data, and certainly no control groups. Recall (asking people what happened in the past) is dodgy at the best of times, and pretty meaningless after a ten year gap.
However, with a good understanding of what the project was trying to achieve, secondary data and monitoring reports, it should be possible to find out something worthwhile about the sustainability of major programmes. For example, evaluation archaeology into a programme that set up self-help groups could learn whether any groups are still going after 10 years, and what factors underlay success and failure. This is likely to tell you far more about what really works than an evaluation conducted straight after the end of the project.
Does anyone know of an evaluation which did this? Is anyone interested in collaborating on such a project?
AID LEAP 
Until the information revolution, record-keeping for development projects was almost comically non-existent. Going into an NGO field office and asking for documentation from their projects from 10 years previously was an exercise in futility.
There’s an incentive for this amnesia, though. If most micro-development has zero demonstrable impact over any significant timescale, it’s not in the interests of the sector to maintain records, let alone re-visit them.
Two longer term evaluations suggested by Paul Harvey, both of cash transfers:
https://t.co/X0SZ7AgHeH
http://t.co/A7ZwrhdR45
Reblogged this on The World… Thoughts, Books and Adventures and commented:
Continual evaluation (after the project has finished) is a HUGE issue. It even came up during my thesis defence as a question, about why organizations never do follow-up because they have no incentive to from donors. I hope that in coming years, more organizations will come up with new techniques for making long-term sustainability a real priority.
You could argue that the assessment of Mali’s Agrometeorological Advisory program that I am leading does just this. We are looking at claims about a program that were established 30 years ago and actually evaluating if they are valid…and if anyone is actually using the advisories. All with field data on farmer behaviors, etc. gathered now. It has been a hell of a process, but our preliminary report will come out in the next few weeks and could provide something to look to…
Thanks! Please post on here when it’s released (if public?) or if you tweet it to us I’ll spread it. Paul Harvey suggested a couple of evaluations (above) – happy to see that the idea is in use, although not that frequently.
Tjip Walker and a group of people in USAID’s Bureau of Policy, Planning and Learning are looking to identify and support a number of ex-post evaluations. You might get in touch with him.
There is a garden near here maybe about one acre in size. The donor sorted out the water pipes, tanks, taps, fences. It is full of vegetables all year round and has been functioning for many years. The donor was ORAP. It is known locally as the ORAP garden.
There was a garden nearer here that other donors were advised not to build. They didn’t take the advice, built it and it no longer exists. Maybe the paper work was burned to hide the evidence. But I don’t think so. Maybe it was just a summarised tick box on an A4 sheet of paper.
Dear Aid Leap,
The San Martin Jilotepeque Development Program in Guatemala (1972-1979), funded by Oxfam UK and managed by World Neighbors, did a baseline survey, a program-end evaluation, and another evaluation 15 years later (in 1994). The baseline survey somehow survived the 1976 earthquake (22,000 people, caused by the fault that runs along the northern edge of San Martin), and a civil war, during which documents had to be rescued several times from buildings that were in danger. Those of us in World Neighbors felt we had learned a tremendous lot from the study, and did 5-year-after evaluations of several of our other programs through the years. When the International Institute of Environment and Development (IIED) in the UK got hold of the report, they decided to organize a conference in Hyderabad that would be about precisely the subject you have raised. They wanted to have NGOs present two such “5-year-after studies” from each continent, and then discuss them and what could be learned from them. They contacted hundreds of NGOs, and could only find two or three other such studies anywhere in the world. I still work in development, and I don’t believe there has been much of an increase in such studies since then, despite the fact that all of us who attended the conference agreed that such studies were of great importance.
Sincerely, Roland Bunch
Author, “Two Ears of Corn” and
“Restoring the Soil”
Brilliant comment, thanks Roland! Is there any report from that we could read? And how did you deal with problems of attribution after 15 years?