I have different motivations for writing blogs. Often, it’s anger at the way the development business functions (or doesn’t.) Sometimes it’s to share something funny or interesting. Today, it’s because I need help.
I’m evaluating a nutrition project which aims to build the ability of the health system to better provide nutritional services to the community. The project team are smart and dedicated, but schooled in traditional ways of delivering aid. They aim for sustainability; but think of this in terms of a continuing benefit from an intervention, rather than a change in the way the system works.
For example, the project provides equipment to nutrition centres and training on how to use and maintain this equipment. They hope that this will be sustainable; the equipment will continue to be used, at least until it falls apart completely. The team don’t, however, think about who in the current system of nutritional services should be providing this equipment and training, nor why it isn’t happening, nor how they can work with the current system to try and get it to happen.
I’m hoping to bring some insights from systems thinking into this evaluation, although I’m no specialist in the area. Keen to avoid a report smelling of horseshit, I thought I would bring some of the challenges that I’ve faced to the infinite wisdom of the blogosphere.
The basic problem is how to work within a dysfunctional system which is broken far beyond the ability of your project to fix it. To pick a simple example, the NGO supports “nutrition coordinators”, government staff who monitor local health facilities to check that they are fully stocked and the nurses are doing their jobs. Nutrition coordinators are a vital part of the healthcare system, but their ability to do their roles is compromised by the fact that they don’t have any fuel, and so can’t visit the health facilities.
In response, the NGO gives fuel to the coordinators. My initial reaction was that this is a superficial solution, but doesn’t fix the cause of the problem. It’s unsustainable and arguably perpetuates the problem by discouraging the state from properly funding their nutrition coordinators. The NGO responded – a bit irritably – that if they don’t do it, then there would be no supervision of the health facilities.
The actions of other donors and NGOs compound these challenges. For example, it would make sense for the local nutrition authority to take on the funding of the training programme after the NGO left. This would make it a real solution – the NGO would have succeeded in changing nutrition services so that training is now provided on a permanent basis. All the trainers come from the local authority, so you would have thought that it would be reasonably cheap for the government to provide training; it would just require their own staff to do their work correctly.
The trouble is, the trainers require a large supplement to their normal salary in return for conducting training, euphemistically termed a per diem. The local nutrition authority can’t afford to pay this. The payment of per diems by the NGO significantly reduces the chances that the training is taken up by local actors, since nobody has the money to pay. But if the NGO hadn’t paid the per diems in the first place, then the training would never have taken place.
Perhaps this reflects the fact that poverty and incompetence are at the root of a large number of problems here. The nutrition centres are falling apart and money to fix them trickles in slowly, if at all. This is partly due to corruption and mismanagement, no doubt, but also to a lack of money. The NGO has very little influence on higher level corruption and mismanagement. Consequently, my critique of their focus on the symptoms of problems (rather than underlying causes) has a bit of a hollow ring to it; the real problems are just too big for them to address.
The ultimate question, perhaps, is to what extent is it acceptable for this NGO to provide unsustainable solutions which enable the project to continue (and benefit people), but which address the superficial problem rather than the underlying causes? And what can an evaluation report do to help them – beyond clever-sounding but impractical truisms?
Answers on a postcard– or even better, in the comments section of this blog.
There have been some really useful comments on the blog. If you’re reading this on the homepage, then click here to see them; otherwise scroll down.
AID LEAP 
Indeed these are bigger problems than one organization can address, and if health service delivery-focused, this particular NGO may not have the know-how on strengthening local accountability frameworks/systems. Perhaps you could provide recommendations around creating concrete (and funded) partnerships with other NGOs that do have the skill set – organizations (preferably local) with experience in scorecards, budget monitoring, etc.
Basically, addressing sustainability with a systems-, rather than a project-mindset, requires whole new ways of thinking, management, and doing within “old school” NGOs. They might not be ready, but you can invite them to learn.
Thanks. One thing which is maybe a question for a different blog – what makes you so sure that local accountability frameworks, at least as developed by NGOs, are a solution? I haven’t worked with them, but would guess they’ve worked in some places but not others – presumably depending on the context, culture and current methods of engagement between service providers and community.
Partnerships with other NGOs is certainly a sensible idea, however. In practice, I think most of the NGOs that I heard discussed had a similar focus on service delivery, but that probably reflects their networks rather than what’s out there. That would make a useful point for future discussion – thanks!
(I am assuming this is not a relief project and because you state the problem in general terms, I am giving general opinions.)
The problems you refer to: corruption at upper levels, lack of money, poorly designed implementation strategies, lack of political will, dysfunctional system, are the kinds of things that can kill any project’s sustainability. That said, health training projects are particularly susceptible to this problem. I think it is because health training is difficult to incentivize, but not impossible. So the first tip: try to figure out a way to incentivize the process. All development/rehabilitation projects must be based upon buy in at the local level. People have to believe they benefit from participation. This would include government administrators, counterparts, trainers, and target groups. Second tip: how do you get all the actors to perceive these benefits? This is where aid workers earn their bones. I have seen projects that were impossible, but the right people made them successful. Conversely, “no way to fail” projects turned into disasters because of the aid workers involved. I would wager that there is some local person in this project that has a set of ideas on how it could work, but for whatever reason, their voice is not making it into the project description. Third tip: find that person. Solutions to the problems that you have specified are often sitting there, hiding in plain sight and usually that solution comes in the form of a national local staff. Final point: If you can figure this out, the project will not look like what was originally written. However, that is what evaluations are for and project proposals/descriptions can be revised if donors are open to “local, sustainable solutions” and what donor would be against that? Okay, good luck.
Thanks Duke, and good to have you back on the blog 🙂 (For those who missed it, you can read more of Duke’s wisdom here: https://aidleap.org/2015/06/14/oskar-schindler-was-the-greatest-aid-worker-of-all-times/)
I think we were aware of your first two tips and tried quite hard to think of ways in which these local health staff could be incentivised to continue with the training (and the managers could be incentivised to provide it). Where we perhaps fell down was your third tip – we spent a lot of time with the senior management of the organisation, but probably not enough casual time getting ideas from those lower down the management chain, who might actually have had more useful things to contribute. If I did it again, I would probably bring all of the field staff into a room, without the senior manager present, and get them to brainstorm the same ideas – everyone was really quiet when their superiors were around. Thanks for the tip.
I agree with symptoms/causes concern and incentives issues you outline, but also the questions of scope and opportunity to influence the system. I wonder if in addressing some symptoms (even in a less than sustainable manner), the NGO can help create a conversation around sustainable change and potentially build a small constituency for such changes, supporting them as they seek incremental movements in that direction. And great that you have raised these questions in the first place.
I think that’s a really helpful contribution. In this case, perhaps funding fuel for health workers can be the start of a sustainable change – if the NGO sees it that way, and tries to show how important it is for health facilities to have adequate supervision. I guess the challenge for the NGO is to avoid getting sucked into the system – such that their funding becomes a replacement for government money, rather than a catalyst for it.
Perennial problem of the aid sector.
a couple of solutions, full of truisms:
– do better programmes. the aid sector constantly and increasingly talks about root causes and underlying causes and dedicating more funding to these instead of addressing the symptoms. Reality is, the aid sector does very little in this regard aside from rhetoric. The less change that is seen, the higher the advocacy in a ‘rights based approach’ needs to go and the less chance there is of actually achieving anything. ANYTHING. But it is good for PR and back slapping.
– drop the God-complex and accept that aid, especially via NGOs is exactly what it does and not what it wants to say on the tin. it is a sticking plaster. foreign funded NGOs with northern bases, disconnected from national and local politics, civil society and accountability can only have limited impact in terms of systemic change and addressing root causes. the aid/development sector does not even have a decent analysis of what the reasons for the systemic/systematic problems are and only guess at what the solutions are. A logframe approach that has long dominated development- and now a ridiculous theory of change – remain exactly that. theory. we are not doing maths. and no-one has figured out development. it comes from robust societal interaction.
– realise that we actually may be doing more harm than good in the long term. as the blog says, the aid/development sector is constantly undermining local and national accountability by acting as a save to local and national responsibility and consider stepping back from the problem and let people take responsibility for themselves instead of undermining them by thinking that the sector has to come into help the poor people
– stop filling your organisation with policy geeks who come up through univercity courses and are indoctrinated in the same stilted and failed language of development, who have never worked and engaged in a meaningful manner in their own democracy; never engaged in economic sector in their own country; never had to fight for the rights and services that they think should be in place for the deserving poor; never had to work in an accountable organisation that lives or dies by its performance; never worked in a sector that doesn’t moralise to the world; never worked in a meaningful manner in the countries and communities that they say they want to bring about ‘change’ for yet are the ones righting the pretentious policy papers that are drifting more and more into naval-gazing wordsmithing and wishing the world to rights;
– stop paying ourselves ridiculous high salaries and saying ‘we deserve it’, stop with the ridiculous per diems; stop with self-righteousness of thinking we are saving the world from our nice padded chairs, saying we need to pay the best to get the best, letting our own entitlements drive any genuine compassion and understanding; stop staying in nice hotels;
Hi Mark,
I feel your pain. One thing you said though is, in my opinion, not quite right: “the aid/development sector does not even have a decent analysis of what the reasons for the systemic/systematic problems are and only guess at what the solutions are.” Maybe you already have, but if not check out Barbara Harrell-Bond’s “Imposing Aid.” (From the sound of your post you might have been one of her students. Ha.) You might also click on the book list at the top of this blog. There’s some good stuff in there. Again, maybe you are way ahead of me, but you could also get some good leads by googling dependency theory, modernization theory, Marx and Third World development, and (one of my favorites,) “Confessions of an Economic Hitman.” So I don’t think the problems you detail have much to do with a lack of critical analysis. As you and the OP suggest, something is wrong with international aid. I agree, but I also think that there are programs out there where the benefits exceed the costs and actually lend themselves to structural, long-term, solutions: women’s projects, micro-credit, support of dissident writers, infant mortality, refugee protection, emergency aid in war zones, etc. However, all of these sorts of programs can be criticized in the specific; for example google critics of the Grameen bank and insecticide treated mosquito nets. Anyway, the bottom line is we are all humans and it is very difficult to adopt a Malthusian line that the country where the people are dying like flies and the cities are being blown up and women are mistreated and kids don’t have enough to eat will eventually be much better off if we just stand on the sideline and watch. Believe me, I agree with much of what you say, but that doesn’t stop me from trying to make it better in a “locally sustainable way.” Thanks.
thanks Duke for a good solid reply.
I agree with you- loose words on my part. I should clarify that I am not correct in saying there is not a decent analysis- i am not sure that there is a correct analysis – or that there even can be. My issue would be much more along the lines of that there is no particular solution but only incremental steps in a particular context to addressing concrete issues rather than looking at concepts, ideas and visions which cloud and solid thinking- particularly in relation to development.
I am not saying that we should stand to the sidelines but that we should adopt – and probably revert to an approach that is considered more of a service than a moralising cause.
I agree with what you say that there are projects out there that can LEND themselves to making a difference and they are very specific and local, with limited and sensible aims, know their context and their issues and don’t have pretense of being everything to everyone. they are not the sanitised advocacy campaigns or musing of policy wonks, but involve getting down and dirty
thanks for the references to the books. i am not a student of anyone apart from some lecturers in concrete and steel design..
Pingback: What does changing complex systems look like in practice? | The humanist view
Pingback: Welcome | Bedside Readings
Pingback: What does changing complex systems look like in practice? | Bedside Readings
You ask “to what extent is it acceptable for this NGO to provide unsustainable solutions which enable the project to continue (and benefit people), but which address the superficial problem rather than the underlying causes?”
The answer to your question should probably be determined by the combination of intervention relevance, effectiveness, and sustainability (and even efficiency, ideally). The project funder/decision maker has to address the trade-offs implied between these, and decide whether to continue as is, redesign the intervention to address sustainability, or try something else entirely. You say that the project “aims to build the ability of the health system to better provide nutritional services to the community” – is that the right objective? What you have described seems more limited. Again, looking at relevance and effectiveness: who uses the nutrition centres and health facilities? Do these facilities meet the objectives of their users? Is the “nutrition centre” approach the best one, based on practice elsewhere? What does this “training” achieve, and can it be delivered as a one-shot effort?
You also ask what an evaluation report can do to help them. It may be able to help change the scope of their activities and to reconceptualize their activities as lines of business – so operating costs for fuel should be built in explicitly. That might mean scaling back some other part of the initiative, or balancing the fuel/supervision with the expensive per-diem training.
It seems to me that having identified sustainability as the crucial question, you need to focus your evaluation on it. Local government bodies have not committed to providing fuel for nutrition coordinators or incentives for trainers, but there may well have been some changes in the attitudes, motivations and intentions of individuals. Systems are made up of people after all. So ask the coordinators what they are going to do when the fuel allowance stops, who in government might (in)conceivably pay for it, whether the project engaged those people and how it might have done that better. If you can speak to those people, so much the better. Same approach for the trainers etc. That for me would be the most valuable content.
And it’s possible you’ll learn of some positive movement, and then I think it’s worth probing as deeply as possible, to understand exactly what elements of the project have led, say, a nutrition coordinator to work at convincing her boss to find fuel funds. That will give you some practical suggestions for the NGO to follow up – and here I think the more detail the better. For instance, maybe a personal relationship was key, so can the NGO look to build on that kind of thing in future?
Of course, you’ll need evaluation budget for this sort of data gathering but in my opinion it’s worth focusing resources on the key questions at the expense of the generic stuff (yes, everyone agrees it was good to get nutrition advisors to visit the centres, it’s been in all the progress reports, there’s no need for the evaluation to go over it…)
Thanks, that’s actually really helpful advice – focusing on small amounts of progress towards change, rather than just saying whether something was sustainable or not.
Reblogged this on Conversations I Wish I Had.
There are a series of reports here http://www.securelivelihoods.org/content/2265/Sierra-Leone which all looked at efforts (and failures) by international aid actors to build government capacity in the nutrition sector in Sierra Leone. It sounds like we found similar things – basically aid agencies doing the wrong thing (too much training that wasn’t working too well) and not enough of a systems focus. It also linked to a wider politics and governance literature about good aid moving from providing stuff to findings ways to facilitate and broker positive change at a local level – http://www.institutions-africa.org/ and Lant Pritchett’s work on PDIA. None of which is easy to do within current aid incentives and architecture of course – but evaluators should be raising awkward truths.
Bucking the trend, can I suggest a different response?
My comment is predicated on the fact that your statement is true: the issues are too great in the system for this NGO to tackle them. I would assume from this statement that you also think that the NGO is incapable / unable to identify partners who may be able to assist in finding a sustainable long term solution. This seems likely as, from your description, the NGO staff do not understand the concept of sustainability it’s ‘true’ sense.
The NGO probably believes (without ever really thinking about it) that it has an indefinite time horizon – that it will continue being able to raise funds and “plug a gap” permanently. My thought is that the evaluation should comment on the efficiency of the current gap-plugging activities, the scope of which to improve should be within the wherewithal of the NGO. However, it is unrealistic to expect that an evaluation report in itself is going to nudge the NGO into new behaviours; realistically you would end up, as you say, with some impractical truisms. To shift understanding a much greater impetus and investment in training / awareness raising would be needed.
Some of the responses to your blog suggest that there might be scope for some incremental steps towards sustainable change, or that there may be solutions hidden away to be found. If the system is as broke as you suggest, isn’t this highly unlikely? Identifying root causes and finding sustainable solutions takes some thought, awareness of systemic change processes and concerted efforts to achieve. Very likely to be beyond the abilities of this NGO?
If this sounds a bit too defeatist, then perhaps I would suggest that the NGO is simply asked how it would plan for exit – that eventually the aid money that funds their activities WILL dry up, or be diverted to more pressing causes. What exit strategies can they work towards to ensure that the service can continue after they leave? This at least might mean they make some steps towards mitigating the damage caused when they do exit, or perhaps prompt those who run the organisation to rethink their fundamental purpose.