Turn left for Development
There’s a new cool kid on the development block. Complexity theory has delivered a devastating critique of existing development programmes, and offered a promising, shiny alternative. With momentum from Ben Ramalingam’s book ’Aid on the Edge of Chaos’, as well as the backing of intellectual heavyweights such as Owen Barder and Duncan Green, there are great opportunities to improve the way that the international development sector functions.
This blog outlines three potential pitfalls facing ‘complex programmes’. While a complex programme could take multiple approaches, this blog focuses on development programmes which spurn direct delivery of assistance in favour of attempts to change the systems that the poor work in, whether private sector, public sector, or civil society.
1) Complex programmes can ignore the poor. All the literature on complexity theory stresses the need to consult with and understand the views, attitudes and potential of the poor. But then again, so do manuals on logical framework analysis, illustrating the potential gulf between rhetoric and practice. There are several reasons why a complex programme might be difficult to reconcile with a participatory approach.
Firstly, the jargon of complexity theory is often impenetrable. Even the clear-thinking (and writing) Ben Ramalingam didn’t manage to avoid getting bogged down in terminology like agent-based modelling, ruggedness, and fitness landscapes. Even the phrase ‘aid on the edge of chaos’ means little more to me after reading the book than it did before. This potentially restricts the design and implementation of complex programmes to ‘experts’ with an academic background in complexity theory, or at least the nerve to bluff their way through.
Secondly, a programme which aims to change systems, rather than directly provide assistance, may have no contact with the poor at all. For example, rather than providing agricultural inputs directly to farmers, a complex programme might facilitate the efforts of businesses, business advocacy groups, farmers associations, and politicians. Since the direct recipient of assistance is the intermediary players rather than farmers themselves, you could run an entire programme without ever meeting with or talking with an actual farmer. Obviously this isn’t how any development programme should be run – but as a complex programme focuses on the system level, it could end up ignoring those who use the system.
2) Complex programmes might find nothing to do. Facilitation, supporting champions, and creating an enabling environment for change often seem to require holding meetings. The trouble is, there is a limit to the number of meetings you can arrange, and relevant stakeholders only have a limited amount of time in which to receive your support, before they spend more time being supported than doing anything useful. A programme which takes a genuinely facilitative approach may simply find that it doesn’t have much to do. There is certainly a place for a small-scale, patient hub that tries to bring people together and deploys expertise strategically – but that is never going to be more than a fraction of a fraction of a percent of the development budget. Putting more money into complex programmes (the donors preferred approach) could just end up reducing their impact.
3) Complex programmes are extremely hard to monitor and evaluate. It’s very difficult to see a system changing, and even harder to tell what part of that change you contributed to. Owen Barder is right to stress the need for better data, and Duncan Green is correct to emphasise flexible, iterative monitoring. Even so, if a programme isn’t delivering tangible outputs, it’s very difficult to judge success. To put it another way – it’s challenging to tell the difference between a transformative organisation facilitating change in complex systems, and a load of people sitting around writing reports and holding meetings. It’s a challenge for evaluators, and a nightmare for donors who want to spend their money wisely. To link to the first point above, an M&E system for a complex programme is likely to be more reliant on ‘expert opinion’ and less reliant on the views of the poor, who will probably never have heard of the programme and may be ill placed to judge its efficacy.
All three points are illustrated by the case of TradeMark Southern Africa, which was recently closed following a red rating from the Independent Commission for Aid Impact. TradeMark Southern Africa aimed to improve Southern African trade performance for the benefit of the poor. Although not explicitly influenced by complexity theory, it worked in a complex environment, primarily through intermediaries rather than directly with the poor. The ICAI evaluation found that it displayed all of the above issues. With such a big distance between operations and beneficiaries, the programme had “inadequate focus on the poor”. Difficulty in spending money led to “wasteful spending that contributes little to overall goals.” And they had difficulty in monitoring their impact; for example, TMSA monitored average traffic crossing times across a border, rather than the more challenging (and difficult to attribute) price of transport and goods.
None of the above points dismiss the many insights that complexity theory has brought to development. However, it should be recognised that there are real dangers in consciously trying to work in complex environments, as well as opportunities. As complexity changes (as I hope it will) from the latest exciting buzzword to a practical approach to implementing development programmes, I believe some of these challenges are addressed.
AID LEAP 
I’m glad you are thinking about complexity. But to be clear: thinking about complexity is not a policy choice. The claim is that the environment we are working in is a complex adaptive system (whether we like it or not) and that, to be effective, we have to recognise this and change our behaviour accordingly.
It may well turn out that “there is nothing to do” (though I doubt it); but if so, that is a consequence of the facts on the ground, not of the decision to think about complexity.
Similarly, a programme designed for a complex environment ignore the poor; but a programme designed for a simple system and implemented in a complex environment might not serve the interests of the poor either. Indeed, I would argue that this is not only possible, but quite common. The interests of the poor will not be served by pretending that complex systems are simple.
Finally, I don’t recognise at all the idea that an M&E framework for a complex system is more likely to rely on expert opinion than the poor. On the contrary, thinking about complexity should encourage us to focus much more on measuring and evaluating real world change, and responding to it, and rather less time sitting in workshops talking about logframes.
Owen
I’m afraid the author of this post has misunderstood some basic aspects of complex systems research and its use in development. Just a few points in response:
1. By stating up front that the post is focusing on projects that work only at the systems level, it is discounting many many projects that employ complex systems precisely in order to enhance participatory approaches. Look at systems approaches to dealing with malaria, peace building, natural resource management. These are all grounded in the basic notion that you cannot understand and work with systems without drawing on and engaging with the perspectives of the poor. To then say these projects ignore the poor and that this is a property of all complex-systems informed efforts is frankly absurd. Mentioning in passing that the language of complexity makes it hard to engage with the poor seems to suggest imply that everything else in development is jargon-free and transparently clear to poor people.
2. The idea that complex projects find nothing to do couldn’t be further from the truth. Complex-systems informed programmes are a way of identifying entry points and ensuring rigorous design and implementation. Done right, it leads to better ways of programming activities, and a systematic rationale for particular mix of things, and how this should change over time. Again, this ‘risk’ is largely a feature of the limitation placed by the author of the post (to focus on projects that are at the systems level) rather than anything inherent to complexity-informed programmes per se. Programmes at a systems level may utilise complex systems principles, but they may equally do nothing of the sort.
3. Monitoring and evaluation is harder in complex environments – but as recent work by DFID has shown, it is no less important to try. And there is effort underway to establish a new centre of excellence on impact evaluation to do address exactly these challenges. By focusing on only those things that are easy to evaluate, development becomes narrowed down to only those things that are measurable – and we are all aware of what the risks are there. Its worth looking at former USAID Administrators Andrew Natsios work on the risks of this approach: http://www.cgdev.org/files/1424271_file_Natsios_Counterbureaucracy.pdf
4. Finally, linking the failed TMSA programme to complexity thinking is such a unfounded leap (no pun intended) as to beggar belief. TMSA was a failure of incompetence, managerial oversight, financial management. The idea that this in anyway would be attributed to a failure of complexity theory just illustrates how poorly the author of the post seems to grasp these issues. There is work underway to apply systems thinking to trade programmes in DFID ad elsewhere. Highlighting TMSA as a failure of complexity thinking just reinforces the highly selective use of evidence throughout this post.
I have spoken about these issues frequently over the past few months and years, and while thoughtful and intelligent, and considered contributions to all debates are always welcome, I fear that this post is sadly none of these things.
Apologies if the tone of this message is rather cross – the author has managed to stir this usually mild-mannered fellow into a very grumpy state. I am sure it will pass 🙂
Sorry to provoke such grumpiness so early in the morning!
This post does explicitly just focus on a narrow range of ‘complex programmes’, focusing on changing systems, which the only type of complex programmes that I know anything about in detail. So perhaps this should be titled ‘risks of working with systems’ or something like that. That type of programme does seem a natural fit with complexity theory, and there’s a clear progression from saying ‘everything is part of a complex system’ to ‘we should be working directly with this system’. So it didn’t seem too big a leap at the time. Perhaps I need to rethink that though.
Hi Owen,
Thanks for that. Agreed that complexity is all around us – a point made by Jamie Pett on twitter – so I guess the risks are in how we respond to it. The risks of not responding are even greater.
On the M&E framework point, the more complex a system you are trying to monitor, the more challenging it is. If the aim of your programme is to make a market system work better, there are simply no easy variables you can measure to see if that is happening or not, and why. There are loads of potential indicators, each of which need to be interpreted in context and assessed based on the quality of data, likelihood of respondent to have answered honestly, and much more. For example, you might be interested in % of the value of good which goes to the farmer – but there’s often no way to get that information, and you don’t know who to trust when you do get varying answers. Similarly, assessing attribution is likely not to be a matter of counterfactuals, but of plausible theories of change (and what is ‘plausible’?) with qualitative information justifying attribution at each step. in practice, all the systems I have seen rely on some large surveys held together by large doses of expert opinion.
As you say, it’s not a choice of whether to work in complex systems or not – just a fact. But poor monitoring and evaluation is a definite risk, and not an easy one to solve.
We work in a difficult environment here in the western borderlands of Pakistan. Our experience is that unless complexity approach is adopted there is little prospect for developing reaching conflict prone areas and marginalized groups. Please go to our web site http://www.srsp.org.pk and read our paper( a practitioner and not an academic one) which shows how the rural support programme( a distinct group of non profit sharing a common approach to development) worked in such an environment and deliver development
Interestingly, just today I read a report from a programme which basically said (I paraphrase) – ‘We try to facilitate change in complex systems rather than directly intervene. Consequently, we have minimal contact with the poor, which creates difficulties with a M&E system requiring us to have field visits.’
So while I take Ben’s point above that this is not necessarily an issue with complexity theory, I think the risks identified above are certainly relevant…
Pingback: Reflection on 2013: Merci beaucoup | AID LEAP
Pingback: Can contractors deliver change in complex systems? | AID LEAP