Why are we so quick to use an index? I’ve been told before that without an index a new publication, piece of research or institution won’t get any coverage. The media certainly love them, but I’m also surprised by the intrigue that researchers and academics sometimes show.

Over a pint recently, a colleague of mine was telling me about a recent field trip that was going to provide the evidence for country X’s ranking in the 2013 version of a well known index. She had been shocked firstly by the length of the questionnaire that participants were faced with and secondly by the speed that said questionnaire was undertaken. The results, she felt, were a list of tick box answers to rather complex and perception based queries.

Foreign Policy’s July/August edition was dubbed the ‘annual failed states issue’. 32 of the magazine’s 112 pages were dedicated to the issue, of which 11 were photographs. The 2013 Failed States Index placed Somalia at the top (though its score was lower than last year), closely followed by DRC, Sudan, South Sudan and Chad. The DRC was pulled out as a case study – alongside Greece and Egypt – and renamed ‘The Invisible State’. Rather apt really. Is the Congo a state? More importantly, as the authors state ‘It’s as if the world wishes to believe in the idea of Congo rather than engage with the actual place that exists’. DRC must be vying for the winner of ‘top five in the greatest number of depressing indices’ award. It’s also received a lot of high profile (read celebrity) visits, and yet, we still seem to be treating it in a similar way to other conflict ridden nations. Yes, we’ve just seen a new ‘offensive’ mandate for the UN. But can we help a failed state by continuing to work through the ‘state’?

I digress – the question at hand is: what impact do indices have?

If the Failed States Index aims to raise awareness of troubled nations or incentivise new action then I’m not convinced that objective has been reached. Those reading Foreign Policy are already interested in foreign affairs and are likely to know that the Sudans, Somalia and DRC are all troubled – basket case – nations. The fact that DRC is ranked 2^nd most failed state in the world may reach the briefing packs of senior officials in foreign ministries, donor organisations or multi-laterals: and in so doing, increase the impact of the point. It is always good, as a policy official, to include a statistic or binary number. Those pushed for time often skim read till they reach numbers.

The power of an index should be derived from the rigor of its methodology.

One of the problems is the subjectivity of weightings used in composed indices. Unless the weightings are 100% equal, then the composer’s bias is inadvertently (or consciously) woven on to the final results. Most indices are composed by numerous components and each component can therefore be given a different weight depending on its perceived significance. Unless the index is entirely independent – meaning that it’s not funded or undertaken by one or a group of organisations with specific goals in mind – then the weightings on each component can be played with until they represent the existing beliefs of the author rather than the unbiased reality on the ground. And this needn’t be malicious but your opinion of what is important may be different from my opinion, but if I’m creating the index then I get to choose and the results of the index will reflect that.

The politically safest methodology, therefore, would be to equally weigh each component. Though some would argue that this requires each component to be of equal significance for the outset otherwise you are creating equals out of unequal values. In 2010, the UN’s Human Development Index was republished as the IHDI or Inequality-Adjusted HDI. The IHDI intends to show how much inequality affects human development, by comparing its values with the more common HDI values.

In comparison, ECHO’s Vulnerability and Crises Indices does in fact directly impact on policy as ECHO (one of the world’s largest humanitarian donors) determines its priority countries depending on the results of the indices. As the Commission states: ‘They are intended to be a common alternative reference framework to ensure consistency in the allocation of resources among the various geographical zones according to their respective needs’.

Indices are always going to be subjective. They are interesting to see which country is better at x or which sector is better at y. But they are less useful in driving or forming policy. They are good for rankings but not absolute values. Therefore, their greatest asset is the comparability of one year’s rankings with the next, and the next, and the next.

For more on weighting and development related indices see Chowdhury and Squire (2006). If you want to look further in to this issue, here are a few (randomly selected) others:

• Center for Global Development’s Commitment to Development Index
• World Bank’s Country Policy and Institutional Assessment
• DARA’s Humanitarian Response Index
• World Bank’s Doing Business Index
• Save the Children’s Child Development Index
• World Bank’s Logistics Performance Index

Do add others . . .

I’ve been reading a lot of Medecins Sans Frontieres (MSF) books recently. It’s interesting to see how a sub-genre of this nature is emerging as part of humanitarian reading matter. In the gender there is already the critical memoirs that aren’t textbooks or philosophical tombs but must reads for all inspiring humanitarians, such as Linda Polman, Fiona Terry, Conor Foley. There are the philosophical or ethical debates, such as The Selfish Altruist or Humanitarian Intervention: Confronting the contradictions. And most recently we’ve seen an attempt at humanitarian fiction with #MMMM.

Following a bit of research, it became apparent that writing, and hence books, is an important thread running through the organisation’s history. These MSF books always try to explain or philosophise around the work that the organisation does. This includes both the historical accounts, such as Medical Innovations in Humanitarian Situations, and the more fictional stories like Band-Aid for a Broken Leg and Six Months in Sudan. I wonder whether it’s the calibre of volunteers that MSF attracts or a result of organisational design or objective setting?

The most recent of the MSF books that I read was Life in Crisis: The ethical journey of doctors without borders by Peter Redfield. My brief qualification in philosophy has meant that I’m drawn to the ethical debates around humanitarian aid and I was very much looking forward to this book. Sadly, I have to say I was disappointed. The sentences are at times unnecessarily long and there is a sense that a phrase or word has been left out. Most of the evidence is derived from the author’s experience of one African country, Uganda. One entire chapter (8) is dedicated to the history of this country.

Redfield poses some interesting questions and draws out appealing conclusions, though the connection between these is not always logical or clear. Nonetheless, I love his reference to the SitRep (Situation Report) as the humanitarian’s Lonely Planet!

For a more efficient means to learn about and reflect on the ethical journey of MSF and particularly the ethical dilemmas their volunteers face, I must recommend the 2010 film/documentary, Living In Emergency. Perhaps, its time for MSF to focus more on these new methods of communication? After all, it’s just as easy to travel with a harddrive as it is a book, and we are all forced to carry laptops, which the harddrive can be plugged in to?

An Imperfect Offering follows James Orbinski’s personal journey as a man and as a humanitarian. Orbinski’s first mission with MSF was to Peru in 1992 and he was President of MSF’s International Council between 1998 and 2001. Simply written, it captivates the reader on a journey through Orbinski’s missions with MSF to places like Somalia, Rwanda, Afghanistan, Zaire, Kosovo, and Sudan. Powerful and compelling, the book is at the same time a very personal autobiography as well as an exploration of humanitarianism and what it means to be a humanitarian.

He shares with the readers his struggles as he tries to apply the principles of humanitarianism in the real world with all its complex political realities. Grown men have been known to cry whilst reading it. It’s a book that really teaches about humanitarianism, both the theory and, perhaps more importantly, the feeling. The only bone I have to pick with Orbinski’s text is his contribution to the representation of humanitarians as chain-smoking drunks…

Another most read for aspiring humanitarian philosophers and practitioners alike is Humanitarian Negotiations Revealed: The humanitarian experience. The honesty of MSF in publishing the details of difficult decisions regarding operations in insecure, political ‘dodgy’ and ethically questionable situations must be congratulated. Following the recent decision to withdraw from Somalia, this would be my recommendation for your September read. Not a bedtime story but yet again central to gaining an understanding of the MSF brand and organisation.

Other books in the MSF sub-genre:

Bortolotti, Dan, 2004. Hope in Hell: Inside the world of MSF

Abu-Sada, Caroline, ed. 2012. In the Eyes of Others: How people in crises perceive humanitarian aid

Abu-Sada, Caroline, ed. 2012. Dilemmas, Challenges, and Ethics of Humanitarian Action: Reflections on MSF’s perception project

Weissmann, Fabrice, ed. 2004. In the Shadow of Just Wars: Violence, politics and humanitarian action

Let us know if any are missing . . .

The estimation of financing needs is becoming a popular exercise in the development arena. Researchers, NGOs, and institutional players have been developing models to estimate costs of different outcomes, such as achieving the MDGs; saving a child’s life; or financing national and international poverty gaps. But, as much as these estimates provide a useful tool for policy makers and development professionals, they also pose some important questions.

First of all, these estimates are in many cases far from accurate: being used mainly for the purpose of public engagement. Values are usually symbolic, and the methodology for calculating them is not clear, nor public. In a recent campaign, one large INGO argued that £2 a month was enough to save a child; how does this compare to the $38 a month ($1.25 per day for 30 days) extreme poverty line, or the monthly plumpy’nut cost of $30 a month per child?

Secondly, when costs and targets become relevant and well known, it is then difficult to change them. The 0.7% target was based on an estimated development cost of poor countries, calculated in the 1970s, with economic structures very different from the current ones. Long-term use of this target, with consequent political consensus, makes it therefore difficult to revise this target, including the use of GDP and not GNI. Why do we still use these out-dated targets?

Thirdly, the quality of the numbers on which the analysis is based is extremely poor. GDP calculation for many African countries is extremely inaccurate and misleading. This is not bad just from a knowledge point of view but it most importantly means a waste of resources. For example, Least Developed Countries (a UN classification based on income and other socio-economic indicators) benefit from favourable trade rules and attention from aid donors. But are the numbers telling us the right story? Are we targeting the right countries?

Finally, and most importantly, most estimates refer to the additional financial resources required rather than considering the impact of systematic changes. For example, the cost of achieving the MDGs is estimated to require between $60 and $120 billion a year, of additional finance. Perhaps, adapting the current development system would cost less and benefit more . . . ?

I am just emerging from a month in Zimbabwe, my first trip to the country, and indeed to Southern Africa. After such a short time in a country, it’s not really possible to say anything meaningful about it. But, in the best traditions of aid-blogging, I’m going to attempt to do so anyway.

Two conveniently alliterative features of Zimbabwe really struck me. Firstly, it’s very racist, which is something I’ve never really encountered before. In my home country, the circles I move in are far too middle-class to be openly racist. When working overseas, the expat white community tends to be, if anything, desperate to demonstrate their multiracial credentials by finding a Black Friend. But in Zimbabwe, the remaining white population numbers about 50,000, and seems completely segregated from the black population. During my month in the country, I never once saw a white person mixing socially with a black person. I was invited to various events which were almost parodies of colonial culture, elderly gentlemen sipping tea on the veranda while the black housekeeper brought scones and jam. There’s a strong current of casual racism. “That’s the University”, my friend said, pointing out a flat, decrepit looking building. “But nobody goes there.” By which, of course, she meant no white people go there.

At first I was fairly judgmental – albeit quietly so, because I liked my white friends and wanted to keep them. After a while, I started to rethink. In the UK, my parents mix almost exclusively with people from their own religious background. In every country I’ve visited, ethnic minorities keep themselves to themselves. Why should I ask the white community in Zimbabwe to be any different?

This links  to my second theme; repression. I’ve never been anywhere as openly repressive as Zimbabwe. As an unashamed politics geek, I always ask people about local politics – and this was the first place I’ve ever been warned not to, as it is actually a criminal offence to ‘insult the office of the President’. Political repression has frequently been at the expense of the white community, such as through land grabs and the proposed indigenisation bill. Faced with this, they have (I think) retreated into their own community, and cut off attempts to integrate into mainstream Zimbabwean culture.

No conclusions then, but it did get me thinking on the nature of racism, how groups stick (or don’t stick) together, and where the responsibility lies. Should the white community be actively reaching out and trying to integrate themselves into mainstream (black) Zimbabwean culture? Or should I accept their right to live separately, just as groups around the world do? Does the legacy of colonialism (and the 1970s apartheid movement) still play a moral role today?

Yesterday (Tuesday 6 August 2013) the UK news outlets were buzzing with a story by The Telegraph suggesting that the Chief Executives of some of the UK’s top International NGOs were overpaid. Sir Stephen Bubb, chief executive of the Association of Chief Executives of Voluntary Organisations, spoke about the need to provide an adequate wage in order to attract quality candidates to the job on BBC Radio 4.  In addition, these top managers  carry enormous responsibility – Barbara Stocking, ex Oxfam GB Executive Director defended her wage explaining that she had to manage a national retail agency (Oxfam shops), the kidnapping of staff members and the lives of thousands of poor people worldwide.

Marc du Bois, MSF UK’s Director, tweeted his salary for 2013 – £71,500 – following this morning’s debates. Some suggested that transparency about salaries might help us feel more comfortable with the amount that these chiefs were being paid but much of the information can already be found in public annual reports. Plus even if it was plastered across buses, transparency about a £1 million wage package for anybody would make me feel uncomfortable.

The amateur economist in me believes that these problems should be solved by the market. A simple demand and supply equation: what are the organisations willing/able to pay and what quality of CEO can that amount buy? But this creates a further problem, namely who should decide what money is available to pay the wages of the CEO? Is it up to the trustees? The beneficiaries? The donors?

It would be interesting to follow private giving flows to those large NGOs named in today’s news stories. If the general public that supports these agencies disagrees with the CEO’s wage then one would expect to see a withdrawal of monthly donations over the next three months by disgruntled or dissatisfied supporters. It is less likely that private companies would withdraw money due to a disagreement with the total wage package – though they may pull out of partnerships due to the danger of bad press coverage. Foundations and trusts provide an interesting case to study of how donors approach staff pay within their grantees. Their donations are usually more tailored and would include a percentage for admin costs, so they have technically already engaged with the debate.

Current private giving flows are not public in a real time sense so we may not be able to track this information to conclude, how big a packet the donors believe is available for paying the ‘top dog’. It might make for an interesting research project for an insider though . . .

If we aren’t able to generate the answer through a demand and supply machine, then we will have to decide what is ‘reasonable’ by some other criteria. Ethics? Moral beliefs about what is right and what is wrong or how much is reasonable and how much is scandalous do not provide us with a strict framework to base decisions on. For instance, is it ok to pay a CEO of a charity £58k? Sounds fair. Is it ok to pay a CEO of a charity £258k? No, that’s unreasonable. Is it ok to pay a CEO of a charity £105k or £89k? Not so easy . . . Where do we draw the line?

Perhaps by unpacking the reasons why large wage packets could be deemed unreasonable for the Chief Executives of aid agencies we can begin to unpack these intuitions. Firstly, any money spent on the wages of staff is no longer available to spend on the ‘cause’. Yet, without staff none of the money accumulated for said cause could be distributed, let alone monitored or evaluated. Second, such large pay packets contribute to the unequal society that many (not all) of these aid agencies are supposedly fighting against. This is only a relevant argument for those who believe that inequality is a central problem and not to those who believe we should just be focusing on removing absolute poverty (which is all relative anyway . . . . future post). Thirdly it concerns us that these ‘top dogs’ might be doing the job for the wrong reasons: to benefit themselves, or their families. Why does someone who cares so much about the poor need over £100k a year?

This is an interesting point. As a middle income aid worker (i.e. I’m not poorly paid and I’m certainly not well paid), I would like to think that I’m working in a sector where there is potential for me to grow and reach exciting heights. My peers all earn over £50k already so it would be strange for me to accept a ceiling of that very same amount for the remainder of my career. I like to dream of having a family, living in a nice house, going on holiday to warmer climes, paying for my children to go to ballet or to play the saxophone etc. But how much does this all cost? If a CEO has two or three children and a partner who also works, then they need to cover the costs of childcare. If they are travelling a lot for work, then childcare may need to include overnight cover and even weekend cover too. Could we add up how much it would cost to live a reasonable life where our children are able to take the opportunities around them and then use that to establish a ‘reasonable’ wage?

There are lots of variables involved, but it would quickly show us beyond what point people are beginning to spend their money on luxury goods or items rather than daily life. Perhaps that is where we draw the line? But whose expectations do we base it on? Those who want to send their children to private schools, summer camps and exotic holidays or those who are content with state schooling, camping holidays and staying with your grandparents? Like everything, the problem is that ‘reasonable’ is a relative concept.

When you get the call from your child – really a grown man/woman – saying they are off to some exotic third world country to work in humanitarian aid, your first thought is “what a wonderful opportunity.  How exciting to see such a place and its people – to learn about different cultures and experience that wonderful new geography.  They’ll have such great stories to tell when they come back.  I’ll miss them but it won’t last forever.”  This is reinforced by the positive response of friends and colleagues.  “You must be so proud to know what a positive contribution they are making to those poor souls in need of basic support.  It’s to your great credit”.

Then, the doubts emerge.  People say, “aren’t you a bit worried about their health and safety?  Isn’t there some conflict going on in that region?  I wouldn’t like all those creepie crawlies and nasty diseases.”   And, what about the risks of kidnapping and hardship, without access to civilised systems and processes.  “What would you do if they get into serious trouble?”.  You wonder why they can’t get a proper, more predictable, job closer to home?”  So, you start to ask questions.  “Where are you staying?  Who are you going to be with?  Have you got all the jabs you need?  What about medicines, finance, insect repellents, clean water and suncreams?  How do we contact you if we need to?  Who’s in charge?  Can we come and visit you?”  All these questions covering the real one, “do you really have to go?”

Eventually, they’ve gone and you wait for an SMS or an e-mail or something to tell you they’ve arrived and what local facilities are like.  Yet, you also dread getting a message – are they ill, hurt or, God forbid, missing?  No news is good news.  Each day gone by takes you closer to their return.  On the one hand, knowing as much as possible about local conditions and what they’re doing helps you to picture what’s going on, however worrying.  On the other hand, not knowing allows an illusion of ‘no need for alarm’ to develop.  These contrasts between wanting as much information as possible but not wanting to hear about the real risks and uncertainties are what is really stressful: the emotional swings of the parental see-saw from pride to doubt.

Finally, they come back.  You utter a great sigh of relief.  And, now you hear all the amazing stories – the successes and failures, the laughter and the learning.  It’s all so exciting and invigorating.  You wonder where they get the strength and determination.  You wish you’d been as adventurous when you were young.  The photos fascinate you, glossing over the deprivation.  Pride swells and a little jealousy rises at the same time.  You tell everybody what a great impact they are having and how wonderful it is to hear their adventures … and to have them home again.

Pride comes before the fall.  You get another call … “Oh no, you’re not really going to …. next”  Here we go – back on the parental see-saw again.

“Be careful when you take taxis”, a grizzled old expat told me when I arrived on my first ever aid deployment, “they rig the fare meters.” I nodded obediently. “And careful at the shops,” he continued, “they’ll cheat you soon as look at you.” I nodded again. “In fact, it’s best you just don’t talk with anyone who isn’t grizzled, old, and an expat.”

He didn’t actually say the last sentence, but could well have done. Large numbers of development workers have a pervasive distrust of local services, institutions, and people. Sometimes cynicism is borne of experience. Often, however, paranoia borders on racism. I’m certainly not immune to this trend. However, two incidents from the last few months have made me think more critically about how we perceive others.

In Sierra Leone, I changed money with a random guy in the street. I foolishly failed to count my money properly, and didn’t notice that he had given me about thirty dollars too little. I was still completely oblivious to this back in the hotel room, when I heard a knock on the door and found the money-changer there, out of breath from having chased my taxi down the street. He apologised for the mistake and gave me the correct change.

In Kenya, I lent the security guard at my guesthouse about ten dollars. Security guards here, incidentally, work a 72 hour week for about a hundred dollars a month, which has certainly put my own complaints about being overworked and underpaid into perspective. We agreed that he would pay me back the money when I returned to Kenya. It was a few months before I arrived back, and the incident had completely passed out of my mind. It hadn’t passed from his; his eyes lit up when he saw me arrive back at the guesthouse, and he ran to pick up an envelope containing the repayment.

This made me realise that the times that I’ve unfairly assumed bad faith far outnumber the times that I’ve been innocently cheated. I think distrust and cynicism can act as a protective barrier, helping to cope with disorientating changes of environment and personnel. This may help keep many aid workers sane – but also contributes to their tendency to lock themselves up in compounds, seek out expat bars and isolate themselves from the world in which they work. So next time you take a taxi, trust the fare meter.

(Does anyone have any stories to add, positive or negative? Add a comment below.)

The international development sector is increasingly obsessed with evidence, results, and value for money. Most aid workers have horror stories about this trend; being told that ‘empowerment’ is not a SMART indicator, asked to quantify the value of training, or struggling to fit a complex project into a rigid proposal format,

This ‘evidence agenda’ has driven a fierce debate. Proponents claim that a focus on results increases the quality of aid programmes, and ensures that scarce resources are well used. Critics argue that donors over-emphasise quantitative, experimental evidence – typified by the increasing role of randomised control trials. They believe that this encourages programmes to focus on easily countable outputs, such as distributing buckets or injecting babies. It prevents staff from working on long-term, more transformative development, which might not be easily counted or photographed.

These fascinating questions have been discussed in depth on FP2P’s Famous Wonkwar and the Big Push Forwards conference. However, these debates have a tendency to become stuck in a loop, preaching to the converted and damning the unbelievers.

With this in mind, I present five recommendations for the vast majority of aid workers stuck in the middle. They accept the right of donors to demand accountability – but are worried that this might undermine accountability to governments and communities in the countries where they work. They understand the importance of effective monitoring and evaluation – but struggle under the weight of unrealistic or downright absurd requirements.

First, assume good faith. Both sides of the debate have a tendency to demonise their opponents. But proponents of evidence-based approaches aren’t trying to return Africa to a colonial state, as is sometimes sinisterly implied. Similarly, critics of quantitative research aren’t trying to hide a terrible aid programme. Assume that the person you’re talking with is genuinely interested in doing the best job they can, and everything will go much better.

Second, find a common language. The development sector has enough jargon, without inventing more for the purposes of this argument. Many critics of the evidence agenda root their argument in concepts of different epistimologies and forms of knowledge, which don’t help anyone outside a narrow academic background. The same goes for economists who wield quasi-experimental jargon like a battering ram. Keep language accessible, and tell off others when they don’t do the same.

Third, pick your battles carefully. You may think that ‘counting beneficiaries’ is an insulting, top-down simplification of the complexity of your programme. It doesn’t matter – in all likelihood, you’ll still have to do it. Save your effort for more important arguments, such as when you’re asked to restructure your programme to fit donor priorities, or your advocacy project is rejected based on a cost benefit analysis.

Fourth, find common ground. Stereotypes about narrow-eyed, viciously-focused calculator-wielding economists are seldom fair. The maestros of randomised control trials – Abhijit Banerjee and Esther Duflo – make plentiful use of qualitative data, continually trying to understand why interventions do or don’t work. Almost all development workers are well aware of the limitations under which they work. Be honest about the problems with the tools you use, and you’ll often find that you’re not alone.

Finally, distinguish between inappropriate tools and appropriate tools used badly. In general, even the worse methodologies have some value. Seek it out, and use it to your advantage. For example, logical frameworks have a terrible reputation – but for simple programmes, they’re excellent ways to focus attention on a single goal and achievable outcomes. Even for complex programmes, rigorous thinking can be helpful in clarifying exactly what you want to achieve. The problems come when they cease to be tools, and become an unchangeable statement of what the project aims to achieve. Try to recognise what is and isn’t valuable, and use it accordingly.

NOTE: We normally put rough notes on all our post – the discussion between the Aidleap team prior to posting. Unfortunately this time they were too busy to develop rough notes – so please add your own comments below and we’ll reply.