These are our rough notes on the blog post ‘Hearing is Not Listening‘ – our internal discussions and debates before we post. Click here for the main blog.

Q: What is the Feminist Task Force?

R: The Feminist Task Force is a global coalition of leaders of women’s organisations focused on framing poverty as a women’s issue in the lead-up to 2015, date when the Millennium Development Goals (MDGs) will reach their expiry target. The Feminist Task Force advocates for gender equality to end poverty as the United Nations General Assembly considers its options for deciding on what will come after the MDGs, known as the post 2015 process.

Q: I’ve heard very similar arguments to your one on empowerment made about the word ‘participation’ itself. Oxfam have a great book on this – Fuzzwords – which is free and well worth a read if you haven’t already. http://policy-practice.oxfam.org.uk/publications/deconstructing-development-discourse-buzzwords-and-fuzzwords-118173

R: Agreed, I have read sections of this book and it’s very good.

Q: I think you perhaps put too much focus on funding. Firstly, it seems to accept money as an indicator for good development. Secondly, your framing of maternal mortality as a technical problem – “prevention methods are widely understood” – sits uneasily with your point above about a ‘vision to devise an alternate economic system’. In other words, I don’t think maternal mortality is a      purely technocratic problem, and so more money would not necessarily help. Finally, I don’t think you establish that we’re doing any worse on MDG 5 than any of the other MDGs, nor that we devote less money to it (both of which you imply.)

R: It is true that the post emphasizes funding because funding is a easily measurable and comparable indicator of donor interest in a particular cause. However I definitely agree that funding is one part of the picture, and policies must also be considered. Maternal mortality is certainly not technical issues which can be quickly fixed, but we do not that the fixing requires investments which are in part financial- investments into midwife training, etc, and in part require shifts in policies and attitudes.

I do think that several MDGs are more on track than MDG 5. According to the 2013 UN report on the MDGs, targets that have been met or are within reach for 2015 include:  targets 1.A, 1.B and 1.C (MDG 1 Eradicate extreme poverty and hunger), target 6.C (MDG 6 Combat HIV/Aids, malaria and other diseases) targets 7.C and 7.D (MDG 7 Ensure environmental sustainability), targets 8.A and 8.D (MDG 8 Develop a global partnership). MDGs 3, 4 and 5 on gender equality, child mortality and maternal mortality respectively fare particularly badly.

Q: You offer two prescriptions at the end of your piece; ‘financial flows to civil society and women’s organisations’ and ‘development vision’. The former I think is risky. There is already a lot of money going into civil society in general, and I suspect a lot of it goes to women’s organisations, or at least organisations that have some kind of focus on women. However,  funding this kind of small-scale advocacy organisation is immensely difficult. Without particularly well defined outputs or short-term outcomes, I think it’s really difficult to distinguish a vibrant, inclusive women’s organisation from one that just runs workshops and soaks up the money. At the very least, there are huge overhead costs in trying to distinguish and support the ‘good’ organisations. For example, the report you cite lists the median income as $20,000. Organisations of that sort just can’t absorb that much money, and the cost of ending/supporting them are huge.

R: These are very important points. It is indeed difficult to identify grassroots women’s organisations which are effective. However, as AWID’s report points out, the lack of core funding to grassroots organisations is exactly what is preventing them from professionalising and developing scaleable programmes which go beyond piecemeal events and day-long workshops. Long-term, predictable core funding is what is needed for these organisations to develop their capacity to absorb aid money in effective ways, and to define and develop their own responses to development challenges.

Q: Can you define what you mean by emancipation?

R: It is hugely interesting to see that the term emancipation is generally absent from most sections of the international development community’s vocabulary. Similarly to ‘empowerment’, ‘emancipation’ represents a change of circumstances for the disempowered, an elevation from one condition to another. However, unlike ‘empowerment’, ‘emancipation’ carries the notion of a liberation, the idea of becoming free from something that was previously stifling. It could perhaps also be argued that ‘emancipation’ is used in contexts of collection action, such as women’s organising in the 1980s, whereas ‘empowerment’ is defined along more individualist lines, signalling one person’s gains.

Q: As for development vision – I think that is where your argument is strongest. I would be very interested to hear from you, or our readers in the comment box, more details about how you/they think the development community needs to act (or think) differently. A  very tough call for a blog post I know!

R: Very tough call indeed, and something I am thinking a lot about currently- can I get back to you on that?!

When I moved to South Sudan, the world’s newest country, it didn’t help me to know that there were 25 aid worker victims of serious violence in 2012. I wanted to know what the risk/rate of violence was (ideally by location). I wanted to know the rate of malaria amongst aid workers and I really wanted to know the fatality rate of malaria amongst aid workers. Because I enjoy going up hills and black mambas are pretty scary stuff, I was also interested in finding out the risk of getting bitten by one…

In the last two blogs, I wrote about security and other risks faced by aid workers. In this blog I’ll put this risks in context by looking at what all the numbers mean for aid workers in South Sudan.

There are 17,000 aid workers in South Sudan, making it one of the largest aid operations in the world. In 2012 there were 25 major attacks on aid workers, which puts the rate of major attacks on aid workers in South Sudan at 147 per 100,000 (that is a risk of 0.1% per year).  This includes aid workers which were killed, wounded or kidnapped. The breakdown by type of attack is shown in the chart. .

With 9 murders of aid workers, that puts the aid worker murder rate in South Sudan at 53 per 100,000. How does this compare to the murder rates of other places?

The murder rate of aid workers in South Sudan is 44 times higher than that of London. Or about the same as that of Detroit.

While violence is a great concern for aid workers, diseases and accidents account for a much larger proportion of events which lead to medical evacuation, hospitalization or death. For expatriate aid workers, violence is responsible for only 9% of deaths, medical evacuations and hospitalizations. Accidents account for 13% of cases, whereas illnesses are responsible for 78% of all cases.

From the studies discussed in the previous blog, one can guesstimate around 10% of aid workers in South Sudan will contract malaria. While I don’t have data on other diseases, it would be reasonable to assume they would add up to a considerable risk. Over a third of aid workers in South Sudan will suffer deteriorating health. I don’t have a figure for the risk of aid workers to traffic accidents in South Sudan, but this will be high, somewhere between the risk of a major attack and the risk of contracting malaria. 40% of aid workers will find their deployment more stressful than they expect.  10% will suffer from anxiety and 20% from depression. Once deployments are finished, there will be a risk of longer term effects, both physical and mental, but there is little data to quantify this risk.

Finally, I have no idea of my risk of dying to a black mamba. And while I try to accept that in a long enough time scale my survival probability will drop to zero, I very much hope a black mamba isn’t involved. In fact, as Woody Allen put it, ‘I don’t want to achieve immortality through my work. I want to achieve it through not dying.’

This is the final blog in a three part series on aid workers and risk. Part 1 is available here and Part 2 is available here.

___

Sources & further reading:

Here is the statement from Toby Lanzer, South Sudan’s Humanitarian Co-ordinator, where he mentions the number of aid workers in South Sudan.

Data on events which lead to medical evacuation, hospitalization or death come from this study by Rowley et Al. Recommended if you are looking for academic works which analyse rates.

Data on numbers of aid workers killed or seriously injured comes from Humanitarian Outcomes (2013), Aid Worker Security Database, https://aidworkersecurity.org/. Note the 2012 data has not yet been verified.

Murder rate data: the data on murder rates from London was obtained here. Data from other cities is from a yearly report by the ‘Citizen Council for Public Security, Justice and Peace’ think tank.

Following on from our recent literature-themed blogs, Aid Leap has been generously sent a copy of blogger J’s new book ‘Missionary, Mercenary, Mystic, Misfit’ to review.

Given our bemoaning the dearth of creative works about the aid world, we were particularly excited to read a current piece of aid fiction.

‘Missionary, Mercenary, Mystic, Misfit’ is a rip-roaring tale of love and disillusionment set in the dusty offices of Dolo Ado refugee camp in Ethiopia.

The protagonist, a young, idealistic aid worker called Mary-Anne is forced to confront her choices about work and love, and learns some hard lessons about herself and the reality of aid work. The ruggedly dashing Jon Langstrom struggles with the conflicting demands of family separation and a challenging senior posting.

Other characters flesh out the narrative and add flashes of colour. The brash CEO Mark is an archetypal baddy. Left-at-home mum Angie inspires exasperation, but you can empathise with her situation. Mulu Alem, the Ethiopian coordinator weary of endless NGO bickering, is a welcome addition into an otherwise heavily expat-centric cast.

It is a chomping, entertaining read full of outrageous clichés and a quick moving plot. For those who work in the humanitarian sector, it covers familiar territory and you can find yourself quickly relating to the characters and the situations they find themselves in. This is just the sort of thing you would want to settle down with after a long day in the field, preferably with a cold beer.

The world itself is well drawn and populated with some brilliantly incisive observations. Places such as the expat bar ‘Billy Bob’s’ are immediately recognisable, and descriptions of the heady, alcohol fuelled and often farcical settings are nicely done. Likewise, descriptions of the gritty day-to-day challenges – gruelling office hours, the political wrangles of securing funding and constant in-fighting amongst NGOs are spot on.

This is a book written by an aid worker, for aid workers, and has all the comfort of a shared in-joke – but with the consequent alienation for those uninitiated. The liberal dosing of acronyms and industry jargon can render the book impenetrable for anyone not well versed in aid speak.

J clearly knows the environment inside out. He is a canny writer and a master of the snarky blog. However, his fiction writing does not always have the same ease and can lapse into laboured, uncomfortable phrasing – such as the description of Somali refugees as ‘scared, too-skinny, intense, chocolate-colored.’

The characters fit well with the rambunctious style (think Jilly Cooper meets a UNOCHA coordination meeting) but at times the more crudely drawn, superficial personalities can grate. Mary-Anne is a sympathetic character, but her dazzlingly blond good looks and meteoric rise through the ranks are a bit much to stomach. Her distant boyfriend Jean-Philippe is not properly developed beyond the stereotype of a passionate MSF Frenchman, and serves mostly to offset Mary-Anne’s relationship with Langstrom.

This is coupled with some odd plot vehicles. Operations Director Brandon’s disappearance is never satisfactorily dealt with. The final twist was somewhat unbelievable yet predictable, having been obviously built up since the start – and added nothing to the plot except a sense of anti-climax.

The book’s real strength is its sense of immersion into the aid world. The career challenges that Mary-Anne face are realistic and well observed, and her conversations with Langstrom are a useful resource for any eager young aid worker. Though Langstrom’s career recommendations do sometimes feel like advice for the reader rather than natural prose, it is nonetheless interesting and makes sense within the narrative.

All in all, ‘Missionary, Mercenary, Mystic, Misfit’ is an enjoyable, rollicking stomp through the hinterlands of a humanitarian field mission. Highly recommended for a long airport layover or evening off – and you can pick it up on Amazon and Kindle. It is great to see pieces of fiction finally emerging – the question is, where are all the others?

Evaluations should enable development practitioners to learn from current programmes, and improve future ones. Sadly, they currently fall far short of that ideal. Most evaluations are short, shallow, and completely ignored. There are a couple of reasons for this:

From the brilliant http://freshspectrum.com

• Bad knowledge management. Large NGOs and consulting firms may commission hundreds of evaluations a year. However, these are generally not stored publically, and often not stored internally either. Consequently, there is no way for this information to be used for anything.
• Easy to suppress negative evaluations. While some evaluations do get read, they will generally be the positive ones. It is extremely to suppress any negative evaluations.
• Poor design. The terms of reference for evaluations are often impossible to implement. They may have vague or inappropriate evaluation questions, or pack a huge number of questions into a very limited number of days.
• Poor execution. Evaluation consultants are often underqualified and overworked. Especially for more technical jobs, they may just not have the skills or knowledge required.

This is ultimately a problem of incentives. There is no real incentive for evaluations to be transparent or high-quality. Neither donors nor NGOs have any desire to showcase their failures, and overwork and staff turnover prevent them from learning from experience. Donors sometimes hold organisations accountable for the result of evaluations – but they seldom consider the quality or appropriateness of the work.

So what could the solution be? I believe that we need a peer-review system for evaluations, similar to that for academic articles. This could establish and utilise criteria for a quality evaluation, and the peer-review itself could be performed by academics, consultants, or monitoring and evaluation specialists at different organisations.

This can be linked to an evaluation database. Each planned evaluation should be registered before being conducted, and then the peer-reviewed report will be placed in the database, alongside the peer reviewing and any other relevant material. The Millennium Challenge Corporation offers a great example of this. .

This will need to be enforced and supported by donors, at least initially. Donors should signal that participation in this quality assurance process will be rewarding for the organisation. Perhaps bids that commit to this approach could be favoured over those that don’t.

If successfully managed, this could address a number of the problems that begun this article. The design and execution may still be poor – but with the knowledge that the evaluation will be peer-reviewed and openly available, there are greater incentives to improve. It would no longer be possible to cherry-pick the best evaluations for publication. Finally, and perhaps most importantly, all these evaluations would be readily available, so the knowledge wouldn’t be lost.

There are of course challenges facing this approach. The database would need to cover a wide range of potential evaluation types, and not be trapped into only considering a certain approach (such as randomised control trials). Finding peer reviewers would be the hardest part – this may need to be a paid rather than voluntary role, especially given the time pressure to produce and use evaluations.

Such a database could be managed by a small, independent secretariat. The best approach, however, would be for it to be taken on by an organisation specialising in evaluations; perhaps ODI or IDS, 3ie (if they can get over their RCT obsession), Learn MandE, or Better Evaluation. Any takers?

p.s. If you enjoyed the post, see our discussion in the rough notes page, and please leave your comments below!

p.p.s. I noted after writing this that the Big Push Forwards conference report proposed a different but related idea:

“Promoting a ‘kite mark’ for evaluators and purchasers of evaluations who ascribe to certain ethical, moral and developmental standards of monitoring, evaluation and impact assessment. This would seek to create a ‘guild’ who would promote these standards and in so doing publicall and confront poor evaluation processes, evaluators and in appropriate ToRs.“

Here are our rough notes on Making Evaluations Work. Please add your own comments in the main post!

P: There are lots of evaluations which are already publicly available (e.g. just the World Bank publishes hundreds of evaluations yearly). One of the main problems with regards to evaluations is that very little learning is done from all the available evaluations: making more evaluations available won’t solve this problem. Part of the reason for this is that carrying out a systematic literature review in one area is a very time consuming job (often about two years in academia, so should be similar here?)

R:  True. There are a lot of evaluations available – but bear in mind that this covers a huge range of countries and type of programmes. Although a lot is published, the amount available on any individual subject (as almost all systematic reviews start off by noting) is actually quite minimal. So having more available would still be valuable I think. Especially as the World Bank tends to evaluate a relatively limited set of projects, in a relatively limited set of ways – not much humanitarian work, for example.

P: The system you suggest should lead to an increase in evaluation quality. But it would come at the expense of greatly increasing evaluation review time. While this would be fine for actual impact evaluations, these seem to be the great minority… In some humanitarian context, this payoff might not be worth it. As much as it pains me, timely mediocre information might be preferable to good information which comes too late.

R: That’s an interesting and important point. There’s a definite trade-off there. I think in most cases the increase in quality would be worth the effort – but not always as you say.

P: I’m unsure one organisation would be able to handle processing the sheer number of evaluations which are being carried out…

P:  I feel we’re carrying out way too many evaluations, often without much point. I think fewer evaluations with a bit more learning might be a better way to go.

R:  Agree with that as well. Or perhaps to have the existing evaluations far more specific. Evaluations tend to be very general – to evaluate the success of a project. It would be much better to have an evaluation which looked at one key, perhaps innovative approach of the project (how did the new accountability mechanism which they used help the project? for example) rather than trying to give an overall judgement..

P: There are quite a few problems with academic peer reviews (such as biases, which you refer to in your blog) – just to note that these same problems would be carried over. Though I still find a system with some shortcomings better than no system at all.

R: For some of those problems, see http://www.sciencemag.org/content/342/6154/60.full

P: Funding might be an issue with some of your suggestions, given the small proportion of programme spend dedicated to evaluations in multilaterals.

R: This relates to your previous point about resources. Funding is of course a concern – but you certainly wouldn’t use this system on anything more than a proportion of existing evaluations. Given that lots of money gets spent on evaluations as ti is (even though a small proportion of project spend) it would be worthwhile to do fewer, more high quality evaluations and end up spending the same

M: I would agree with both of you – that evaluations currently constitute poor knowledge management, and that a better system is needed to improve this. However – the concern with creating a system like you suggest runs the danger of too much work (creating a further layer of bureaucracy?), potentially diminishing the value of the subsequent output. Though as a peer-reviewed reference bank of evaluations then this could perhaps work.. Its certainly an important issue and a good idea. I would highlight the same concerns to the lack of proper contextual analysis that is actually built into programming, but that is a separate blog post… !

(c) Dilbert by Scott Adams

In my previous blog I looked at the number of aid workers who had been victims of major attacks, while I argued for the need of better data to analyse risks and rates. In this blog I’ll discuss other types of risks faced by aid workers, and show that in most environments, it’s these ‘other risks’ which are the main threat to aid workers.

Operating in a violent environment is a great risk for aid workers. In some areas it can be the main risk. Even if not the main risk directly, it indirectly affects many aspects of aid workers’ well-being contributing to other risks to their physical and mental health. But I feel there are other risks which are often neglected.

Data for aid workers who are killed in violent attacks tends to be very good, since these cases are frequently reported. Data on kidnappings is less reliable, as many are not reported. Data from non-violent threats, such road traffic accidents or malaria deaths is very weak. Data from other diseases is even weaker, and data on mental health of aid workers weaker still.

I would very much like to present you with a pie chart showing you the different components of the overall risk rates for aid workers, but the data is just not good enough. Instead, I’ll look at some of these other dangers faced by aid workers, and attempt to provide some estimate of the risk they pose based on the available data.

Road Traffic Collisions and Other Accidents
Road traffic accidents are believed to be one of the greatest danger to humanitarian aid workers. According to some, like Fleet Forum, they are responsible for the largest proportion of aid worker deaths. While there isn’t good data available on casualties of aid workers due to road traffic collisions, Fleet Forum estimates that around 25% of aid worker deaths are caused by traffic accidents.

Diseases
Studies have shown that over 35% of aid workers report deteriorating health during their missions. An ICRC study found one-in-ten (10.1%) of humanitarian aid workers returning from Sub-Saharan Africa had tested positive for malaria. Worldwide it was just under one in 20 (4.2%).

Malaria is just one of many other diseases which are endemic in many of the countries aid workers deploy to. Since I don’t have access to data, I will make an exception to my dislike for anecdotal evidence to point out health problems appear to be the main issue amongst the aid workers I’ve spoken to in South Sudan and India (note the emphasis: please don’t draw any conclusions from my anecdotal observations). I hear of people falling seriously ill far more often than I hear of aid workers being victims of violence or being involved in traffic accidents.

Mental health
A recent study showed over 40% of aid workers found their deployments more stressful than expected, which strongly suggests training and communication need to be improved. A study of 212 humanitarian aid workers from 19 different NGOs found symptoms of anxiety in three times as many aid workers post deployment as pre-deployment. For depression, the rates were twice as high post-deployment as pre-deployment. The study showed some of these effects were long lasting: three to six months post deployment, while there was some improvement in rates of anxiety, rates of depression were even higher than immediately post deployment.

Conclusion
So what does all this add up to? If calculating risk is difficult, so is working out the overall effect of all these hazards. Being an aid worker is dangerous, but it’s hard to quantify exactly how dangerous. A comprehensive comparative study found aid workers are approximately twice as likely to die than non aid workers. The risk is probably higher, as the study only looked at aid workers who died during deployment, and not those who died post-deployment as a result of diseases contracted during their deployment. The study also doesn’t tell us of the long-term health effects of being an aid worker.

Most agencies spend considerable time and effort addressing security, to minimise the risk to their staff. I am among those who applaud these efforts. But I’d also like to see more effort in measuring, communicating, and addressing other risks, especially in environments where these ‘other risks’ are greater than the risk of targeted violence. While dying from malaria or from a traffic accident might not attract headlines, it does leave you equally dead.
__
References and further reading:
Data on road traffic collisions comes from this report. This contains more recommended reading.
The Centers for Disease Control and Prevention (CDC) has published this overview of the risks to aid workers and how to mitigate them. It contains an extensive bibliography.

This section is rather short. Do you have any other reference materials to recommend? In particular, do you know of any comparative studies on the morbidity of aid workers? Let us know!