Independent evaluation is as important for donor agencies as for individual programmes. That’s why the Independent Commission for Aid Impact (ICAI) is a fantastic idea. It’s responsible for scrutinizing large chunks of DFID’s spending, and giving an independent judgement on impact and effectiveness. Their reviews have ranged from water and sanitation in Sudan, to bilateral aid to Pakistan, and offered important critiques of DFID’s work.

ICAI’s latest review is of “private sector contractors”. Responding to renumeration scandals that hit the press last September, ICAI set out to answer key questions such as: does the use of contractors affect the ability of developing countries to build up expertise? Is DFID too reliant on buying in core skills rather than developing them in their own staff? And are consultants’ profits justifiable?

Unfortunately, ICAI completely fails to answer these questions. The review is technocratic and dull, favouring bland advice such as “update DFID’s bid evaluation process” and “strengthen learning from contractor-delivered programmes”. Useful, but hardly transformational.

Despite the lack of substance, ICAI actually offer firm conclusions, saying that “contractors are an effective option for DFID”. However, they do not present any evidence to justify this. In fact, the authors say that “DFID does not have a way of assessing and learning whether contractors are a better way of delivering aid than other channels open to it”, which seems to somewhat contradict their initial assertion that contractors are good value for money.

This is because of a basic problem with ICAI’s methodology. To assess contractor’s value for money you need a comparison, which in this case should be another delivery method such as multilateral, direct CSO awards, or self-delivery. While the report gives five case studies of consultant delivery, it fails to consider or compare with any other approaches. Consequently, it is unable to offer any kind of sensible judgement. It’s like trying to award an Oscar after watching one film. You might have some intelligent things to say about the film, but there’s no way you can make any kind of comparison.

I’ll leave the question of whether contractors are actually a sensible way to deliver aid for another time. There are good arguments to be made either way – visit this blog from INTRAC for an informed and balanced opinion. As DFID prepares to spend £1.3 billion on consultants in 2013, it’s a shame that the ICAI report left us no wiser about whether they are an effective use of money.