A manager in a field programme that I evaluated recently showed me the glowing findings from his latest monitoring trip – based on a total sample size of two farmers. When I queried the small sample size, he looked shocked that I was asking. “It’s OK”, he explained, “We’re not aiming for scientific rigour in our monitoring.”
I regularly hear variants of this phrase, ranging from the whiny (“We’re not trying to prove anything”), to the pseudo-scientific (“We don’t need to achieve 95% confidence level!”) It’s typically used as an excuse for poor monitoring practices; justifying anything from miniscule samples, to biased questions, to only interviewing male community leaders.
I think managers use this excuse because they believe there is a difference between what we do (monitoring, finding things out, investigating), and what other people do (science, evidence, proof, academia). This is, however, a false dichotomy. There is no magic bar which you jump over, and suddenly find yourself conducting scientific or academic research.
Of course, there are some techniques which require high levels of time or expertise, and so are more appropriate for specialists in that area. This might include randomised control trials, or ethnographic analysis. These techniques are not, however, somehow more ‘scientific’ or ‘rigorous’ than others, but more suitable for certain questions.
For example, if you want to know the effect of a new medicine, then a controlled trial is the best approach. If you want to understand society from the point of view of the subjects of the study (and have plenty of time on your hands) then ethnography is more suitable. If you want to understand how a community-based intervention affected a community, then a focus group discussions mixed with a wider survey might be appropriate. Academics typically asks different research questions, and so use different techniques to those used in monitoring and evaluation.
This doesn’t mean that monitoring and evaluation departments are somehow justified in conducting terrible research. If your focus group discussions are dominated by men, a sampling method poorly constructed, or your analysis relies on cherry-picking data, it’s not suddenly OK just because you’re monitoring rather than an academic. Bad research is bad research, no matter who does it.
Monitoring and evaluation would get a lot better if donors, programme staff, and even some M&E professionals stopped comparing monitoring practices to some kind of imagined scientific ideal. Instead, it would be more profitable if they thought more clearly about the impact of the monitoring, and what it is designed for. Do you need to estimate the attributable impact of an intervention? Do you want to understand how this change has happened? Or do you plan to understand the context in which you work?
All are reasonable purposes, and all have different implications for the type of research that you might want to do. Pretending that basic principles of rigour and good research practice don’t apply, however, perpetuates the idea that monitoring and evaluation is a kind of glorified feedback form, a box-ticking exercise that gathers ‘results’ but doesn’t worry about what they mean.