Sudan, circa 2007. At the time I was a consultant doing free-lance evaluations for humanitarian organisations big and small. I had spent the last three months assessing a major international NGO operating in several developing countries. The team leader and I were supposed to leave together for the final stop, a field office in a remote village South-East of Juba, the capital of the now independent South Sudan. At the last minute, though, we had to change plans: my colleague would stay in Juba to meet with local authorities while I would have to travel to the village alone and be back in time to catch an earlier flight back to Europe. Total time to assess the field office: 24 hours.
Not without hesitation, I left for my mission impossible. Over the course of the following day, it became clear that I was actually on mission absurd. I was supposed to understand what was going on with this NGO’s programme (a rehabilitation initiative in favour of marginalised people), its staff (five expatriates and around fifty local employees), and its finance and administration in just one day. Oh, and I was also supposed to consult the aid ‘beneficiaries’ – the people for whom the NGO, the donor financing its programme and me as its evaluator were doing all this work. By the time I finished my seventh 30-minute meeting, my notes looked like a blur. Despite my best intentions, I cancelled the consultation with local representatives. I went back to Juba hoping I would never have to go through the same experience again.
Luckily, I haven’t but the lesson I learned in Sudan has haunted me for years. Even more troubling, I know several colleagues who still experience a similar challenge today. How can we assess the impact of development aid programmes if we don’t even know the local context and its dynamics? I am not the only one asking these questions. In fact, just around the time I was on my mission absurd two researchers met in Bangladesh and decided to try to answer these questions by using a radically different approach – the Reality Check Approach (RCA).
This is how it works. Instead of carrying out field visits and short interviews, researchers spend several days and nights with the people the project is meant to benefit. They witness their everyday life, listen to their struggle, learn about their hopes and plans for the future. No computer or notebook in sight, no pressing questions, but free-flowing conversations, just like what you might hear between two good acquaintances. It is up to the host to decide what to share, when and how.
What we learn from these conversations gives us a much fuller picture of the situation. A good example is RCA’s recent report, Us in Their Shoes (2015), which gathers the stories of 27 families living in poverty in Indonesia, where RCA is based. You read about people hiding their real economic conditions to avoid being labelled as ‘poor’. Some of them will even go as far as living under the highway rather than in their neighborhood so they won’t be stigmatised for their poverty. You read about migrant workers who send money home for their children’s savings only to come back and discover their relatives have squandered all the funds, leaving them in complete destitution. You read about mothers having to make a choice no mother should ever have to make – deciding which of their children to support and live with. It is a kaleidoscope of perspectives on poverty that speaks volumes, far better than elaborate recommendations and a lot more nuanced than monolithic statements.
RCA researchers limit themselves to connecting the dots. They note how these and other families they visited usually feel overwhelmed, with little time for themselves, and yet they remain strong. Some families take pride in ‘living modestly’, refusing to be called poor. Indeed this confirms the findings from other, more traditional research studies. The only poverty these families are comfortable talking about concerns the public services they use – poor phone connections, poor roads, and poor water supply.
These stories also speak of a fast-changing environment that leads people to need more cash to pay for their everyday needs, like food, phone cards, transport, and health care. The pressure to earn cash is such that farming is no longer a viable lifestyle option. Too risky, too unpredictable. Salaried work is a much more attractive alternative and the way to get it is by investing in one’s education.
We need more of this kind of research before and after putting in place aid programmes. We need to rely on local researchers who are already familiar with the context they analyse. To their credit, a few donors like the UK’s Department for International Development are starting to use the Reality Check Approach. Let’s hope this trend picks up so we can learn more from multiple ‘small voices’ on a regular basis. Or at least until it becomes the exception to parachute international consultants on one-day trips.