Quantifying quality with participatory numbers

“How do you measure social change?” As a consultant, I am often asked this question when I evaluate programmes or campaigns that are meant to produce some sort of social progress in the global South. I guess the underlying dilemma is about whether we can quantify qualitative changes, such as better dialogue between aid donors and civil society organisations. Is there a way to measure quality with numbers?

Participatory statistics provides an answer. Recently promoted by researcher Jeremy Holland in his Who Counts? The Power of Participatory Statistics (2014), the idea of using numbers to describe complex social interactions builds on two key elements.

First, you complement official stats – usually produced by a government agency in a given country – with data gathered from a broader group of relevant people. In development co-operation programmes, this group could be a local community or a network of women’s associations, for example. Adopting a participatory approach allows you to answer data requests from different angles, making the overall picture more accurate and granular. That’s the case of a sustainability project in Suriname, in the Amazon, where indigenous groups use participatory GIS technology to map their land. The locally generated data is then compared with the official information provided by the government and used by these groups to claim their rights to land and natural resources.

Second, you use a wider set of tools than usual to generate more reliable data. Participatory statistics brings together methodologies that have been around for decades, such as ranking, scoring, social mapping and estimating. Through these methodologies it is easier to analyse the social dynamics around a certain programme, even on a massive scale. For example, by 2010 thousands of community groups across Rwanda had ranked wealthy and poor households in 14,837 villages by visualising them on cloth maps based on six agreed categories. The result was an unprecedented trove of information, which Rwanda’s Ministry of Health used to better understand who should receive free health care services and who should pay for insurance.

Over and over again, the evidence gathered through participatory statistics has dispelled several misconceptions about this approach – that it is primarily based on perceptions; that it engages people with little or no numeracy skills; that it is simplistic. Quite to the contrary, engaging a broader statistics community, particularly at local level, has led to discover that most community members consulted have developed their own ways of calculating things. Even those who are illiterate may use visual or tactile tools to enumerate, like seeds and beans. As a consequence, it is now possible to generate whole sets of specific data that would otherwise escape researchers. Also, engaging communities in validating data helps reduce margins of error or even the unintended bias of the researcher who carries out the study.

In the context of social change, what’s more important is to look at what happens when communities actively participate in statistical studies concerning them. Once people are recognised as analysts in their own right, with tools and knowledge of their own, they gain confidence in their ability to interpret the data about their community. They feel more capable of identifying challenges and solutions. And they are more likely to translate this awareness into action.

A brilliant example is the self-evaluation carried out by members of a donor-funded social movement project in Bangladesh about a decade ago.  By sorting through more than 8,000 statements made by movement members about their ability to bring about change, it was possible to detect behaviour patterns and measure the level of members’ economic, social, political and personal empowerment. The process itself was empowering, particularly for women and girls, as it led movement members to continuously ask themselves how they could improve things on multiple levels.

To use a trendy term these days, participatory statistics is deeply transformational in nature, with the potential to alter the complex web of power relations in development co-operation projects for the better. As Holland says, using participatory methods to produce reliable numbers is also an effective way to generate both a qualitative and quantitative description of social change. I am not aware of participatory numbers being used to complement the official statistics measuring progress on the UN Global Goals but I certainly hope it is the case. Do you know?

Photo credit: FAO/Olivier Asselin