It is an exciting time to be part of the world of measurement and evaluation. Having attended three conferences this autumn, it is clear that those with a critique of the traditional ways of doing things are finding a voice, and being given a platform. In the wake of Black Lives Matter everyone seems more open to looking deeper into the implicit assumptions that we make about each other, and along with that, into the power dynamics of measurement and evaluation.
NPC ignites was one of these events and it was the session “Rebalancing data for the 21st century” that really captured my attention. Jara Dean Coffey, Director of the Equitable Evaluation Initiative presented a five-year plan she is leading to change the way funders in the United States think about evaluation. Bonnie Chui of The Social Investment Consultancy is leading an initiative bringing together people of colour working in evaluation. Here were some of their key messages:
Co-create knowledge rather than extract data
Traditional approaches to the evaluation involve experts collecting data and taking it away to analyse and draw conclusions. The subjects of the evaluation are passive in the process. Bonnie described this as like using research as tool of ‘command and control’. Jara argued, like several others I have heard this year, that we learn more when knowledge is co-created – researcher and subject bringing together their very different expertise to build a more complete and informed picture. This is one way to challenge the power relationships in evaluation and promote greater equity. The Outcomes Star’s collaborative approach to measurement brings these ideas alive in day-to-day service delivery.
What is the Outcomes Star
Get comfortable with complexity
“We need to let go of causality and be OK with contribution”
Jara made the case that although funders who commission evaluations want certainty and yes/no answers, the complex reality of service provision can’t be reduced to a few numbers. Funders and evaluators need to embrace the complexity that comes from working in open systems where it isn’t possible to control all the variables and come up with answers that are always true no matter what the context. Bonnie also made the point that top down funder-driven monitoring and evaluation frameworks can perpetuate power imbalances. It is difficult for funded organisations to raise these issues because of their dependence on the funders so it is important that evaluators use their influence. This very much echoes points we have been raising at Triangle for some time. Data is helpful but must be interpreted in context. The numbers help to focus our questions rather than providing definitive yes/no answers.
De-colonise evidence
Bonnie Chui argued that we need to ‘decolonise’ evidence and ensure that people of colour are both reached by research and represented in the research and evaluation community. Jara is promoting multi-cultural validity alongside statistical validity, a point which chimes with issues Triangle has raised about moving beyond traditional formulations of what is a ‘good’ tool (keep an eye on our homepage for a blog on this coming out soon).
Both presenters made the case that evaluation is a human process. Those doing the evaluation have to do their own personal work to understand their own implicit biases as well as those that are hardwired into the context in which they are working. The biases identified were racial ones as well as foundational ideas such as the preference for doing over being and our belief in scarcity rather than abundance.
I found it very inspiring to hear an analysis connecting up racism, core orientations towards life and the way services are valued and measured. I can’t do it all justice here, so I recommend that you take a look at the recording of the session.
***
Triangle is the social enterprise behind the Outcomes Star™. Triangle exists to help service providers transform lives by creating engaging tools and promoting enabling approaches. To talk to Joy MacKeith or another member of the Triangle team, or for any other information, please email info@triangleconsulting.co.uk.