This article outlines key discussion points from a session on Realist Evaluation at the Mesh Evaluation Workshop 2017. Initial presentations were delivered by Robin Vincent, Evaluation Consultant, and Emma Richardson from the Centre for Ethical, Cultural and Social Risk ad St. Michael’s Hospital in Toronto and part time faculty member of McMaster University in Toronto before the workshop attendees discussed the relevance of realist evaluation for their work. This case study weaves together the presentation and discussion from the event in an attempt to go beyond presentation of the theory towards thinking about its application in the context of Community Engagement in Global Health Research.

Introduction

Public and community engagement initiatives are complex interventions. They take place in contexts which consist of a complexity of actors, goals and other factors (social, historical, political etc.) These factors can have unpredictable influences on the outcomes of an engagement intervention and make it hard for any societal change to be strictly attributable to any one project or action.

Realist evaluation is an approach which has been growing in popularity in the last decade or so and may prove a useful approach for evaluation of engagement as it helps to understand complex social interventions and their likely different outcomes across different contexts whilst retaining a sight on what the project aims to achieve.

A realist approach to evaluation permits us to revise our understandings and assumptions of how we might affect change as a project unfolds within a dynamic and fluid context. In this way it supports learning and responsive project management.

The Philosophy and Context of Realist Evaluation

Realist evaluation is part of the family of ‘theory driven’ evaluation used in a range of health related interventions, in the global North and South. It is concerned with clarifying how change happens in projects and is often used alongside the increasingly popular ‘theory of change’ approach given that both commit to making the ‘programme theory’ underpinning an intervention and how it is expected to work explicit. When using realist evaluation, the theory of change created is underpinned by a rigorous philosophy (Critical Realism) and accompanying assumptions about the nature of causality in social systems:

• you can never separate a project from its context;
• there are multiple stakeholders in a project each with their own interpretation of the ideas held within the project’s ‘theory of change’;
• meanings, concepts and beliefs are just as ‘real’ as physical processes, and they have tangible effects even if they are not directly observable and need to be inferred from evidence;
• theories of change are always provisional and partial, but evidence can be used to improve them and decide between them.

Realist evaluation, despite having some of it’s own jargon, can provide some useful ways of navigating real world complexity and addressing the challenges of evaluating community engagement.

In addition to original texts on the subject there is a growing range of more accessible introductions and resources, including resources available on Mesh such as filmed interviews which Robin Vincent conducted with one of the pioneers of realist approaches to evaluation, Pawson: Available here

What is Realist Evaluation? 

Realist Evaluation is based on the premise that:

• Engagement activities are complex social interventions.
• Interventions consist of chains of steps or processes.
• These chains or steps are often not linear.
• Interventions are embedded in social systems.
• Interventions are prone to modification and social systems are dynamic.
• They are open systems and change through learning.

And is driven by three questions:

• What works for whom?
• Under what circumstances?
• In what respects?

In critical evaluation, intervention strategies are thought of as “theories”; there is a theory behind the strategy being implemented, which assumes that change will happen in a particular way. It is for this reason that ‘Theories of change’ are an associated tool in realist evaluation approaches as they clearly lay out assumptions that have been made about how interventions will lead to outcomes. 

What critical realism endeavours to do is to capture the associations between a set of conditions and a consequence; to define the active ingredient that leads from an intervention to a given outcome. It does so taking into account that this active ingredient will not be one condition but a set of conditions, interacting in a complex way in a complex context. 

In other words, through critical evaluation one might get closer to understanding the set of conditions which drive any given project outcome (expected and unexpected), and the interactions between these conditions. Explaining why the same intervention may lead to a given outcome under one set of conditions, or in one context, but not in another.  In realist evaluation jargon the active ingredient created by the interplay of conditions and context is  known as the “Mechanism” behind the change (which is different to the strategy). Critical realism is characterised as:

“generative ontology, whereby associations between phenomena come about as a consequence of hidden mechanisms enacted under certain circumstances.  A useful metaphor here is that of a candle, which though causally linked with production of a flame if lit, requires a certain association of circumstances (dry match with the correct chemical composition of tip, dry wick, presence of oxygen, lack of high wind) that must all be present to reach the expected outcome.  This view postulates that there are ‘real’ and dynamic underlying connections between phenomena that may result in causal links under certain conditions at certain times (Pawson and Tilly, 1997). Phenomena are therefore deemed to have emergent properties, that is, characteristics that under the right conditions may result in a certain outcome (Pawson and Tilly, 1997).” (Quote from Clark, 2007).

The Value of Tacit Knowledge and the “Fallible Expert”

The realist approach sees project stakeholders as ‘fallible experts’ whose insider information on the above characteristics needs to be documented, formalised and tested. Therefore realist evaluation does not see it as compromising for those who are part of the programme itself to be deeply involved in the evaluation. 

It is sometimes challenging to understand that the mechanism was not the project activity itself, but the way it triggered responses in some people. One of the reasons why the fallible experts’ are important in realist evaluation is that they may have the tacit knowledge to be able to identify and conceptualise the mechanisms. Things are even more complex when there are multiple competing mechanisms involved – such as in examples of safe sex. There are lots of reasons why people don’t have safe sex to do with economic pressures or to do with coercion, which can be competing mechanisms. Sometimes what was part of a mechanism can become part of context over time, such as in the example of CCTV in the UK, which is now ubiquitous.

Examples of Realist Evaluation

Example 1

The Project: A school’s project in which pupils met with scientific researchers, resulting in some pupils reporting an increased interest of pursuing careers in research.

The Strategy: To bring scientific researchers together with pupils. 

The Objectives: There may have been various objectives within this strategy held by different stakeholders e.g. To improve science education in schools and consequently student’s grades; to increase likelihood of people within the community feeling favourably towards medical research; to better understand attitudes and understanding of science and medical research within the population represented by school pupils etc.

The Theory of Change might have been: That knowing more about science necessarily leads to an increased knowledge of science. 

The Mechanism might have been: An individual child, meeting a researcher of a similar identity group to themself enabling them to imagine themselves in a scientific career where they had not imagined someone like them being employed before the interaction. Within realist evaluation however we would recognise that the context of each pupil was unique and that some pupils may not have been influenced in the same way as others because the same mechanism can have the different expression in different contexts. For example; if the visiting researchers were predominantly of a different class or gender to particular pupil perhaps the pupil will not have felt the same degree of identification with the visitors and therefore have been less affected by the interaction than another. Realist evaluation therefore invites the evaluator to examine intersecting contextual factors and encourages a degree of criticality and granularity to the analysis of any outcome. And, ultimately, the theory of change is adapted in the light of such findings.  

Example 2

The Project: Tissue Culture Bananas [https://www.ncbi.nlm.nih.gov/pubmed/28288608]
This project aimed to promote a particular bio-technological innovation to increase commercial banana production amongst smallholder farmers in Kenya.

It was expected that the farmers would resist this new cash crop but through a process of exposure visits, in which some farmers were taken to see other farmers who were growing the crop, they found that they experienced more uptake and less resistance than expected.  

“There were all kinds of reasons to believe farmers would not have trust in the intervention they had been screwed over by outside institutions too much.” (Lavery)

The programme team had to ask themselves: what is the best way to introduce the technology, and all the accompanying practices and opportunities. They saw this as a challenge in community engagement.

Research: A research team led by Jim Lavery and Sunita Bandewar conducted a retrospective qualitative case study informed by grounded theory to explore:

• The CE strategies used to introduce the banana and accompanying agricultural practices
• The uptake from smallholder farmers
• The nature of the relationships between the lead organisation (Africa Harvest Biotech Foundation International), farmers and other stakeholders.

The team found six specific features of community engagement in this project that were critical to its effectiveness:

• Adopting an empirical, "evidence-based" approach
• Building on existing social networks
• Facilitating farmer-to-farmer engagement
• Focusing engagement on farmer groups
• Strengthening relationships of trust through collaborative experiential learning
• Helping farmers to "learn the marketing game"

The Mechanism? Jim and his team tried to reflect on these factors and the mechanisms which might have been at work behind the intervention-outcome relationship.

“It was something to do with lowering perception of risk. It was normalisation or lowering risk thresholds” (Lavery)

What the example above demonstrates is a Context-Mechanism-Outcome Chain. That in the context where mistrust can be expected, lowering risk perceptions can lead to the outcome of farmer buy in to the new cash crop.

Often realists look for these context-mechanism-outcome configurations; ‘In this context you will have this outcome; in this you will have another and in another you will have this.’ Having a transparency around how you make these connections and the decisions you base on these is key and usually this happens in teams. This minimises the risk of becoming overly focused on a single context-mechanism-outcome chain at the expense of others. It is important to involve as broad a range of perspectives in the process as possible.

Challenges of Realist Evaluation

There is no doubt value in critically thinking through and re-examining ones assumptions, but there is also a challenge of time, and the danger of falling down the rabbit hole. There is a need to be proportionate to the investment in the engagement project itself. Realist evaluation, like other evaluation methods has to be pragmatic and the principles and approach can be used to help focus an evaluation. It is important to focus on what matters – what would you be able to change in practice and respond to? – and focus your energies there.

Realist Synthesis: Using Realist Analysis to Look for Patterns

There is a particular value in realist approaches for drawing lessons from across programme contexts (realist synthesis). Where, realist evaluation focuses on evidence for the evaluation of a particular project drawing, which is used to develop a theory of change, ‘realist synthesis’ looks across a broad range of disciplines and topics for patterns of how context affects outcomes.

“ By taking programme theory as its unit of analysis, realist review has the potential to maximize learning across policy, disciplinary and organisational boundaries”.
Pawson et al (2005)

This is unlike the analogous ‘systematic review’ which also tries to draw lessons from across projects and sites but which tends to stay within a particular silo, e.g. malaria, and neglects contextual distinctions across projects. 

An illustrative example of realist synthesis is if you are designing a policy to introduce seat belt use in cars. To do this, you might look across the literature and think about what prompts people to use their seat belts; such as the fear of getting fined if they don’t. In realist synthesis you will then look to see if that mechanism (the fear of the fine) works in other areas such as smoking in a public place. You can then bring that knowledge to bear on the issue of seatbelts; working across disciplines and themes rather than staying in one silo.

Definitions and Glossary of Terms

Realist analysis does suffer from being reliant on some jargon, and key terms are still disputed, even in the field:

Context: Context often pertains to the “backdrop” of programmes and research. For example, in our work it pertains to the conditions connected to the development of research partnerships. As these conditions change over time, the context may reflect aspects of those changes while the programme is implemented. Examples of context include cultural norms and history of the community in which a programme is implemented, the nature and scope of existing social networks, or built programme infrastructure. They can also be trust-building processes, geographic location effects, funding sources, opportunities, or constraints. Context can be broadly understood as any condition that triggers and/or modifies the behaviour of a mechanism. 

[In engagement projects we might be interested also in the characteristics of project ‘participants’, which means they react to the intervention differently as in the example above.]

Mechanism: A mechanism is the generative force that leads to outcomes. It often but not always denotes the reasoning (cognitive or emotional) of the various actors in relation to the work, challenges, and successes of the partnership. Mechanisms are linked to, but not synonymous with, the programme’s strategies (e.g., a strategy may be a rational plan, but a mechanism involves the participants’ display of responses to the availability of incentives or other resources). Identifying the mechanisms advances the synthesis beyond describing “what happened” to theorizing “why it happened, for whom, and under what circumstances.” 

Outcomes: Outcomes are either intended or unintended and can be proximal, intermediate, or final. Examples of PR [participatory research] outcomes are greater empowerment, participation, enrolment, education, knowledge, development of programme infrastructure, and enhanced research processes. Examples of intervention outcomes are improved health status, more use of health services, and enhanced research results.

Taken From Jagosh et al (2012)

Resources

Introduction to Realist Evaluation on Mesh (with links to further resources)

The Science of Evaluation: A Realist Manifesto

Blamey & Mackenzie (2007). Theories of Change and Realistic Evaluation Peas in a Pod or Apples and Oranges? Evaluation, 13, 439-455. Link to PDF 

Introduction to Theory of Change: Mesh

Theory of Change Wikipedia: useful to find examples of organisations who have used the approach 

Evans et al (2014) Public involvement in research: assessing impact through a realist evaluation. Health Services and Delivery Research, No. 2.36. Link to Article Online 

Jagosh et al (2012) Uncovering the Benefits of Participatory Research: Implications of a Realist Review for Health Research and Practice. The Milbank Quarterly, Vol. 90, No. 2, (pp. 311–346) Link to Article Online
 

This resource resulted from the March 2017 Mesh Evaluation workshop. For more information and links to other resources that emerged from the workshop (which will be built upon over time) visit the workshop page.

For a comprehensive summary of Mesh's evaluation resources, and to learn how to navigate them, visit theMesh evaluation page

This work, unless stated otherwise, is licensed under a Creative Commons Attribution 4.0 International License