Realist Evaluation - A Mesh Introduction

Introduction

Realist evaluation is a valuable method for addressing the complexity of social programmes. It seeks to understand: What works? For whom? In what circumstances? And why? Rather than providing one-off verdicts on programme success or failure (Pawson and Tilley 2007).

Realist evaluation seeks to clarify the programme theories behind the way change is expected to happen in an intervention and uses a range of evidence to test whether these theories accurately account for the process of change. It is multi-method and draws on quantitative, qualitative, comparative and narrative evidence, as well as grey literature and the insights of programme staff. In each case it looks to illuminate and sharpen the understanding of how change happened in the programme and why the programme works in some circumstances and not others.

Realist evaluators consider social interventions always to be “complex systems thrust amidst complex systems” (Pawson 2006: 25). It is not the programmes themselves working, but rather the way people respond to the resources, ideas and practices that those programmes introduce (programme ‘mechanisms’ in realist jargon), that create programme outcomes. In this way, a programme will work differently for different people, and there is no expectation that any particular intervention will be the same when it plays out in different contexts.

At the same time, realist evaluation seeks to build a cumulative understanding of how programmes play out differently in different contexts, to produce patterns of outcomes. This picture can then usefully inform policy and subsequent action and programming in any new context. ‘Realist synthesis’ brings the same principles to the systematic review of research and evaluation studies.

Caption: This video gives a short overview (14 minutes) of realist evaluation in which Ray Pawson introduces the basics of Realist evaluation. There is also a series of four introductory videos which expand on this:

• Realist evaluation – introduction Part 1 (7 minutes) introduces the key elements of realist evaluation.
• Realist evaluation – introduction Part 2 (9 minutes) looks at how realist evaluation helps to address complexity in social programmes. 
• Realist evaluation – introduction Part 3 (8 minutes) looks at how the method makes use of evidence and the value of a multi-method approach 
• Realist evaluation – introduction Part 4 (7.5 minutes) looks at how evaluation can better contribute to cumulative learning and employ ‘scientific’ methods 

Video Credit: Ray Pawson and Robin Vincent

Further Reading

1. A Brief introduction to Realist Evaluation (2 pages) by Dr Gill Westhorp. An introduction to Realist evaluation, what makes it distinctive, and key aspects of the approach. 
2. Realist Impact evaluation: an introduction (12 pages) by Dr Gill Westhorpe. A more detailed but accessible introduction to Realist Evaluation which highlights distinctive features of the approach, key concepts and when and where the approach is most appropriate. It provides a range of illustrative examples.

3. Realist Evaluation and Realist synthesis presentation 

   (29 slides) by Ray Pawson, 2006. The first half of this presentation focuses on Realist Evaluation, outlines basic principles and provides practical examples. The second half covers the related Realist approach to systematic review known as Realist Synthesis and how to draw out cumulative learning from evaluations.
4. Realistic Evaluation: an overview (13 pages). A conference presentation introduction by Nick Tilley that is more about the rationale of the approach rather than the practicalities. The presentation highlights the roots of the approach in the philosophies of Popper and Campbell and their concerns to see gradual beneficial social change. It gives a critique of the simplistic application of traditional experimental approaches to social programmes and shows how realistic evaluation addresses the complex causality in social processes and programmes.
5. (Book) Realistic Evaluation (London: Sage,1997). This influential introductory text written by Ray Pawson and Nick Tilley gives a critique of existing evaluation practice. It covers the basic principles and approach of realist evaluation, including for realist interviews, and outlines the potential for evaluation to support cumulative learning.
6. (Book) Evidence-Based Policy: a realistic perspective, (London: Sage, Ray Pawson, 2006). This book looks at the companion approach of Realist Synthesis, which uses realist principles to synthesise evidence to inform policy. It provides a critique of dominant approaches to systematic review and meta-analysis for the way they tend to average out and ultimately obscure the differential effects of context. It includes a range of illustrative examples of how evidence can be gathered and synthesised to make context sensitive recommendations (such as schemes for youth mentoring and crime prevention).
7. (Book) The Science of Evaluation: a realist manifesto, (London: Sage, Ray Pawson 2013). This book makes the case for a science of evaluation that can build a coherent body of learning and evidence on social programmes. It summarises key principles of critical realism that underpin the method and includes a critical review of a range of recent published realist evaluations to sharpen the methodological debate and improve practice. It includes a detailed account of an application of the method to inform policy on banning smoking in cars carrying children (also available as a paper – see below Wong et al.)
8. (Book) Realist Evaluation (2004) (Chapter - 36 pages) Ray Pawson and Nick Tilley.
9. An email Discussion list on Realist Evaluation and Realist Synthesis, including archived discussions. 
10. Online Dossier: KIT Royal Tropical Institute Netherlands Dossier on ‘Realist Enquiry’. This includes a short introduction and a range of published resources and examples of applying realist evaluation.
11. The Realist Hive.
12. Theory-driven inquiry for health systems research.
13. Case Study: How Do You Modernize a Health Service: a Realist Evaluation of Whole-Scale Transformation in London. The Milbank Quarterly, Vol. 87, No. 2, 2009 (pp. 391–416) by Trisha Greenhalgh. This paper is about applying realist evaluation to understand a range of factors supporting change across four large health care organisations and their delivery of three areas of health services. It also reviews some of the challenges of applying realist methods in this early case study. 
14. Case Study: Policy guidance on threats to legislative interventions in public health: a realist synthesis. (Wong, G., Pawson, R., and Owen, L, in BMC Public Health 2011, 11: 222). This paper explores a detailed case study of legislation to ban smoking in cars carrying children, and how the authors drew on a range of different bodies of relevant evidence to assess the likelihood such legislation would be feasible and likely to succeed. 

More resources on Realist Evaluation can be found in the Mesh Realist Evaluation resources category. To see how this resource fits in with the rest of Mesh's evaluation resources, and to learn how to navigate them, visit the Mesh evaluation page. 

This resource was developed by Robin Vincent as part of supporting the Wellcome Trust linked community of practice on evaluation of public and community engagement and was originally hosted on the eMOPs website.

This work is licensed under a Creative Commons Attribution 4.0 International License