Public and community engagement initiatives take place in settings with multiple stakeholders, contextual factors that may have an unforeseen influence, and dynamic circumstances which may lead to unexpected change. Monitoring and evaluation of engagement work in these contexts is complex and there are many aspects to consider to ensure it is useful and meaningful.
Caption: Robin Vincent, Evaluation Consultant, UK gives an introduction to evaluating engagement with particular focus on the Theory of Change approach and how complexity in engagement can be navigated in evaluation.
Download the Presentation [PDF 991 KB]
Accountability vs. Learning and the Theory of Change Approach
Accountability: Demonstrating a project's impact to funders, communities, and other stakeholders.
In complex and dynamic environments, there are limits to what can be achieved in terms of demonstrating impact. There is a question of how honest this kind of evaluation can be when indicators are decided in advance but projects often need to adapt as they develop.
Learning: Reflecting on practice for improvement.
Looking at the why and how rather than the impact is useful to project development but it does not allow us to show whether an impact was achieved.
A Theory of Change approach can allow for both accountability and learning. It enables users to show clarity about what they expect to happen and lay out any assumptions around how they expect that change to occur and what contribution the project will make. This puts the user in a much better position to learn about the project as it develops whilst also getting a sense of whether it has had an impact. It is an appealing approach as it is both an emergent process and has the planning aspects of a traditional log frame approach. Robin Vincent gives more detail about the approach in the presentation above and more resources are available in the Mesh Theory of Change category including a full introduction to the approach.
Caption: Dorcas Kamuya, KEMRI-Wellcome Trust Research Programme (KWTRP), Kenya, gives a detailed account of how and why KWTRP have evaluated their individual engagement activities and overarching programme of engagement to understand the impact it is having on knowledge, understanding and relationships.
Download the Presentation [PDF 3MB]
The Risks of Trial and Error
“…it seems curious that we invest millions of dollars in product development, clinical training, design and building of facilities, etc., but often leave vital processes of community engagement largely to trial and error."
Peter Newman, The Lancet, 2008
There is a certain amount that can be learnt from trial and error. Many fields, for example engineering are built on a foundation of trialling, identifying error, and gaining insights from these errors that lead to success. It is also inevitable that with each new context bringing its own complexity there will be a degree of trial and error with each new project. However, to rely on trial and error risks overcrowding a community, creating fatigue and wasting resource as too many initiatives are trialled. To overcome this, it is important to think carefully about how and why a particular engagement initiative is suitable and, wherever possible, to pilot ideas and learn from them before upscaling.
To browse a suite of resources on Mesh about evaluating engagement visit the Mesh Evaluation Map.
The content on this page forms part of the online report for the 2017 International Engagement Workshop "It's Complicated: navigating scientific complexity in public and community engagement". To read more visit the introduction to engagement theme from the workshop or to access the rest of the report including video presentations, discussion summaries, and tools, visit the workshop page.
This work, unless stated otherwise, is licensed under a Creative Commons Attribution 4.0 International License
Please Sign in (or Register) to view further.