Evaluation is one way to systematically assess an initiative to establish impact and improve practice using evidence-based insights. Results from an evaluation of a PPIE initiative will assist with reporting to leadership, funders and collaborators as well as improving how activities are conducted, while evaluating the impact of PPIE activities will advance the practice and science of engagement. Both types are amenable to publication in scientific journals.
Evaluating PPIE initiatives & activities
An evaluation of your PPIE initiative should be built-in from the start in order to enable you to collect necessary information throughout your project. Planning an evaluation in the midst of your activities or at the end is cumbersome and you’ll likely find you have forgotten half the things you have done. The evaluation can be informal, with a simple report on progress towards objectives, or more formal and comprehensive with multiple evaluation activities (see an example protocol). This depends on your ultimate goal, whether that is to compile a report, identify lessons learned, assess impact or conduct an evaluation study with comparable findings.
Some considerations:
● Timing: Consider whether to perform an evaluation periodically, in the middle or at the end, and what an evaluation will involve at each stage.
● Records: Keep records of activities and impact throughout the lifecycle of your project, as it is easier to capture this during rather than at the end of your project. Appendix P is a Master Log where relevant activities can be recorded and data later used for evaluation, for example, the number of meetings, attendance rates, and reach of social media posts and events.
● Methods: Evaluations can employ many different methods depending on the goals of the evaluation, such as surveys, interviews, focus groups, written reflections, document analysis and quantification of outcomes.
● Participants: You may be interested in the experiences and perspectives of PPIE members or coordinators, but could also consider exploring perceptions of senior management or other collaborators whose work may have been influenced by the PPIE initiative (e.g. investigator of a study that was reviewed by the PPIE group).
● Evaluators: Ideally, the evaluation should be co-led and co-written with your patient and public partners.
Evaluating impact
The evidence on the impact of PPIE on the design, conduct and reach of research is limited, with even fewer studies attempting to measure the impact of PPIE on clinical outcomes. Part of the reason for this is that the impact of PPIE on research or clinical outcomes is indirect with multiple confounders and the desired impact of PPIE is not easily measured (e.g. more patient-centred research processes). However, often engagement practitioners fail to think about evaluation and measurement of impact.
Progress can be made by:
i) dedicating time to think through a theory of change for how the PPIE initiative and activities is expected to lead to specific changes; and
ii) designing high-quality studies to understand the impact of PPIE initiatives.
Some research ideas include:
● Study within a trial comparing participant understanding of trial-related information between consent materials and procedures created by researchers to those co-created with patient and public partners.
● Comparison of recruitment rates before and after the implementation of changes to recruitment methods and materials recommended by patient and public partners.
● A narrative case study on the evolution of a clinical trial from a research question to the recruitment of the first participant outlining changes made based on patient and public partner input.