This paper, Ecological Validity and the Study of Public Engagement Methods, is not open access yet may be of value to Mesh members who are thinking about approaches to researching engagement or approaches to understanding public opinion through engagement. The main points of the paper are summarised below.
This research paper, is concerned with engagement approaches which uses samples of a population to represent a wider public and their likely attitudes, reception and decisions with respect to a particular area of scientific research. The paper highlights a range of challenges with this which it perceives to be the standardised approach to both public engagement and research into engagement across the sector. A main concern is that experimental samples are drawn and brought into experimental engagement spaces without much consideration for the diversity of ‘public’ and the contextual influences on people’s understanding and decision making processes. The author is also concerned that within existing framings and prefigurings of ‘publics’ we may still miss the role that public engagement can play in understanding public and how they come to opinion and action.
If a great deal of “engagement” has been generating aggregated populations that do not represent actual publics, placing them in artificial surroundings, and directing them toward prescribed modes of deliberation, then only by a feat of extraordinary rhetoric has such research been able to claim that it studies or engages publics. (Gherke p85)
In order to remedy this Gherke asserts that we ought not to rely too much on textual criticism, which can introduce “intentional fallacies” but that it is important to observe how people enact citizenship and in what spaces and through what existing social practices and discourses. Gherke suggest that we might consider using the five criteria for ecological validity (an approach used in cognitive psychology and jury studies). These are: setting, sample, communication medium, amount and type of deliberation and perceived consequences. This would lead to what she terms as ‘organic engagement’ which recognises and is situated within existing public configurations and spaces rather than artificially constructed ones.
Organic engagement begins by developing an understanding of the publics being engaged and adapts methodology to suit that public’s ecology. (Gherke p87)
It is suggested that the most appropriate disciplinary approach for researching this form of engagement is ethnography, quasi-ethnography or multi-sited ethnography (In depth narrative based research approaches that emphasis thick description, site specificity and complexity rather than reductionism and universality).
Like any method, ethnography has limitations, but when one asks about discourses and practices within existing publics, an ethnographic sensibility helps maintain ecological validity while avoiding the intentional fallacy. (Gherke p86)
One justification (amongst the others) for this kind of approach is that it potentially reduces cost. The engagement practitioner does not have to create contrived spaces for engagement, which may be costly but can enter existing conversations and spaces. The burden of cost would then fall toward documentation and analysis of findings rather than event costs themselves.
The paper makes a final comment on the kinds of theory that might be expected to be built from such studies stating that whilst there may be concern that findings could only be considered relevant to the one situation from which it arose there is the potential for multi-sited studies not so as to construct universal claims (which are either impossible or to broad brushed to be useful) but so as to identify ‘middle-range theory’:
“Middle-range theories retain the adaptability required by organic engagement but also have heuristic and practical value beyond specific cases. They allow us to know enough about a phenomenon to assist in confronting problems and have enough truthvalue that they cannot be ignored as anomalies or intentional fallacies.” (Gherke p89)
Readers of Mesh may or may not recognise these criticisms and challenges within their own engagement practice, evaluation and research. Many may already be inclined towards more ethnographic research approaches rather than the experimental designs such as those criticized by Gherke, yet it may be worth thinking about and being equipped to articulate the failings of the overly experimental research design which is perhaps that which a lot of the natural scientists we work alongside might expect.
Paper abstract and full access options here
This work, unless stated otherwise, is licensed under a Creative Commons Attribution 4.0 International License
Please Sign in (or Register) to view further.