Case Study Overview 

This presentation outlined an evaluation approach designed by the Oxford University Clinical Research Unit in Vietnam (OCRU-VN) to appraise their Science Theatre Programme, part of their wider schools programme, which sits within a broad programme of engagement work.  Mary Chambers is director of the OUCRU-Vietnam Engagement Programme and Van Anh has been coordinating the schools component over the past six years. The team are now at a point where they hope to reflect on the design and impact of activities with a view to bringing learning into the next iteration of work.

The Engagement Interventions

Science Theatre Activities

The team develop theatre performances which haveso far centred on the themes of infectious disease and dengue fever (containing messages about the symptoms, spread and treatment). These scripts are developed along with doctors and researchers associated with OUCRU-Vietnam and are performed by theatre practitioners within rural schools and communities such as Hà Nam province outside of Hanoi; provinces in the South Vietnam around Ho Chi Minh city; and parts of the Mekong Delta. The shows last around 15 minutes with an additional 15 minutes of introductory games. Children who attend are given brochures with further illustrated information and a quick game for them to play and share with their friends or parents. The play is also broadcast on national television and DVDs of the play are given to health centres, primary schools and hospitals in the provinces to show in the waiting rooms.

The team deliver training workshops in communications skills for school teachers and health staff.

Aims and Objectives

The science theatre project has three main Aims:

• To engage children (6-12 years old) with science
• To raise community awareness (children’s in particular) especially in rural and remote areas were there is least information about infectious disease and low education levels.
• Capacity building for school health staff, to know better how to care for children’s health and for teachers to develop their knowledge and teaching methods.

Objectives

• Raising knowledge
• Enhancing children’s skills around personal health care
• Increased appreciation of science through the activities and through the teachers being equipped with more ‘participatory’ methods in their teaching.
• Improve the general health of children
• Lastly we can aim for that they have a good time and that they somehow associate science with enjoyment and feel positively towards it.

There is also an unexpressed top-level objective to reduce the incidence of diarrhoea, yet the team recognised that to focus on this would move the emphasis of the work towards a public health intervention and research rather than engagement for which there will be . as it would demand a lot of the funds.  It was suggested that she might think of this as above their ‘accountability ceiling’ (outcomes beyond those which the intervention can be expected to have the power to change or claim credit)

Partnerships

In this project OUCRU-Vietnam worked with the department for health and the department for education. These groups help OUCRU access the locations where disease is highest. This is very important in Vietnam where the government must oversee and approve any community-based projects.

There is perhaps another high-level outcome to the work which links to these partnerships. The team were keen to generate favourable relationships with government partners as these may help pave the way for future research. Whilst this was not a direct objective of the programme, it isan unintended outcome worth tracking as it may present problems as well as benefits. For example, does it make it difficult for a government to say no to other things that the team does because they want the schools programme?  

It worth noting that in this context, although government departments are bought into the project it can not be assumed that this might influence government policy due to the hierarchical nature of governance in Vietnam. In other countries, for example Kenya, policy influence would be something that could be expected of public and community engagement, but the same is not true for Vietnam.

Feedback

So far feedback on the interventions from teachers, pupils and government partners has been positive. Teachers have shown special enthusiasm as they often struggle to teach science in engaging ways. Government collaborators are thinking about bringing these new methods into their own health communications repertoire and are often keen to see further iterations of the project should OUCRU-VN secure the funds.

Next Steps

At the time of writing, the team are developing further activities and are keen to strengthen their evaluation approaches.  They plan to produce work on general health, nutrition, oral hygiene and hand washing in a hope to promote healthy behaviours (things which they believe  children  have control of within their own lives).

The team plan to deliver teacher training again and to conduct science festivals in the provinces, giving teachers who have been through training an opportunity to use what they have learnt with their classes. 

So far the team’s evaluation approach has tended to centre on the project objectives and activities conducted to meet those objectives. However, the OUCRU-Vietnam team are interested in thinking about using realist evaluation and theory of change approaches to build upon this.

Although OUCRU-Vietnam’s research looks like it is very simple and linear (designing activities to meet specific objectives) the team are in fact already working with ‘theories of change’ (ToCs) in that they believe that there certain activities will result in certain outcomes within their basic project assumptions.  [https://mesh.tghn.org/articles/theory-change-mesh-introduction/] in an implicit way.

For example:

1- The hope that if the teachers improve their teaching methods then young people will increase enjoyment of science.

2-That in building capacity of health care workers to communicate health information they will share health messages more widely which will lead to increased knowledge within the community, leading to increased community understanding, leading to change in behaviour, leading to increased health.

These assumptions could act as a starting point for a theory of change, which could be adapted as more is learnt about the activities and outcomes. Also, in the spirit of the ‘realist evaluation approach’ which encourages interventions to consider ‘what works, how, in which conditions and for whom’ [link realist pages on mesh] the team could think about the mechanisms, the generative force behind the changes that they expect to see.  For example,. rather than assume that increased knowledge automatically leads to behaviour change, the team may find, through comparing participants and contexts, that change is instead driven by peer pressure, fear of consequences or perhaps just because they are more confident in their decisions. Exploring the mechanisms would test assumptions that the team know they are making as well as highlighting overlooked assumptions such as whether or not people have access to water and soap, toothbrushes and toothpaste. Such mechanisms for change might be identified using questionnaire and interview data (current methods used by the team- see below) and by looking for differences between pupils. They might also be able to add depth in a realist fashion through comparing shining examples with less shiny ones and trying to understand what was different contextually.

Evaluation Methods

Key methods used in evaluation so far have been:

Interviews

The team interviewed students and their teachers and asked about whether they see the information within the performances as applicable to their daily lives.

Collaborators such as the government organisations will also be interviewed.

Questionnaires

Questionnaires were disseminated to children before and after the performances. This is to gauge if there has been any increase in children’s knowledge. The first time that this was conducted it looked like children’s understanding had actually gone down and so the team revisited their questions and realised that they had made an error in asking children about details of the show itself rather than of the science within it.

The team recognise that their evaluation methods might not be able to indicate long-term knowledge acquisition nor actual change in behaviour.

The plan for next steps in evaluation is to:

1-    Continue with the questionnaire and with interviews before and after the performances.

2-    To observe teachers in the classroom to assess if they were able to pick up the skills delivered in the training             sessions.

The team have never used participant observation before and are concerned that the process of observation will not be neutral as it is likely to impact on the teacher’s activity. This could mean that the evaluation methodology  needs to be considered and traked aspart of the intervention itself, as well as being purely an evaluation tool. Fostering and building on partnerships with other teaching institutions such as a university within Vietnam may also minimise teacher’s fear of being observed

3-   To revisit five months later and interview people who were not at the shows but were in the community and other children who were not in that school to see if can  capture a ripple out effect.

4-    To conduct surveys of non-participants such as . parents in whether they see a change in their children’s skills or behaviour.

5-    To invite children to self-report on their hand washing and teeth brushing in a tick box exercise over 1-2 months.

Although there will probably be biases in children’s self reporting of hand washing and tooth brushing behaviour the results would be an interesting way of bringing people into the conversation about research and data reliability by posing questions to them such as: ‘Do you really think that 100% of children brush their teeth everyday?’.

6-    To do a follow up activity in ten schools in two districts of the province six months after the initial intervention. This will be compared with schools where there was no activity to see how no activities impact with other schools.

It was suggested that the team could also consider conducting a literature review exploring the extensive academic literature spanning 50-60 years on the impact of inset training on teaching methods. And that perhaps there is scope to partner with a university to tap into this knowledge. 

Key Challanges 

One challenge of this project is achieving teacher buy in and input. It is something that workshop participant Vicky Nembaware is concerned with in her work within H3Africa . As Vietnam is a highly hierarchical society the routes to achieving teacher commitment may differ to other contexts. By working through the Vietnamese Department for Education, school’s participation is guaranteed. This is not to say that there is enthusiastic buy-in but it does mean that there is guaranteed participation and more enthusiastic buy-in might follow. 

The relative cost of the evaluation in proportion to the programme was thought to be a challenge.  Just because we identify things, which are useful to measure, it doesn’t mean we have to measure everything in this instance it was suggested that the department of health might already be monitoring things, which are useful to the programme. It was suggested that if the team could hook into such other monitoring systems then the team could think about which level of question is really relevant to them and where their ceiling of accountability lies whilst staying aware of higher level changes which if thy are not able to claim responsibility for they may find valuable in factoring in the context in which they are working. 

This resource resulted from the March 2017 Mesh Evaluation workshop. For more information and links to other resources that emerged from the workshop (which will be built upon over time) visit the workshop page.

For a comprehensive summary of Mesh's evaluation resources, and to learn how to navigate them, visit theMesh evaluation page

This work, unless stated otherwise, is licensed under a Creative Commons Attribution 4.0 International License