Quality Appraisal

main content

Use of Theory in Synthesis

All research, both primary research and research reviews, involve theory. Theories determine the concepts, questions and methods used and the interpretation of any results. In statistical meta analysis, a hypothesis is often being tested. In conceptual synthesis, such as meta ethnography, conceptual understandings are being developed. Narrative synthesis aims to help clarify the theories and concepts involved to further understand an issue.

In addition, 'Realist synthesis 'and 'Critical Interpretative Synthesis' are approaches to synthesis which focus on theory.

Realist synthesis aims to develop and examine evidence concerning mid-range theories from multiple contexts. The aim is to articulate the theory and to check the empirical evidence in support of that theory. This evidence may be across domains/disciplines. For example, a realist Synthesis that aims to test the theories of mentoring as an effective educational strategy might examine evidence from many different areas of education. The quality of evidence depends on its relevance in checking the theory (‘nuggets’ of evidence) rather than some pre ascribed criteria for checking quality of studies.

Critical interpretative synthesis is similar and starts with a review topic and develops the question iteratively through the review as the issues become clearer. It thus uses emergent, iterative approaches as in much qualitative research. In a sense the research question emerges from the process of analysis of the research being considered. The aim is to develop a synthesizing argument.

This second part of Unit 5 can only provide an indication of these different approaches to synthesis. For those interested in learning more on any one of these approaches, please see :

  • Statistical meta analysis: Cochrane Handbook
  • Non quantitative empirical reviews: Popay et al; Roberts and Petticrew chapter 6 section 6.2
  • Meta ethnography: Noblitt and Hare;
  • Britten et al 2002 Mixed methods synthesis:
  • Thomas et al, 2003 Realist synthesis: Pawson, 2006
  • Critical interpretative synthesis: Dixon Woods 2006

Full source details are available in the References section.

Learning Task

Analysing Data From Primary Studies For Synthesis

Imagine the following scenario: you are working with a review team on two separate but related review questions:

  • Is parental involvement in homework effective for supporting children’s learning? 
  • What are parents and children’s views of parental involvement in homework?

Using the research studies listed below, consider the following questions: 

1. What type of evidence would be most appropriate for answering each of the review questions? (Think about the research designs). 

2. Considering the author’s descriptions of their findings, what evidence from these papers is available to answer these questions? How would you synthesise this evidence? Why use this method of synthesis? 

 3. If you had 40 papers, with similar types of evidence, how would you synthesis these findings?

  •  Battle Bailey, L., Silvern, S., Brabham, E., and Ross, M. (2004)‘The Effects of Interactive Reading Homework and Parent Involvement on Children’s Inference Responses, Early Childhood Education Journal, Vol. 32, no. 3: 173-178 
  • Tizard J., Shofield, W., Hewison, J. (1982) ‘Collaboration between Teachers and Parents in Assisting Children’s Reading’, British Journal of Educational Psychology, Vol. 52: 1-15 
  • Warton, P (1998) ‘Australian mothers views about responsibility for homework in Primary School’, Research in Education, No 59

Quality Appraisal

Different critical appraisal methods were used for each of the two syntheses. As mentioned in part 1 of this unit, studies are critically appraised and then 'weighted' according to their quality and relevance to the question. The criteria against which they were 'weighted' are listed below.

Synthesis 1 : quality appraisal criteria for quantitative studies (randomized controlled trials): 

  • Provision of pre- and post- data on outcomes 
  • Provision of data on all outcomes measured 
  • Employment of equivalent control/comparison group 
  • Resulted in ‘high’, ‘medium’ and ‘not sound’/ ‘low’ trials.

Synthesis 2 : quality appraisal criteria for qualitative studies (‘Views’): 

  • Quality of reporting (5 items) 
  • Sufficiency of strategies for reliability/validity (4 items) 
  • Extent to which study findings were rooted in children’s own perspectives (3 items)

The methods and findings of each synthesis differed.

Synthesis 1 (Quantitative)

'Trials'

This considered the effect sizes from the trials and pooled them using statistical meta-analysis on different outcome measures. Heterogeneity of results across studies was explored via sub-group analyses and by qualitative analysis of textual data from trials. The final product of this synthesis was a forest plot.

The forest plot of the effect sizes in shown in Figure 6. This graph shows that there was a small positive impact of health promotion interventions on vegetable and fruit intake.

Figure 6

Synthesis 1 Forest Plot

Synthesis 2 (Qualitative)

'Views Studies'

This synthesis involved several stages of thematic analysis similar to the stages for meta ethnography described earlier.

Stage 1: Thematic analysis of textual data. The review team analysised the ststudy authors’ descriptions of findings to identify Descriptive themes.

There were thirty six descriptive themes identified in the studies and these are grouped and summarized in Figure 7.

Figure Seven

Synthesis 2 Descriptive Themes

Stages 2 and 3

Stage 2: The descriptive themes were examined in light of the review question. The question was ‘What are the barriers and facilitators to healthy eating and what are the recommendations for interventions?’ this was addressed in several ways by developing more detailed aspects of the overall question:

  1. What are children's perceptions of and attitudes towards healthy eating?  
  2. What do children think stops them from eating healthily? 
  3. What do children think helps them to eat healthily? 
  4. What ideas do children have for what could or should be done to promote their healthy eating?

Stage 3: The interrogation of the descriptive themes by the review question involved a process of revising the themes, and developing six Analytical Themes. These represent the final product of the synthesis and are listed in the left hand column of Figure 8. In essence, these are the interpreted meta findings of the research studies combined (synthesized) to answer the review question.

The right hand side column of Figure 8 lists some of the implications of the synthesis of the studies of the views of children and young people for developing ideas about how interventions might be developed to encourage health eating.

Figure Eight

Synthesis 2 Analyatical Themes

Synthesis 3

Combining Quantitative and Qualitative Syntheses

The third part of the review involved combining the two sub syntheses of quantitative trials and qualitative views studies. The starting point was the thematic synthesis of views studies. This was used as the mechanism to combine trials and views studies. The views synthesis findings were juxtaposed against the findings of the trials synthesis.

This analysis was guided by 3 questions:

  1. Which interventions match recommendations derived from children’s views and experiences?
  2. Which recommendations have yet to be addressed by soundly evaluated interventions?
  3. Do those interventions which match recommendations show bigger effect sizes and/or explain heterogeneity?

It was found, for example, that all of the experimental trials evaluated interventions that promoted fruit and vegetables together. This is unfortunate as the views studies suggest that fruit are perceived differently from vegetables by children and young people. health promotion interventions might be more effective if fruit was promoted separately from vegetables. 

Similarly, the views studies suggest that health is not considered a prime driver of food choices by children and young people. However many trials evaluate ‘eat fruit and veg and be healthy’ health promotion messages. These insights were only possible by combining the findings from synthesis 1 and synthesis 2.

Figure 9 shows that interventions that had little or no emphasis on health reported stronger effects, but this results needs to be treated with caution. It is a post hoc examination of the research data and may be due to chance,. We need to undertake new studies to test the hypothesis that attempts to increase healthy eating are more effective with children and young people if they do not focus on health messages.

Figure Nine

Synthesis 3 Sub Group Analysis

The overall message is that mixed methods reviews allows you to study more complex issues than can be studied by individual reviews on their own.