Reliability, validity, objectivity

main content

Reliability, Validity, Objectivity

Considerations of reliability and objectivity are important when it comes to observations. Some of these have already been discussed in the previous sections. For example, the notion that observations can provide us with objective measurements of human behaviour (when we are talking about social sciences research) should be looked at in relation to how the observations units are defined and operationalised, but also how they are measured and recorded. Highly structured observations in particular are linked with the assumption that the observed behaviour provides evidence to a latent construct, which can sometimes be problematic. If we are observing non-verbal behaviours, we might make inferences about them that will influence our observation schedule. Think about a researcher recording positive non-verbal interactions between two individuals. Would a smile be considered one of them, and if yes, shouldn’t the researchers differentiate between a friendly smile and a rather sarcastic one? Thus, there are issues about validity that the researcher needs to take into account. In addition there are issues related to the observer and his/her role, and even more so when more than one observer is involved. Consider the following:

  • Expectancy Effects: knowing the hypothesis and aims of the research could potentially influence the observations made and recorded (as well as participant behaviour). Double-blinded observations would help to that end. (where neither the participant nor the researcher are fully aware of the research hypotheses; in the case of experiments, the participants are “blind” to the treatment, while the researcher is also blind to the allocation of participants in different groups)  
  • Observer omissions (failure to record a behaviour that is actually specified in the observational schedule) can occur due to personal bias but also when behaviours are rather infrequent or rare and thus may go unnoticed. 
  • Selective attention and selective data entry when it comes observing and to recording the behaviour: like in other stages of the research, the researchers needs to keep in mind that sometimes their own experiences, expectations or judgements could influence their perception of what they are witnessing during an observation and affect what is being recorded. 
  • Faulty memory, attention deficits of the observer, and selective memory: sometimes even a second of being distracted can result in missing out on a whole observation window. By recording our observations during or immediately after the event we observe we can minimise selective memory issues.  
  • Linked to the above is the notion of “Reliability Decay”, where data recorded during the later phases of the data collection process are likely to be less reliable. 
  • Recency effects may be present, meaning, behaviours that occur near the end of the observation window are more readily available to the observer’s memory, and thus they may influence the way the information is recorded. 
  • The halo effect means that early impressions can influence latter observations. For example, if the observer notices some positive qualities in one thing, these may influence the observer’s perceptions of similar qualities in related things or in the whole. 
  • The central tendency effect is referred to the situations when people will avoid ticking extreme categories (i.e. use mainly midpoint scores). 
  • Observer Drift occurs when the observer starts to redefine the observational variables, to the extent that the data no longer reflects the original definitions of the observed units.  
  • Reactivity effects, when the presence/behaviour of the researcher might alter the participants’ observed behaviour, quite often unintentionally (in some types of research this effect appears to fade with time). 
  • Counter-transference: distancing oneself from the phenomenon under investigation (where possible) might help with problems that occur when the observer’s own judgements about the phenomenon and people they observe affect their observations. 
  • In addition, different observers may give slightly different accounts of an event. Training the observers can help resolve this issue and help them be more consistent in what they observe, how they define it and how they will record it. In projects where more than one observer is used, usually the degree of agreement in their observations is calculated (inter-rater agreement): [(number of times 2 observers agree / number of possible opportunities to agree) * 100]. We should expect at least a 90% inter-rater agreement before we can be confident that the observers are recording the same instances in the same way. In order to see how to calculate Cohen’s Kappa follow the link: http://faculty.chass.ncsu.edu/garson/PA765/reliab.htm#rater and to see how to do it on SPSS go to http://faculty.chass.ncsu.edu/garson/PA765/standard.htm#agreement.
  • Following from the point above, observer-related issues should be seen not only in relation to the behaviours under investigation (how they need to be well understood by the observers and therefore clearly defined and observable), but also in relation to the quality of the research process: observers should stay focused for the duration of the observation, they should put attention to detail etc. The notion of “observer drift”, fatigue and the halo effects discussed above are also relevant here.