Postgraduate research opportunities Evaluating & improving group-based judgment & decision making

Summary of Research

Groups provide several benefits over individuals for judgment and decision making, but they suffer from problems too. Structured-group techniques, like Delphi, use strictly controlled information exchange between individuals to retain positive aspects of group interaction, while ameliorating negative. These methods regularly use 'nominal' groups that interact in a remote, distributed, and often anonymous manner, thus lending themselves to internet applications, with a consequent recent increase in popularity. However, evidence for the utility of the techniques is scant, major reasons for which being difficulties maintaining experimental control and logistical problems in recruiting sufficient empirical ‘groups’ to produce statistically meaningful results. As a solution, we present the Simulated Group Response Paradigm, where individual responses are first elicited in a pre-study – or created by the experimenter – then subsequently fed back to highly-controlled simulated groups. This paradigm facilitates investigation of factors leading to virtuous opinion change in groups, and subsequent development of structured-group techniques.

Research Aims & Objectives/Literature Review/Overview of Methodology

Delphi achieves its advantages over individual judgment and interacting groups by first surveying the anonymous opinions of several individual experts, who are thus able to make judgments free of any anchors provided by other group members or fear of group censure. Group members then receive feedback – usually edited and summarised by a facilitator – regarding the opinions of other experts in their nominal group. Next, each individual expert is invited to revise his or her opinion and then these revised opinions are again collated by the facilitator to be fed back in subsequent rounds of revision - or released as the final outcome if a sufficient degree of consensus (or a stable dissensus) has been reached. Between-round feedback typically includes summary statistics describing the group’s responses and participants can also be asked to provide written rationales in support of their judgments.

Delphi started out equally concerned with accuracy and consensus its use has, until the last few years, largely migrated towards the latter. The consequence of this orientation is that there have been almost no studies in recent years that attempt to evaluate the method in terms of its ability to produce more accurate judgments or forecasts, either during the course of the method itself (e.g. comparing the aggregate judgments from the first and final rounds), or relative to other structured-group techniques or unstructured (face-to-face) groups. Consequently, the most recent review of the accuracy of Delphi was over 20 years ago (Rowe & Wright, 1999) where it was found that Delphi has often shown improved performance versus interacting groups or individual panellists’ statistically aggregated first round responses – at least for short-term forecasting problems and tasks involving judgments of quantities, where accuracy can easily be assessed. Since that point, George Wright has become the world-leader in such empirical research on group-based judgment and decision-making using the Delphi process. His nine papers, listed in the later section "How does the project fit", have achieved about 6,000 citations in the Google Scholar database.

Our assertion is that structured-group research should refocus on the conditions in which it is likely to improve judgment accuracy, a more precise framing of the research agenda is:

Structured-group research should seek to understand how virtuous opinion change is achieved in individual panellists as a consequence of the interaction of three broad factors, namely, the characteristics of the individual group members; the characteristics of the technique itself (i.e. the particular variation used); and the characteristics of the context in which the technique is used.

The Simulated Group Response Paradigm (SGRP)

To aid in the conduct of research in this area, we suggest that research should focus on identifying how to achieve valid or virtuous opinion change in individual ‘group’ members, and that this is best done using simulated groups in which the nature and quantity of feedback from supposed fellow panellists (who may or may not be real) is manipulated in a controlled manner in order to allow the answering of precise experimental questions with sufficient power to enable the obtaining of statistically significant results. We call the protocol we have developed the Simulated Group Response Paradigm (SGRP). We use the term ‘paradigm’ deliberately, as the approach involves a general methodology within which are a variety of specific procedures that are informed by our theoretical outlook, which is that the intent of research should be to establish and explain the factors (and their interactions) that are responsible for virtuous opinion change in individuals .

The SGRP consists of two main phases, which we briefly describe now in reverse order. The second phase is where hypotheses about influences on virtuous opinion change are tested in a Delphi procedure with – in the basic form – two rounds. In Round 1 participants individually provide answers to a judgment or reasoning task. At Round 2 the participants then receive a number of other answers to the same task and are invited to revise their Round 1 answer. Thus Phase 2 of the SGRP is, from the point of view of a participant, identical to a standard two-round Delphi. There are, however, important differences from the perspective of the experimenter, both preceding Phase 2, and after Round 2 of Phase 2 in the analysis of the data. The most significant difference is that the answers fed-back at Round 2 do not derive from other group members working on the task more-or-less synchronously – as is usually the case – but are elicited, or created, at some earlier point in time during SGRP’s Phase 1. Thus, at Phase 2 there are no ‘real’ groups interacting through a facilitator, and the unit of analysis is the change in each individual participant’s answers between the two rounds.

In short, we propose a new research approach to the study of opinion change in Delphi and other structured groups, introducing the Simulated Group Response Paradigm (SGRP). We developed this approach in order to help researchers answer the call of Rowe and Wright (2011) to conduct a greater number of more rigorous studies into the working mechanisms of Delphi so as to inform the application of the method and thus maximise its potential for improving human judgment and decision making: we have also argued that the SGRP has relevance to understanding and improving structured-group techniques more generally. To this end we identified a variety of factors that will be the source of our future research, the study of which will be enabled by our paradigm – the traditional approach with ‘real’ groups suffering as it does from a lack of experimental control and insufficient power for suitable analysis, unless a generally unfeasible number of participants are recruited. Topics to study include: individual differences (e.g., personality types, expertise levels, cognitive styles, social skills, etc), situational aspects (e.g., time available, degree of prior consensus, type of problem posed, response complexity, etc), and Delphi method variant ( e.g., group size, type of between-round feedback, number of rounds, etc). The exact focus will depend on the interests of the research student.

We believe that research into opinion formation and change in online groups is particularly timely due to the huge increase in group-working over the internet resulting from the novel coronavirus pandemic: such working is likely to remain more common than beforehand after the pandemic recedes, due to realised gains in productivity and an associated reduction of carbon emissions. Even prior to COVID-19, recent years have seen a burgeoning of online meetings for commercial, government and academic purposes, as well as the emergence of numerous public fora and communities such as Quora, Reddit, Mumsnet, TripAdvisor and Change My View, to name but a few. We submit that research using the SGRP can help investigate how such online information exchanges and opinion makers can lead to improved decision making in organizations and wider society (e.g. for promoting desired environmental and health behaviours, or increasing public participation in government) rather than perpetuating – or even amplifying – bias, prejudice and misinformation.

One additional possibility that we find exciting is the prospect of researchers throughout the world using the SGRP paradigm for a range of problems (judgmental, decision making and problem solving) and then making their databases accessible more widely. This approach could significantly aid the Delphi and nominal group research community by shortening the research process, aiding consistency, and potentially leading to a quantum leap in the number of rigorously conducted, well-analysed studies. Furthermore, such studies, we hope, would likely be aimed at answering more sophisticated questions than “does technique x ‘work’?”, notably addressing the relative utility of different variations and compositions of structured-group procedures and their capacity to lead to virtuous opinion change in individuals. Results from such research could then be used inform the conduct of real-world structured-group instantiations seeking specific answers to policy issues etc, thereby increasing the likelihood of producing high quality answers to important questions that Society wishes to find group-based answers.

Timeline

The timeline of the SGRP-linked sequence of experiments will depend on the capabilities and interests of the student. We expect the student to have a prior degree in psychology - such that she/he has the capabilities to develop his/her research quickly - but this may not be the case.

Anticipated timeline

• attendance/completion of the research methodology course delivered by the Business School (Months 1-18)
• systematic review of the group-decision making literature (Months 1-12)
• development of a conceptual framework (Months 7-12)
• writing a critical literature review with the aim of publication (Months 13-18)
• empirical research design & piloting - based on the SGRP method (Months 13-18)
• data collection & analysis for at least four experiments (Months 13-30)
• interpretation of findings & linking of these to the results of the literature review (Months 25-30)
• writing up of one experiment in detail for publication (Months 25-36)
• writing up & presenting at least one conference paper (Months 25-36)
• writing up the thesis (Months 31-36)
• thesis submission (Month 36)
• writing a second experiment as a potential journal article (Months 31-48)

Further information

The proposed research aligns with the University main strategic theme of supporting decision making in the area of Society and Policy. It is also part of the Department of Management Science's focus on "Informed Decision Analytics. The Delphi method is a much used method in national foresight studies in technology in Europe and so the basic research proposed also fits easily within the University's main strategic theme of " Innovation and entrepreneurship". The second supervisor, Dr Stathis Tapinos, is based in the Hunter Centre at Strathclyde.

The student will, naturally, become a co-author in George Wright's ongoing, journal-focused publication steam on the Delphi method - as his past PhD students and research assistants have been. For example, recently, Dr Ian Belton, Dr Megan Grime, and Dr Iain Hamlin. In the past, Dr Fergus Bolger and Dr Rene Rowe.

The student will attend the general, underpinning b-school research methodology course and participate in the seminar steam on strategy/foresight developed by Stathis Tapinos (second supervisor) within the Hunter Centre, as well as participating in the seminar steam delivered within the Management Science department. The student will be trained in experimental techniques by George Wright and will become part of his active research team in the area of Delphi research. George has published a text on experimental design and statistics in psychology, "Investigative design and Statistics" (Penguin), and his underpinning knowledge on experimental design and statistical analysis will be transferred to the student over the 36 months of the studentship.

List of references

• Graefe, A., & Armstrong, J.S. (2011). Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task. International Journal of Forecasting, 27(1), 183–95
• Kuhn, T. (1962). The Structure of Scientific Revolutions. Chicago: Chicago University Press
• Parenté, R.J., Hiöb, T.N., Silver, R.A., Jenkins, C., Poe, M.P., & Mullins, R.J. (2005). The Delphi method, impeachment and terrorism: Accuracies of short-range forecasts for volatile world events. Technological Forecasting and Social Change, 72(4), 401-411
• Rowe, G. & Wright, G. (1999). The Delphi technique as a forecasting tool: Issues and analysis, International Journal of Forecasting, 15, 353-375
• Rowe, G. & Wright, G. (2011). The Delphi Technique: Past, present, and future prospects – introduction to the Special Issue. Technological Forecasting and Social Change, 78(9), 1487-1490