A new review shows that randomised control trials (RCTs) in education are often uninformative. It is essential for the educational research community to consider why this is the case, so that future RCTs give us the evidence that will help us improve teaching and learning.
While considered the gold standard in medical research, RCTs in educational research are less common and arguably more challenging. RCTs aim to tell us whether a ‘treatment’ is effective; in education this means establishing whether a new strategy or tool is better (usually for learning) than what is already happening in classrooms. In the spirit of testing what works, RCTs have been tested and don’t seem to be working.
One hundred and forty-one educational RCTs, comprising a total of 1,222,024 participants, were examined in a recent review by Hugues Lortie-Forgues and Matthew Inglis. Each of these RCTs was designed to improve academic performance in students. Overall, the review found that the results of these trials were largely uninformative; they could not adequately establish whether or not the new classroom measure was effective. The authors of the review argue that given the expense of RCTs, it is essential that researchers (and their teacher-collaborators) seek to understand why this is the case.
What went wrong?
Three possible explanations of this finding are presented by the authors. Each of the RCTs considered in the review was based on promising initial research that indicated that the ‘treatment’ to be tested would be effective. It is possible that the initial research underlying many of the RCTs was unreliable, that RCTs were run based on flimsy evidence. In part this may be due to the ‘replication crisis’ in psychology, where a number of key research findings have not been replicated in further studies (see upcoming post on this topic).
A second explanation is that the all-important translation from basic science to classroom practice was poor. If something works in a tightly controlled situation in a laboratory study, it does not necessarily work in a real school setting, where the environment is very different. It is also unlikely that all schools implemented the trial in exactly the same manner, as trials have to fit around normal school life.
“In the spirit of testing what works, RCTs have been tested and don’t seem to be working.”
Finally, it is possible that the new measures were indeed effective, but the analysis was unable to pick up on the positive result. This could be due to the sample size of the study being too small, or the wrong outcome being measured. All of the RCTs included in the review aimed to improve academic outcomes. It is possible that there were positive effects on other aspects of school life that were not directly measured in the studies.
For example, a maths study may have improved maths enjoyment but not directly affected maths performance; the trial would be deemed “ineffective”, and would not have had the intended consequence, but may have nonetheless had a real, important impact. There might also be long-term outcomes not measured by the trial. In the maths example, greater maths enjoyment might lead to improved maths performance in a few years, but not during the period of the trial.
It is likely that all three of these explanations had some role in the review’s finding. The research community now needs to consider what can be done to enhance the utility of RCTs. This might mean a closer examination of the initial research leading to an RCT, or further basic research before an RCT is run. It might mean researchers and teachers working more closely together to establish how best to translate basic research findings into practice, or schools only being included in a trial if they are able to commit to following the trial plan strictly.
“This might mean researchers and teachers working more closely together to establish how best to translate basic research findings into practice.”
It also might mean rethinking what success in a trial looks like; taking a more holistic approach to improving education. Given concerns with the sometimes huge expense of conducting RCTs, teacher-led trials may be a cheaper way of conducting high quality research into what works best in school.
Although the results of the review were disappointing, this is an important opportunity for the research community to take stock, consider what can be done to make RCTs informative, and then design and use them effectively. This will allow educational trials to serve their intended purpose: to establish how teaching and learning can be improved.