Translating questionable research to the classroom

Psychological science is considered by many to be going through a time of crisis. Given that much educational neuroscience research draws on findings from psychology, what does this mean for education?

As many of us were taught at school, it is essential that the results of scientific studies are replicated; that experiments are run multiple times so that we can be sure that the results are real. In 2015, one hundred psychology studies were replicated as part of the Reproducibility Project. Overall, only about a third to a half of the replications led to the same result that was initially observed when the study was first run. This was concerning because it indicated that many of the psychological effects we assume are true may not be. The lack of reproducibility in psychology is sometimes referred to as the ‘replication crisis’.

This has implications for teaching and learning, since educational neuroscience research often relies on findings from psychology. The impact of the replication crisis for education was highlighted in a recent review which found that many trials in education do not provide useful results. If an educational trial is based on a psychological finding that has not been replicated and shown to be reliable, the chances of the trial working are slim, which the authors of the review suggested could indeed be one of the reasons for the finding that many such trials are uninformative.

Why is psychology in this state?

There are a number of reasons for the poor reproducibility seen in psychological science. One reason is known as the publication bias, whereby novel and exciting results are more readily accepted for publication than results that are less exciting, such as attempts at replications. A researcher may therefore run a replication study, but find they are unable to publish it. The result of this is that fewer attempts at replication are made, and so research is subject to less scrutiny.

Also, during the research process, scientists may engage in ‘questionable research practices’. These are often small decisions made by researchers that can change the results of a study, giving the impression of a stronger effect than really exists. Examples include only reporting studies that showed an interesting result, or excluding participants from analyses to make findings look more impressive, or collecting more data in the hope that more participants will lead to a significant effect.

“If an educational trial is based on a psychological finding that has not been replicated and shown to be reliable, the chances of the trial working are slim.”

These practices are surprisingly common, since researchers can fool themselves into thinking they are doing these things to show the ‘true effect’. Thankfully, the fact that psychologists are aware of and exploring the issues with their science means that a number of initiatives are being created to combat the replication crisis. In addition to running replications, such as in the Reproducibility Project, there is an increasing move towards pre-registering studies, whereby all aspects of the study and analysis are decided in advance. This guards against questionable research practices, since there is a record of the plan that must be kept to (changes to the plan are allowed, but are clearly marked).

Another initiative is to conduct peer review before a study is run, rather than afterwards, which is how it’s usually done. Increasing numbers of journals now accept publications on the basis of peer review before running a study, meaning that scientists will no longer only publish based on the novelty of their results. Similarly, journals are now more welcoming of replication studies. This means they are more worthwhile for individual scientists to conduct, since they will have the incentive of an all-important publication from of their work.

“Thanks to the recent review of educational trials, future trials will hopefully more closely consider the initial research which led to the trial, and perhaps seek to first replicate those findings.”

Clearly, the replication crisis is real and needs to be taken seriously. But it does not mean that we can no longer trust the results of educational neuroscience studies. Thanks to the recent review of educational trials, future trials will hopefully more closely consider the initial research which led to the trial, and perhaps seek to first replicate those findings. The replication crisis is not the end of research, but will instead usher in a new and more robust era of education research.