Psychological science is considered by many to be going through a time of crisis. Given that much educational neuroscience research draws on findings from psychology, what does this mean for education?

As many of us were taught at school, it is essential that the results of scientific studies are replicated; that experiments are run multiple times so that we can be sure that the results are real. In 2015, one hundred psychology studies were replicated as part of the Reproducibility Project. Overall, only about a third to a half of the replications led to the same result that was initially observed when the study was first run. This was concerning because it indicated that many of the psychological effects we assume are true may not be. The lack of reproducibility in psychology is sometimes referred to as the ‘replication crisis’.

This has implications for teaching and learning, since educational neuroscience research often relies on findings from psychology. The impact of the replication crisis for education was highlighted in a recent review which found that many trials in education do not provide useful results. If an educational trial is based on a psychological finding that has not been replicated and shown to be reliable, the chances of the trial working are slim, which the authors of the review suggested could indeed be one of the reasons for the finding that many such trials are uninformative.

Why is psychology in this state?

There are a number of reasons for the poor reproducibility seen in psychological science. One reason is known as the publication bias, whereby novel and exciting results are more readily accepted for publication than results that are less exciting, such as attempts at replications. A researcher may therefore run a replication study, but find they are unable to publish it. The result of this is that fewer attempts at replication are made, and so research is subject to less scrutiny.

Also, during the research process, scientists may engage in ‘questionable research practices’. These are often small decisions made by researchers that can change the results of a study, giving the impression of a stronger effect than really exists. Examples include only reporting studies that showed an interesting result, or excluding participants from analyses to make findings look more impressive, or collecting more data in the hope that more participants will lead to a significant effect.

“If an educational trial is based on a psychological finding that has not been replicated and shown to be reliable, the chances of the trial working are slim.”

These practices are surprisingly common, since researchers can fool themselves into thinking they are doing these things to show the ‘true effect’. Thankfully, the fact that psychologists are aware of and exploring the issues with their science means that a number of initiatives are being created to combat the replication crisis. In addition to running replications, such as in the Reproducibility Project, there is an increasing move towards pre-registering studies, whereby all aspects of the study and analysis are decided in advance. This guards against questionable research practices, since there is a record of the plan that must be kept to (changes to the plan are allowed, but are clearly marked).

Another initiative is to conduct peer review before a study is run, rather than afterwards, which is how it’s usually done. Increasing numbers of journals now accept publications on the basis of peer review before running a study, meaning that scientists will no longer only publish based on the novelty of their results. Similarly, journals are now more welcoming of replication studies. This means they are more worthwhile for individual scientists to conduct, since they will have the incentive of an all-important publication from of their work.

“Thanks to the recent review of educational trials, future trials will hopefully more closely consider the initial research which led to the trial, and perhaps seek to first replicate those findings.”

Clearly, the replication crisis is real and needs to be taken seriously. But it does not mean that we can no longer trust the results of educational neuroscience studies. Thanks to the recent review of educational trials, future trials will hopefully more closely consider the initial research which led to the trial, and perhaps seek to first replicate those findings. The replication crisis is not the end of research, but will instead usher in a new and more robust era of education research.

3 comments

  1. Annie, we have been in touch before. I am a retired head but retain an interest in teaching, especially into school based research and new evidence that works to promote more effective classroom practice apparently emerging from neuroscience. This is why I look forward to reading the BOLD bulletins. Your latest piece is especially relevant in light of how much weight is now put on applying new techniques and technologies aimed at improving the effectiveness of classroom practice, not least because of the huge amount of pressure from government to ‘raise standards’. I followed through on all the links in this piece and have some questions I would be grateful to get your views on.

    Your final paragraph contains some pretty direct and challenging statements about research standards in psychology. As you point out “much educational neuroscience research draws on findings from psychology”. However, elsewhere you state that “Overall, (out of 100 psychology studies) only about a third to a half of the replications led to the same result…” You identify the replication crisis and through your links I found that this has been an issue of some concern to researchers for at least the last five years, though I feel it might have always have been thus. When you express the opinion that “future trials will hopefully more closely consider the initial research which led to the trial,”, it suggests that this is not an assured development. Is that a fair assessment?

    You mention the Reproducibility Project which seems to have highlighted the weaknesses you write about in relation to psychological research in particular. In summary, you write that, “the replication crisis is real and needs to be taken seriously”. With this in mind, how widely is this known in education circles? What efforts are made when discussing evidence from research with education professionals to highlight that with them? How aware would they be, for example that “If something works in the tightly controlled situation in a laboratory study, it does not necessarily work in a real school setting”?

    I would be interested to learn how much progress has actually been made to address the issues you and others have identified in reviewing the robustness of new developments in neuroscience.

    Many thanks
    John Mountford

    1. Hi John, thanks for your thoughtful questions and comments.

      I would hope that many researchers carrying out trials will take the review that found many trials uninformative seriously and look closely at the initial research, carrying out additional research if necessary. Funders may also take note of this review and require a stronger evidence base before funding trials. As you suggest, these are not assured but are my hopes for the future. Given the rise in preregistered studies, I’d also hope that one day we get to a point where the research base is much more reliable.

      It’s difficult to say how well known these issues are in education circles as my engagement with teachers has typically been through educational neuroscience conferences and with teachers involved in my research, so these will all have been particularly interested in research. However I can say that in my experience there are efforts made to draw these issues to the attention of teachers. For example, when I have spoken to teachers about educational neuroscience (and seen others give similar talks), a lot of attention is given to the challenge of translating lab studies to school settings, and explanations of the steps involved are given. Where researchers have accessed teachers, my feeling is that a pretty good job has been done of explaining these issues, although perhaps not explicitly in relation to the replication crisis.

      Thanks again for your questions,
      Annie

Leave a Reply

Your email address will not be published. Required fields are marked *

Keep up to date with the BOLD newsletter