Mitigating Illusory Results through Preregistration in Education

Summary by: Claire Chuter

PDF Version

Good researchers thoroughly analyze their data, right? Practices like testing the right covariates, running your analyses in multiple ways to find the best fitting model, screening for outliers, and testing for mediation or moderation effects are indeed important practices… but with a massive caveat. The aggregation of many of these rigorous research practices (as well as some more dubious ones) can lead to what the authors call “illusory results” – results that seem real but are unlikely to be reproduced. In other words, implementation of these common practices (see Figure 1 in the article), often leads researchers to run multiple analytic tests which may unwittingly inflate their chances of stumbling upon a significant finding by chance.

Potential Solutions

So, what can be done about illusory results? Preregistration is one of several “open science” approaches designed to improve transparency and reproducibility in research. In preregistration, researchers describe their hypotheses, methods, and analyses before data are collected or prior to analyses, in a way that can be externally verified (e.g., by posting specific hypotheses and/or statistical code on a website). This process can benefit the researchers as well as the research, by providing a clear timestamp of when ideas were first generated, and a verifiable map of the analytic path the researcher took to arrive at their conclusion. Preregistration has recently become more feasible through online repositories and is increasingly encouraged by many journals (including JREE).

How do you preregister a study?

By now, protocols for preregistering experiments in psychology are reasonably well developed.  However, because the practice of pre-registration is new to the field of education, guidelines for the content of registrations are still being developed, especially for non-experimental studies. Gehlbach and Robinson suggest that researchers should highlight the study’s measures, procedures and specific analytic pathway (Table 2) that were decided upon a priori.

Table 2. Sample of types of analytic logistics for authors to provide in study preregistration.

What are the next steps for the field of education?

Researchers voice a number of common concerns about preregistration.  For example, some worry that preregistering a study may undermine the value of exploratory research. The authors argue that, on the contrary, a new norm to divide results sections into pre-specified results and exploratory results easily solves this problem.  Other concerns will require some trial and error before educational researchers can arrive at satisfactory compromises or solutions.  For instance, what is the best way to preregister hypotheses regarding existing big data sets.

The authors conclude several norms will need to change to diminish the prevalence of illusory findings. Journals could have a powerful influence by elevating the value of preregistered studies, encouraging the publication of null findings, and providing guidance to their reviewers.  Making successful improvements in educational practice is hard enough, scholars need to ensure that educators are working from a research base of real, rather than illusory, results.

Full Article Citation:
Gehlbach, H., & Robinson, C. D. (2018). Mitigating illusory results through preregistration in educationJournal of Research on Educational Effectiveness11(2), 296-315.

Share this post:

Comments on "Mitigating Illusory Results through Preregistration in Education"

Comments 0-5 of 0

Please login to comment