Daniel Litwok, Austin Nichols, Azim Shivji, and Robert Olsen

PDF Version

Experimental studies of educational interventions are rarely designed to produce impact evidence, justified by statistical inference, that generalizes to populations of interest to education policymakers.  This simulation study explores whether formal sampling strategies for selecting districts and schools improve the generalizability of impact evidence from experimental studies.

Which selection strategies produced samples with the greatest generalizability to the target population?

Daniel Litwok, Laura R. Peck, and Douglas Walton

PDF Version

How do the earnings impacts of healthcare training vary?

This article explores how earnings impacts vary in an experimental evaluation of a sectoral job training program. We find that over the first two years in the study, those who completed long-term credentials (defined as college degrees or certificates that require a year or more of classes to earn) had program impacts that were about $2,000 larger per year than those who did not complete long-term credentials (whether they completed a short-term credential or no credential at all). A possible explanation for this finding is that those who earned a long-term credential had different experiences in the program, including more engagement with support services, and different post-program outcomes, such as greater employment in high-wage healthcare occupations like registered nurse.

Davide Azzolini, Sonia Marzadro, Enrico Rettore, Katja Engelhardt, Benjamin Hertz, Patricia Wastiau

PDF Version

Does providing teachers with personalized support help them complete online training courses?

Yes, but not for all and not everywhere. The TeachUP policy experimentation found large effects of personalized support on course completion in nine European Union Member States among professional (i.e., in-service) teachers (+10.6 percentage points), but not among student teachers. Moreover, no effects are found in Turkey. More studies are needed to investigate the contextual and learner characteristics that drive the heterogeneous effects.

Jonté A. Myers, Elizabeth M. Hughes, Bradley S. Witzel, Rubia D. Anderson, and Jennifer Owens

PDF Version

How are students performing on assessments of word problem solving?

Students' ability to think critically and abstractly is essential for their success in post-secondary education and career advancement. K-12 schools have increased their focus on assisting students in building these skills through word problem solving (WPS). However, students’ WPS performance on national assessments remains discouragingly low, especially among students with disabilities.

Diego Catalán Molina, Tenelle Porter, Catherine Oberle, Misha Haghighat, Afiya Fredericks, Kristen Budd, Sylvia Roberts, Lisa Blackwell, and Kali H. Trzesniewski

PDF Version

How much students benefit from a school intervention depends on how well the intervention is delivered

When a new curriculum is introduced at a school, the quality of its implementation will vary across teachers. Does this matter? In this study, teachers varied widely in how well they implemented a 20-lesson social and emotional blended-learning curriculum. Teachers who delivered the program at higher quality, for example, encouraged student reflection and participation and provided feedback to students on how to improve skills. Teachers who delivered the program at higher quality had students with higher levels of motivation (growth mindset, effort beliefs, and learning goals) at the end of the program compared to teachers who delivered at lower quality.

Kaitlyn G. Fitzgerald & Elizabeth Tipton

PDF Version

What type of data do clearinghouses communicate?

As the body of scientific evidence about what works in education grows, so does the need to effectively communicate that evidence to policy-makers and practitioners. Clearinghouses, such as the What Works Clearinghouse (WWC), have emerged to facilitate the evidence-based decision-making process and have taken on the non-trivial task of distilling often complex research findings to non-researchers. Among other things, this involves reporting effect sizes, statistical uncertainty, and meta-analytic summaries. This information is often reported visually. However, existing visualizations often do not follow data visualization best practices or take the statistical cognition of the audience into consideration.

Summary by: Hang (Heather) Do

PDF Version

What was this study about?

Chronic absenteeism (missing more than 10% of school days or more in one year) negatively impacts children’s school achievement and development. Yet, little is known about how absenteeism influences the effectiveness of interventions. In this study, the authors examined whether absenteeism affected the impacts of an intensive two-year professional development (PD) intervention aiming to improve the quality of Chilean public preschool and kindergarten and enhance the language and literacy outcomes of participating children (UBC (Un Buen Comienzo/A Good Start)).

Wendy Chan

PDF Version

Will this intervention work for me?

This is one of the questions that make up the core of generalization research. Generalizations focus on the extent to which the findings of a study apply to people in a different context, in a different time period, or in a different study altogether. In education, one common type of generalization involves examining whether the results of an experiment (e.g., the estimated effect of an intervention) apply to a larger group of people, or a population.

Howard Bloom, Steve Raudenbush, Michael Weiss, & Kristin Porter

PDF version

Multisite randomized trials are experiments where individuals are randomly assigned to alternative experimental arms within each of a collection of sites (e.g., schools).  They are used to estimate impacts of educational interventions. However, little attention has been paid to using them to quantify and report cross-site impact variation. The present paper, which received the 2017 JREE Outstanding Article Award, provides a methodology that can help to fill this gap.

Why and how is knowledge about cross-site impact variation important?