Peter A. Edelsbrunner, Maja Flaig, Michael Schneider

PDF Version

How can we best tell how many different learning patterns there are in our data?

Latent transition analysis is used to describe different learner patterns. However, it is often hard to tell how many patterns there are. Is there a pattern of learners who have little knowledge, another pattern of learners with a specific misconception, and another pattern of learners who have properly understood everything that we tried to teach them? Or are there some of these patterns but not all, or even additional ones? This is really hard to tell, and different indicators (called “relative fit indices”) are available for helping us determinate how many patterns there really are. We compare the performance of several relative fit indices. We find that the Bayesian information criterion (BIC), which is commonly used to determine the number of learning patterns, is not very accurate in finding the right number of patterns in comparison to other indices.

Eric Jenner, Katherine Lass, Sarah Walsh, Hilary Demby, Rebekah Leger, and Gretchen Falk

PDF Version

How does a cross-age peer mentoring program affect ninth-grade outcomes?

Ninth-grade students who were offered Peer Group Connection High School (PGC-HS) were less likely to receive a suspension or disciplinary referral and self-reported higher levels of school engagement and postsecondary expectations. However, offering the program had no effect on academics (credit attainment, attendance at school, GPA) and other non-cognitive skills (e.g., decision-making skills).

Ben Clarke, Christian Doabler, Marah Sutherland, Derek Kosty, Jessica Turtura, and Keith Smolkowski

PDF Version

The importance of early mathematics

The importance of a successful start to learning mathematics has been a national priority for several decades. Mounting evidence indicates that trajectories of mathematics performance are established early and remain relatively stable across time. This may in part be due to substantial disparities in young students’ access to early mathematics experiences and instruction with preschool-aged students from upper- and middle-class backgrounds already outperforming their economically disadvantaged peers.

Davide Azzolini, Sonia Marzadro, Enrico Rettore, Katja Engelhardt, Benjamin Hertz, Patricia Wastiau

PDF Version

Does providing teachers with personalized support help them complete online training courses?

Yes, but not for all and not everywhere. The TeachUP policy experimentation found large effects of personalized support on course completion in nine European Union Member States among professional (i.e., in-service) teachers (+10.6 percentage points), but not among student teachers. Moreover, no effects are found in Turkey. More studies are needed to investigate the contextual and learner characteristics that drive the heterogeneous effects.

Summary by: Wen Wen

PDF Version

What are “state-” and “trait-” math achievements in early education?

Interventions can boost early math skills, but the role of these early skills on later math achievement is unclear. Consider that students who demonstrate stronger early math skills tend to demonstrate stronger later math achievement, yet some interventions that improve early math skills do not improve later math achievement – that is, the early benefits fade substantially after 2 or 3 years.

Daniela Alvarez-Vargas, Sirui Wan, Lynn S. Fuchs, Alice Klein, & Drew H. Bailey

PDF Version

Despite policy relevance, long term evaluations of educational interventions are rare relative to the amount of end of treatment evaluations. A common approach to this problem is to use statistical models to forecast the long-term effects of an intervention based on the estimated shorter term effects. Such forecasts typically rely on the correlation between children’s early skills (e.g., preschool numeracy) and medium-term outcomes (e.g., 1^st grade math achievement), calculated from longitudinal data available outside the evaluation. This approach sometimes over- or under-predicts the longer-term effects of early academic interventions, raising concerns about how best to forecast the long-term effects of such interventions. The present paper provides a methodological approach to assessing the types of research design and analysis specifications that may reduce biases in such forecasts.

What did we do?

Sam Sims, Jake Anders, Matthew Inglis, Hugues Lortie-Forgues

PDF Version

Randomized controlled trials (RCTs) have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. Yet RCTs are only unbiased in expectation (on average across many RCTs).

Estimates of the effect size from specific RCTs will in general diverge from the true effect due to chance differences between the treatment and control group. In suitably powered trials, this imbalance tends to be small and statistical inference helps to control erroneous findings.

Jonté A. Myers, Elizabeth M. Hughes, Bradley S. Witzel, Rubia D. Anderson, and Jennifer Owens

PDF Version

How are students performing on assessments of word problem solving?

Students' ability to think critically and abstractly is essential for their success in post-secondary education and career advancement. K-12 schools have increased their focus on assisting students in building these skills through word problem solving (WPS). However, students’ WPS performance on national assessments remains discouragingly low, especially among students with disabilities.

Mark White, Bridget Maher, Brian Rowan

PDF Version

Many educational interventions seek to directly shift instructional practice. Observation systems are used to measure changes in instructional practice resulting from such interventions. However, the complexity of observation systems creates the risk of instrumentation biases. Instrumentation bias is bias resulting from changes to the ways that an instrument functions across conditions (e.g., from pre-test to post-test or between control and intervention conditions). For example, teachers could intentionally show off intervention-specific practices whenever they are observed, but not otherwise use those practices. Alternatively, an instructional intervention could shift instruction in ways that increase observation scores without impacting the underlying instructional dynamics that support student learning.

This conceptual paper with a case study exemplar provides a validity framework for using observation systems to evaluate the impact of interventions. Inferences about an intervention’s impact generally involve determining whether a teaching practice has changed within some setting. Observation scores, the evidence for these conclusions, are specific raters’ views of how a rubric would describe observed lessons. The conclusions are far more generalized than the observation scores. The framework (see Figure below) systematically breaks down the processes necessary to operationalize an aspect of teaching practice and sample from a setting to obtain observation scores that can be generalized to draw conclusions.

Catherine Armstrong Asher, Ethan Scherer, James S. Kim

PDF Version

What features of text messaging campaigns for early elementary families might increase their effectiveness?

Text messaging interventions are an increasingly popular way to support students and their families. We compared how three features of text messages, sent to parents, affect the reading behavior and test scores of their early elementary school children:

Ann-Katrin van den Ham and Aiso Heinze

PDF Version

Is an early mathematics support program based on formative assessment effective?

Yes, it is, according to a study conducted with 135 elementary school classes from 40 schools in Germany. The study shows that students at-risk for mathematical difficulties benefited from the two-year "Mathe macht stark (MMS) - Grundschule" (Maths makes you strong - primary school) implementation in Grades 1 and 2. This effect is maintained one year after the intervention ends and without providing Grade 3 formative assessment material. Moreover, students not at-risk for mathematical difficulties also benefited from the program, despite not being the target of the program. Hence, the formative assessment elements the teachers used in the mathematics classrooms for at-risk students were also beneficial for the other students. Interestingly, in an enhanced version of the program, including two extra teacher working hours per week, did not add value for at-risk students in the follow-up test at the end of Grade 3.

Lam D. Pham

PDF Version

Motivation and Context

Evidence suggests that school turnaround can be effective, but why are some models successful, while others fall short? This paper tests whether changes in particular schoolwide practices help explain the effect of turnaround reforms on student outcomes. I study these mechanisms using the Innovation Zone (iZone) model of reform in Shelby County Schools district (serving Memphis, TN) because previous research suggests that the Memphis iZone reforms increased student achievement. Specifically, I examined four mechanisms that potentially explain the positive iZone effects: (1) the recruitment of effective teachers, (2) increased opportunities for teacher collaboration, (3) a more positive learning environment, and (4) increased opportunities for teacher professional development.

Adrian Simpson

PDF Version

Effect size and policy

Standardized effect size estimates are commonly used by the ‘evidence-based education’ community as a key metric for judging relative importance, effectiveness, or practical significance of interventions across a set of studies: larger effect sizes indicate more effective interventions. However, this argument applies rarely; only when linearly equatable outcomes, identical comparison treatments and equally representative samples are used in every study.

Diego Catalán Molina, Tenelle Porter, Catherine Oberle, Misha Haghighat, Afiya Fredericks, Kristen Budd, Sylvia Roberts, Lisa Blackwell, and Kali H. Trzesniewski

PDF Version

How much students benefit from a school intervention depends on how well the intervention is delivered

When a new curriculum is introduced at a school, the quality of its implementation will vary across teachers. Does this matter? In this study, teachers varied widely in how well they implemented a 20-lesson social and emotional blended-learning curriculum. Teachers who delivered the program at higher quality, for example, encouraged student reflection and participation and provided feedback to students on how to improve skills. Teachers who delivered the program at higher quality had students with higher levels of motivation (growth mindset, effort beliefs, and learning goals) at the end of the program compared to teachers who delivered at lower quality.

Matthew P. Steinberg and Haisheng Yang

PDF Version

Is principal induction effective at raising student achievement?

Yes, according to a study of Pennsylvania’s Inspired Leadership (PIL) induction program. In schools where principals completed the PIL induction program, teachers became more effective, resulting in modest improvements in student achievement of approximately 1-2 weeks of additional schooling. These benefits were concentrated in schools that served the most economically disadvantaged and minority students in urban districts.

Ira Nichols-Barrer, Maria Bartlett, Thomas Coen, & Phil Gleason

PDF Version

Do KIPP Middle Schools Boost Long Run Student Outcomes?

Yes they do, according to a rigorous national study of 13 KIPP middle schools. Building on prior studies of KIPP that show KIPP middle schools have strong positive effects on students’ middle school achievement, this study found that KIPP middle schools also improve longer-term rates of enrollment in four-year college programs. Winning a lottery-based admissions offer to a KIPP middle school increased a student’s probability of enrolling in college by 7 percentage points, even though a third of these students never enrolled at KIPP. Adjusting for enrollment, attending KIPP increased college enrollment rates by 13 percentage points. This boost is similar in size to nationwide disparities in college enrollment across racial groups—a relevant benchmark since nearly all KIPP students are Black or Latinx. 

Keith Smolkowski, Hill Walker, Brion Marquez, Derek Kosty, Claudia Vincent, Carey Black, Gulcan Cil, & Lisa A. Strycker

PDF Version

Can Social Skills Instruction be Efficient and Effective?

Yes. A rigorous study shows that the We Have Skills program efficiently and effectively taught the academically related social skills needed for early elementary students to succeed in school. We Have Skills appealed to children, and teachers quickly mastered and readily implemented the program in their classrooms.

Kaitlyn G. Fitzgerald & Elizabeth Tipton

PDF Version

What type of data do clearinghouses communicate?

As the body of scientific evidence about what works in education grows, so does the need to effectively communicate that evidence to policy-makers and practitioners. Clearinghouses, such as the What Works Clearinghouse (WWC), have emerged to facilitate the evidence-based decision-making process and have taken on the non-trivial task of distilling often complex research findings to non-researchers. Among other things, this involves reporting effect sizes, statistical uncertainty, and meta-analytic summaries. This information is often reported visually. However, existing visualizations often do not follow data visualization best practices or take the statistical cognition of the audience into consideration.

Summary by: Hang (Heather) Do

PDF Version

What was this study about?

Chronic absenteeism (missing more than 10% of school days or more in one year) negatively impacts children’s school achievement and development. Yet, little is known about how absenteeism influences the effectiveness of interventions. In this study, the authors examined whether absenteeism affected the impacts of an intensive two-year professional development (PD) intervention aiming to improve the quality of Chilean public preschool and kindergarten and enhance the language and literacy outcomes of participating children (UBC (Un Buen Comienzo/A Good Start)).

Jake Anders, Francesca Foliano, Matt Bursnall, Richard Dorsett, Nathan Hudson, Johnny Runge, and Stefan Speckesser

PDF Version

What is formative assessment?

'Formative assessment', often used interchangeably with the term 'assessment for learning' and in contrast to 'summative assessment', refers to assessment activities undertaken by teachers – or students themselves – to obtain evidence which is then used to adapt teaching and learning methods to meet student needs and improve learning outcomes.