BLOG

Quantifying ‘promising trials bias’ in randomized controlled trials in education

Sam Sims, Jake Anders, Matthew Inglis, Hugues Lortie-Forgues

PDF Version

Randomized controlled trials (RCTs) have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. Yet RCTs are only unbiased in expectation (on average across many RCTs).

Estimates of the effect size from specific RCTs will in general diverge from the true effect due to chance differences between the treatment and control group. In suitably powered trials, this imbalance tends to be small and statistical inference helps to control erroneous findings.

Read More

Mathematical Word-Problem-Solving Instruction for Upper Elementary and Secondary Students with Mild Disabilities and Students at Risk for Math Failure: A Research Synthesis

Jonté A. Myers, Elizabeth M. Hughes, Bradley S. Witzel, Rubia D. Anderson, and Jennifer Owens

PDF Version

How are students performing on assessments of word problem solving?

Students' ability to think critically and abstractly is essential for their success in post-secondary education and career advancement. K-12 schools have increased their focus on assisting students in building these skills through word problem solving (WPS). However, students’ WPS performance on national assessments remains discouragingly low, especially among students with disabilities.

Read More

A Framework for addressing Instrumentation Biases when using Observation Systems as Outcome Measures in Instructional Interventions

Mark White, Bridget Maher, Brian Rowan

PDF Version

Many educational interventions seek to directly shift instructional practice. Observation systems are used to measure changes in instructional practice resulting from such interventions. However, the complexity of observation systems creates the risk of instrumentation biases. Instrumentation bias is bias resulting from changes to the ways that an instrument functions across conditions (e.g., from pre-test to post-test or between control and intervention conditions). For example, teachers could intentionally show off intervention-specific practices whenever they are observed, but not otherwise use those practices. Alternatively, an instructional intervention could shift instruction in ways that increase observation scores without impacting the underlying instructional dynamics that support student learning.

This conceptual paper with a case study exemplar provides a validity framework for using observation systems to evaluate the impact of interventions. Inferences about an intervention’s impact generally involve determining whether a teaching practice has changed within some setting. Observation scores, the evidence for these conclusions, are specific raters’ views of how a rubric would describe observed lessons. The conclusions are far more generalized than the observation scores. The framework (see Figure below) systematically breaks down the processes necessary to operationalize an aspect of teaching practice and sample from a setting to obtain observation scores that can be generalized to draw conclusions.

Read More

Using a Factorial Design to Maximize the Effectiveness of a Parental Text Messaging Intervention

Catherine Armstrong Asher, Ethan Scherer, James S. Kim

PDF Version

What features of text messaging campaigns for early elementary families might increase their effectiveness?

Text messaging interventions are an increasingly popular way to support students and their families. We compared how three features of text messages, sent to parents, affect the reading behavior and test scores of their early elementary school children:

Read More

Evaluation of a state-wide mathematics support program for at-risk students in Grades 1 and 2 in Germany

Ann-Katrin van den Ham and Aiso Heinze

PDF Version

Is an early mathematics support program based on formative assessment effective?

Yes, it is, according to a study conducted with 135 elementary school classes from 40 schools in Germany. The study shows that students at-risk for mathematical difficulties benefited from the two-year "Mathe macht stark (MMS) - Grundschule" (Maths makes you strong - primary school) implementation in Grades 1 and 2. This effect is maintained one year after the intervention ends and without providing Grade 3 formative assessment material. Moreover, students not at-risk for mathematical difficulties also benefited from the program, despite not being the target of the program. Hence, the formative assessment elements the teachers used in the mathematics classrooms for at-risk students were also beneficial for the other students. Interestingly, in an enhanced version of the program, including two extra teacher working hours per week, did not add value for at-risk students in the follow-up test at the end of Grade 3.

Read More

Why Do We Find these Effects? Mechanisms that Explain the Effect of School Turnaround Reforms

Lam D. Pham

PDF Version

Motivation and Context

Evidence suggests that school turnaround can be effective, but why are some models successful, while others fall short? This paper tests whether changes in particular schoolwide practices help explain the effect of turnaround reforms on student outcomes. I study these mechanisms using the Innovation Zone (iZone) model of reform in Shelby County Schools district (serving Memphis, TN) because previous research suggests that the Memphis iZone reforms increased student achievement. Specifically, I examined four mechanisms that potentially explain the positive iZone effects: (1) the recruitment of effective teachers, (2) increased opportunities for teacher collaboration, (3) a more positive learning environment, and (4) increased opportunities for teacher professional development.

Read More

A recipe for disappointment: policy, effect size and the winner’s curse

Adrian Simpson

PDF Version

Effect size and policy

Standardized effect size estimates are commonly used by the ‘evidence-based education’ community as a key metric for judging relative importance, effectiveness, or practical significance of interventions across a set of studies: larger effect sizes indicate more effective interventions. However, this argument applies rarely; only when linearly equatable outcomes, identical comparison treatments and equally representative samples are used in every study.

Read More

How to measure quality of delivery: Focus on teaching practices that help students to develop proximal outcomes

Diego Catalán Molina, Tenelle Porter, Catherine Oberle, Misha Haghighat, Afiya Fredericks, Kristen Budd, Sylvia Roberts, Lisa Blackwell, and Kali H. Trzesniewski

PDF Version

How much students benefit from a school intervention depends on how well the intervention is delivered

When a new curriculum is introduced at a school, the quality of its implementation will vary across teachers. Does this matter? In this study, teachers varied widely in how well they implemented a 20-lesson social and emotional blended-learning curriculum. Teachers who delivered the program at higher quality, for example, encouraged student reflection and participation and provided feedback to students on how to improve skills. Teachers who delivered the program at higher quality had students with higher levels of motivation (growth mindset, effort beliefs, and learning goals) at the end of the program compared to teachers who delivered at lower quality.

Read More

Does Principal Professional Development Improve Schooling Outcomes? Evidence from Pennsylvania’s Inspired Leadership Induction Program

Matthew P. Steinberg and Haisheng Yang

PDF Version

Is principal induction effective at raising student achievement?

Yes, according to a study of Pennsylvania’s Inspired Leadership (PIL) induction program. In schools where principals completed the PIL induction program, teachers became more effective, resulting in modest improvements in student achievement of approximately 1-2 weeks of additional schooling. These benefits were concentrated in schools that served the most economically disadvantaged and minority students in urban districts.

Read More

KIPP Middle Schools Increase Students’ College Enrollment Rates

Ira Nichols-Barrer, Maria Bartlett, Thomas Coen, & Phil Gleason

PDF Version

Do KIPP Middle Schools Boost Long Run Student Outcomes?

Yes they do, according to a rigorous national study of 13 KIPP middle schools. Building on prior studies of KIPP that show KIPP middle schools have strong positive effects on students’ middle school achievement, this study found that KIPP middle schools also improve longer-term rates of enrollment in four-year college programs. Winning a lottery-based admissions offer to a KIPP middle school increased a student’s probability of enrolling in college by 7 percentage points, even though a third of these students never enrolled at KIPP. Adjusting for enrollment, attending KIPP increased college enrollment rates by 13 percentage points. This boost is similar in size to nationwide disparities in college enrollment across racial groups—a relevant benchmark since nearly all KIPP students are Black or Latinx. 

Read More

We Have Skills, Effective and Efficient Social Skills Instruction for Early Elementary

Keith Smolkowski, Hill Walker, Brion Marquez, Derek Kosty, Claudia Vincent, Carey Black, Gulcan Cil, & Lisa A. Strycker

PDF Version

Can Social Skills Instruction be Efficient and Effective?

Yes. A rigorous study shows that the We Have Skills program efficiently and effectively taught the academically related social skills needed for early elementary students to succeed in school. We Have Skills appealed to children, and teachers quickly mastered and readily implemented the program in their classrooms.

Read More

The Meta-Analytic Rain Cloud (MARC) Plot: A New Approach to Visualizing Clearinghouse Data

Kaitlyn G. Fitzgerald & Elizabeth Tipton

PDF Version

What type of data do clearinghouses communicate?

As the body of scientific evidence about what works in education grows, so does the need to effectively communicate that evidence to policy-makers and practitioners. Clearinghouses, such as the What Works Clearinghouse (WWC), have emerged to facilitate the evidence-based decision-making process and have taken on the non-trivial task of distilling often complex research findings to non-researchers. Among other things, this involves reporting effect sizes, statistical uncertainty, and meta-analytic summaries. This information is often reported visually. However, existing visualizations often do not follow data visualization best practices or take the statistical cognition of the audience into consideration.

Read More

Experimental Impacts of a Preschool Intervention in Chile on Children's Language Outcomes: Moderation by Student Absenteeism

Summary by: Hang (Heather) Do

PDF Version

What was this study about?

Chronic absenteeism (missing more than 10% of school days or more in one year) negatively impacts children’s school achievement and development. Yet, little is known about how absenteeism influences the effectiveness of interventions. In this study, the authors examined whether absenteeism affected the impacts of an intensive two-year professional development (PD) intervention aiming to improve the quality of Chilean public preschool and kindergarten and enhance the language and literacy outcomes of participating children (UBC (Un Buen Comienzo/A Good Start)).

Read More

The effect of embedding formative assessment on pupil attainment

Jake Anders, Francesca Foliano, Matt Bursnall, Richard Dorsett, Nathan Hudson, Johnny Runge, and Stefan Speckesser

PDF Version

What is formative assessment?

'Formative assessment', often used interchangeably with the term 'assessment for learning' and in contrast to 'summative assessment', refers to assessment activities undertaken by teachers – or students themselves – to obtain evidence which is then used to adapt teaching and learning methods to meet student needs and improve learning outcomes.

Read More

Modeling and Comparing Seasonal Trends in Interim Achievement Data

James Soland & Yeow Meng Thum

PDF Version

Introduction

Interim achievement tests are often used to monitor student and school performance over time. Unlike end-of-year achievement tests used for accountability, interim tests are administered multiple times per year (e.g., Fall, Winter, and Spring) and vary across schools in terms of when in the school year students take them. As a result, scores reflect seasonal patterns in achievement, including summer learning loss. Despite the prevalence of interim tests, few statistical models are designed to answer questions commonly asked with interim test data (e.g., Do students whose achievement grows the most over several years, tend to experience below-average summer loss?). In this study we compare the properties of three growth models that can be used to examine interim test data.

Read More

Performance Evaluations as a Measure of Teacher Effectiveness When Implementation Differs

James Cowan, Dan Goldhaber, Roddy Theobald

PDF Version

Overview

We use statewide data from Massachusetts to investigate the school role in teacher evaluation. Schools classify most teachers as proficient but differ substantially in how frequently they assign other ratings. We show these patterns are driven by differences in the application of standards across schools, not by differences in the distribution of teacher quality.

Read More

The Effects of Higher-Stakes Teacher Evaluation on Office Disciplinary Referrals

David Liebowitz, Lorna Porter & Dylan Bragg

PDF Version

Discussions about teacher evaluation often focus on how evaluations can inform high-stakes job decisions. However, when designing teacher evaluation frameworks, policy makers and system leaders also explicitly seek to promote the development of teachers’ pedagogical skills. One clear emphasis in most measures of teacher practice is that teachers develop a safe and supportive classroom learning environment. Thus, as intended, introducing more intensive teacher evaluation practices could decrease the rate at which teachers send students to the office for misbehavior. This might happen if, through the cycle of observations and feedback, teachers’ classroom management skills improve and they are better equipped to minimize student misbehavior.  In contrast, unintentionally, higher-stakes teacher evaluation policies could increase the rate at which teachers send students to the office for misbehavior. For example, if teachers are stressed about how orderly their classroom seems during an unannounced observation, they might send more students out of class in the hopes that fewer disruptions occur during an evaluative visit. They might also send disruptive students out if they perceive that doing so will maximize the learning environment for their other students.

In this study, we tested whether higher-accountability teacher evaluation polices, implemented in response to the 2009 Race to the Top competition, affected how teachers responded to classroom misbehavior in the form of Office Disciplinary Referrals (ODRs). We examined this question using a sample of 2,564 schools, all of which were attempting to implement the Positive Behavioral Interventions and Supports (PBIS) framework throughout the 12 years of our sample.

Read More

Effects of a Reading Intervention and Mentoring Intervention for Ninth-Grade English Learners with Reading Difficulties

Sharon Vaughn, Leticia Martinez, Kelly J. Williams, Jeremy Miciak, Anna-Maria Fall, & Greg Roberts

PDF Version

How does an intensive reading intervention affect the reading achievement of ninth-grade English learners with reading difficulties?

Some English learners (ELs) in ninth grade have difficulty comprehending grade-level text, preventing them from learning content in other subject areas. To address this we implemented an intensive, multicomponent reading intervention for one year with ninth-grade ELs with reading difficulties. The intervention was provided in addition to core instruction and focused on word-reading, fluency, vocabulary, comprehension. Foundational skills and strategies were taught through explicit instructional methods, and students worked in cooperative learning groups to apply these skills and strategies to content area texts. ELs who received the reading intervention performed better on some measures of reading achievement (sentence-level fluency and comprehension, taught vocabulary words) than ELs who did not receive the reading intervention. On other measures, there were not substantial differences between the two groups (word-reading, untaught vocabulary words, and text comprehension).

 

Read More

The Effects of the Louisiana Scholarship Program on Student Achievement and College Entrance

Heidi H. Erickson, Jonathan N. Mills, and Patrick J. Wolf

PDF Version

How did the LSP affect student outcomes?

A new study estimates the average impact of the Louisiana Scholarship Program (LSP) on student achievement in math, English Language Arts (ELA), and science after four years of the program. After large initial declines in achievement in the first year of the program, students in grades 1-5 who used LSP scholarships won through lotteries regained some of their lost ground. However, they continued to perform below similar students who did not receive a scholarship by a lottery.

Read More

English Corequisite Remediation Improves Students’ Early Course Progression Outcomes but Does Not Increase Persistence Rates

Trey Miller, Lindsay Daugherty, Paco Martorell, and Russell Gerber

PDF Version

What are corequisites?

Faced with troubling evidence on the success of students who take traditional developmental education (DE) courses, states and higher education institutions across the United States are rethinking the way they address college readiness. Corequisite remediation is one promising and common approach to DE reform. Under corequisite remediation, students skip the traditional DE course(s) and move immediately into a foundational college-level course, while also being required to enroll in concurrent DE support in that same semester. Corequisites also call for changes to instruction to better align content in DE with college-level coursework and some models build in opportunities for more personalized support and/or peer support through various design features such as smaller class sizes and the mixing of college ready and DE students.

Read More