Filtered by tag: RCT Remove Filter

Conjuring power from a theory of change: The PWRD method for trials with anticipated variation in effects

Timothy Lycurgus, Ben B. Hansen, and Mark White

PDF Version

Many efficacy trials are conducted only after careful vetting in national funding competitions. As part of these competitions, applications must justify the intervention’s theory of change: how and why do the desired improvements in outcomes occur? In scenarios with repeated measurements on participants, some of the measurements may be more likely to manifest a treatment effect than others; the theory of change may provide guidance as to which of those observations are most likely to be affected by the treatment.


Figure 1:
Power for the various methods across increasing effect sizes when the theory of change is correct.  

Read More

ICUE Intervention Improves Children’s Understanding of Mathematical Equivalence

Jodi L. Davenport, Yvonne Kao, Kristen Johannes, Caroline Byrd Hornburg, and Nicole M. McNeil

PDF Version

Does the ICUE intervention improve math learning?

Yes, second grade students in classrooms using the Improving Children’s Understanding of Equivalence (ICUE) materials and lessons scored higher on measures related to mathematical equivalence, including equation solving and conceptual problem solving. These higher scores came with no observable trade-offs in computational fluency.

Read More

A Cautionary Tale of Tutoring Hard-to-Reach Students in Kenya

Beth Schueler, Daniel Rodriguez-Segura

PDF Version

What was this study about?

Covid-19 school closures have generated significant interest in tutoring to make up for lost learning time. Tutoring is backed by rigorous research, but it is unclear whether it can be delivered effectively remotely. We study the effect of teacher-student phone calls in Kenya when schools were closed. Schools (j=105) were randomly assigned for 3rd, 5th and 6th graders (n=8,319) to receive one of two versions of a 7-week weekly math intervention—5-minute accountability checks or 15-minute mini-tutoring sessions—or to the control group.

Read More

How Do the Impacts of Healthcare Training Vary with Credential Length? Evidence from the Health Profession Opportunity Grants Program

Daniel Litwok, Laura R. Peck, and Douglas Walton

PDF Version

How do the earnings impacts of healthcare training vary?

This article explores how earnings impacts vary in an experimental evaluation of a sectoral job training program. We find that over the first two years in the study, those who completed long-term credentials (defined as college degrees or certificates that require a year or more of classes to earn) had program impacts that were about $2,000 larger per year than those who did not complete long-term credentials (whether they completed a short-term credential or no credential at all). A possible explanation for this finding is that those who earned a long-term credential had different experiences in the program, including more engagement with support services, and different post-program outcomes, such as greater employment in high-wage healthcare occupations like registered nurse.

Read More

Effects of Cross-Age Peer Mentoring Program Within a Randomized Controlled Trial

Eric Jenner, Katherine Lass, Sarah Walsh, Hilary Demby, Rebekah Leger, and Gretchen Falk

PDF Version

How does a cross-age peer mentoring program affect ninth-grade outcomes?

Ninth-grade students who were offered Peer Group Connection High School (PGC-HS) were less likely to receive a suspension or disciplinary referral and self-reported higher levels of school engagement and postsecondary expectations. However, offering the program had no effect on academics (credit attainment, attendance at school, GPA) and other non-cognitive skills (e.g., decision-making skills).

Read More

Examining the Impact of a First Grade Whole Number Intervention by Group Size

Ben Clarke, Christian Doabler, Marah Sutherland, Derek Kosty, Jessica Turtura, and Keith Smolkowski

PDF Version

The importance of early mathematics

The importance of a successful start to learning mathematics has been a national priority for several decades. Mounting evidence indicates that trajectories of mathematics performance are established early and remain relatively stable across time. This may in part be due to substantial disparities in young students’ access to early mathematics experiences and instruction with preschool-aged students from upper- and middle-class backgrounds already outperforming their economically disadvantaged peers.

Read More

Raising Teacher Retention in Online Courses through Personalized Support. Evidence from a Cross-national Randomized Controlled Trial

Davide Azzolini, Sonia Marzadro, Enrico Rettore, Katja Engelhardt, Benjamin Hertz, Patricia Wastiau

PDF Version

Does providing teachers with personalized support help them complete online training courses?

Yes, but not for all and not everywhere. The TeachUP policy experimentation found large effects of personalized support on course completion in nine European Union Member States among professional (i.e., in-service) teachers (+10.6 percentage points), but not among student teachers. Moreover, no effects are found in Turkey. More studies are needed to investigate the contextual and learner characteristics that drive the heterogeneous effects.

Read More

Does Early Mathematics Intervention Change the Processes Underlying Children’s Learning?

Summary by: Wen Wen

PDF Version

What are “state-” and “trait-” math achievements in early education?

Interventions can boost early math skills, but the role of these early skills on later math achievement is unclear. Consider that students who demonstrate stronger early math skills tend to demonstrate stronger later math achievement, yet some interventions that improve early math skills do not improve later math achievement – that is, the early benefits fade substantially after 2 or 3 years.

Read More

Design and Analytic Features for Reducing Biases in Skill-Building Intervention Impact Forecasts

Daniela Alvarez-Vargas, Sirui Wan, Lynn S. Fuchs, Alice Klein, & Drew H. Bailey

PDF Version

Despite policy relevance, long term evaluations of educational interventions are rare relative to the amount of end of treatment evaluations. A common approach to this problem is to use statistical models to forecast the long-term effects of an intervention based on the estimated shorter term effects. Such forecasts typically rely on the correlation between children’s early skills (e.g., preschool numeracy) and medium-term outcomes (e.g., 1st grade math achievement), calculated from longitudinal data available outside the evaluation. This approach sometimes over- or under-predicts the longer-term effects of early academic interventions, raising concerns about how best to forecast the long-term effects of such interventions. The present paper provides a methodological approach to assessing the types of research design and analysis specifications that may reduce biases in such forecasts.

What did we do?

Read More

Quantifying ‘promising trials bias’ in randomized controlled trials in education

Sam Sims, Jake Anders, Matthew Inglis, Hugues Lortie-Forgues

PDF Version

Randomized controlled trials (RCTs) have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. Yet RCTs are only unbiased in expectation (on average across many RCTs).

Estimates of the effect size from specific RCTs will in general diverge from the true effect due to chance differences between the treatment and control group. In suitably powered trials, this imbalance tends to be small and statistical inference helps to control erroneous findings.

Read More

Using a Factorial Design to Maximize the Effectiveness of a Parental Text Messaging Intervention

Catherine Armstrong Asher, Ethan Scherer, James S. Kim

PDF Version

What features of text messaging campaigns for early elementary families might increase their effectiveness?

Text messaging interventions are an increasingly popular way to support students and their families. We compared how three features of text messages, sent to parents, affect the reading behavior and test scores of their early elementary school children:

Read More

KIPP Middle Schools Increase Students’ College Enrollment Rates

Ira Nichols-Barrer, Maria Bartlett, Thomas Coen, & Phil Gleason

PDF Version

Do KIPP Middle Schools Boost Long Run Student Outcomes?

Yes they do, according to a rigorous national study of 13 KIPP middle schools. Building on prior studies of KIPP that show KIPP middle schools have strong positive effects on students’ middle school achievement, this study found that KIPP middle schools also improve longer-term rates of enrollment in four-year college programs. Winning a lottery-based admissions offer to a KIPP middle school increased a student’s probability of enrolling in college by 7 percentage points, even though a third of these students never enrolled at KIPP. Adjusting for enrollment, attending KIPP increased college enrollment rates by 13 percentage points. This boost is similar in size to nationwide disparities in college enrollment across racial groups—a relevant benchmark since nearly all KIPP students are Black or Latinx. 

Read More

We Have Skills, Effective and Efficient Social Skills Instruction for Early Elementary

Keith Smolkowski, Hill Walker, Brion Marquez, Derek Kosty, Claudia Vincent, Carey Black, Gulcan Cil, & Lisa A. Strycker

PDF Version

Can Social Skills Instruction be Efficient and Effective?

Yes. A rigorous study shows that the We Have Skills program efficiently and effectively taught the academically related social skills needed for early elementary students to succeed in school. We Have Skills appealed to children, and teachers quickly mastered and readily implemented the program in their classrooms.

Read More

Experimental Impacts of a Preschool Intervention in Chile on Children's Language Outcomes: Moderation by Student Absenteeism

Summary by: Hang (Heather) Do

PDF Version

What was this study about?

Chronic absenteeism (missing more than 10% of school days or more in one year) negatively impacts children’s school achievement and development. Yet, little is known about how absenteeism influences the effectiveness of interventions. In this study, the authors examined whether absenteeism affected the impacts of an intensive two-year professional development (PD) intervention aiming to improve the quality of Chilean public preschool and kindergarten and enhance the language and literacy outcomes of participating children (UBC (Un Buen Comienzo/A Good Start)).

Read More

The effect of embedding formative assessment on pupil attainment

Jake Anders, Francesca Foliano, Matt Bursnall, Richard Dorsett, Nathan Hudson, Johnny Runge, and Stefan Speckesser

PDF Version

What is formative assessment?

'Formative assessment', often used interchangeably with the term 'assessment for learning' and in contrast to 'summative assessment', refers to assessment activities undertaken by teachers – or students themselves – to obtain evidence which is then used to adapt teaching and learning methods to meet student needs and improve learning outcomes.

Read More

Effects of a Reading Intervention and Mentoring Intervention for Ninth-Grade English Learners with Reading Difficulties

Sharon Vaughn, Leticia Martinez, Kelly J. Williams, Jeremy Miciak, Anna-Maria Fall, & Greg Roberts

PDF Version

How does an intensive reading intervention affect the reading achievement of ninth-grade English learners with reading difficulties?

Some English learners (ELs) in ninth grade have difficulty comprehending grade-level text, preventing them from learning content in other subject areas. To address this we implemented an intensive, multicomponent reading intervention for one year with ninth-grade ELs with reading difficulties. The intervention was provided in addition to core instruction and focused on word-reading, fluency, vocabulary, comprehension. Foundational skills and strategies were taught through explicit instructional methods, and students worked in cooperative learning groups to apply these skills and strategies to content area texts. ELs who received the reading intervention performed better on some measures of reading achievement (sentence-level fluency and comprehension, taught vocabulary words) than ELs who did not receive the reading intervention. On other measures, there were not substantial differences between the two groups (word-reading, untaught vocabulary words, and text comprehension).

 

Read More

The Effects of the Louisiana Scholarship Program on Student Achievement and College Entrance

Heidi H. Erickson, Jonathan N. Mills, and Patrick J. Wolf

PDF Version

How did the LSP affect student outcomes?

A new study estimates the average impact of the Louisiana Scholarship Program (LSP) on student achievement in math, English Language Arts (ELA), and science after four years of the program. After large initial declines in achievement in the first year of the program, students in grades 1-5 who used LSP scholarships won through lotteries regained some of their lost ground. However, they continued to perform below similar students who did not receive a scholarship by a lottery.

Read More

English Corequisite Remediation Improves Students’ Early Course Progression Outcomes but Does Not Increase Persistence Rates

Trey Miller, Lindsay Daugherty, Paco Martorell, and Russell Gerber

PDF Version

What are corequisites?

Faced with troubling evidence on the success of students who take traditional developmental education (DE) courses, states and higher education institutions across the United States are rethinking the way they address college readiness. Corequisite remediation is one promising and common approach to DE reform. Under corequisite remediation, students skip the traditional DE course(s) and move immediately into a foundational college-level course, while also being required to enroll in concurrent DE support in that same semester. Corequisites also call for changes to instruction to better align content in DE with college-level coursework and some models build in opportunities for more personalized support and/or peer support through various design features such as smaller class sizes and the mixing of college ready and DE students.

Read More

Impacting 9th Grade Educational Outcomes: A Randomized Controlled Trial of the BARR Model

Trisha Borman, Johannes Bos, So Jung Park & Amelia Auchstetter

PDF Version

BARR students earn more core course credits, perform better on math standardized tests, and report better in-school experiences

The transition from eighth grade to ninth grade is a critical point for students. It can set them on a path toward successfully graduating from high school or dropping out. A growing number of schools have sought to support ninth-grade students through the Building Assets, Reducing Risks (BARR) model. BARR focuses on building in-school relationships and using students’ strengths to improve student outcomes.

Read More

The Effects of Teacher Professional Development on Children’s Attendance in Preschool

Summary by: Jonathan Seiden

PDF Version

What was this study about?

When children are absent from early childhood education (ECE) at centers and preschools they are unable to fully realize the positive effects ECE can have on their lives. Younger children and those from families with lower income are more at risk for absenteeism and may benefit most from ECE. Therefore, efforts to reduce absenteeism could have greater-than-average benefits for these students.

Read More