Filtered by tag: Policy Remove Filter

Quantifying ‘promising trials bias’ in randomized controlled trials in education

Sam Sims, Jake Anders, Matthew Inglis, Hugues Lortie-Forgues

PDF Version

Randomized controlled trials (RCTs) have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. Yet RCTs are only unbiased in expectation (on average across many RCTs).

Estimates of the effect size from specific RCTs will in general diverge from the true effect due to chance differences between the treatment and control group. In suitably powered trials, this imbalance tends to be small and statistical inference helps to control erroneous findings.

Read More

A recipe for disappointment: policy, effect size and the winner’s curse

Adrian Simpson

PDF Version

Effect size and policy

Standardized effect size estimates are commonly used by the ‘evidence-based education’ community as a key metric for judging relative importance, effectiveness, or practical significance of interventions across a set of studies: larger effect sizes indicate more effective interventions. However, this argument applies rarely; only when linearly equatable outcomes, identical comparison treatments and equally representative samples are used in every study.

Read More

Performance Evaluations as a Measure of Teacher Effectiveness When Implementation Differs

James Cowan, Dan Goldhaber, Roddy Theobald

PDF Version

Overview

We use statewide data from Massachusetts to investigate the school role in teacher evaluation. Schools classify most teachers as proficient but differ substantially in how frequently they assign other ratings. We show these patterns are driven by differences in the application of standards across schools, not by differences in the distribution of teacher quality.

Read More

The Effects of Higher-Stakes Teacher Evaluation on Office Disciplinary Referrals

David Liebowitz, Lorna Porter & Dylan Bragg

PDF Version

Discussions about teacher evaluation often focus on how evaluations can inform high-stakes job decisions. However, when designing teacher evaluation frameworks, policy makers and system leaders also explicitly seek to promote the development of teachers’ pedagogical skills. One clear emphasis in most measures of teacher practice is that teachers develop a safe and supportive classroom learning environment. Thus, as intended, introducing more intensive teacher evaluation practices could decrease the rate at which teachers send students to the office for misbehavior. This might happen if, through the cycle of observations and feedback, teachers’ classroom management skills improve and they are better equipped to minimize student misbehavior.  In contrast, unintentionally, higher-stakes teacher evaluation policies could increase the rate at which teachers send students to the office for misbehavior. For example, if teachers are stressed about how orderly their classroom seems during an unannounced observation, they might send more students out of class in the hopes that fewer disruptions occur during an evaluative visit. They might also send disruptive students out if they perceive that doing so will maximize the learning environment for their other students.

In this study, we tested whether higher-accountability teacher evaluation polices, implemented in response to the 2009 Race to the Top competition, affected how teachers responded to classroom misbehavior in the form of Office Disciplinary Referrals (ODRs). We examined this question using a sample of 2,564 schools, all of which were attempting to implement the Positive Behavioral Interventions and Supports (PBIS) framework throughout the 12 years of our sample.

Read More

English Corequisite Remediation Improves Students’ Early Course Progression Outcomes but Does Not Increase Persistence Rates

Trey Miller, Lindsay Daugherty, Paco Martorell, and Russell Gerber

PDF Version

What are corequisites?

Faced with troubling evidence on the success of students who take traditional developmental education (DE) courses, states and higher education institutions across the United States are rethinking the way they address college readiness. Corequisite remediation is one promising and common approach to DE reform. Under corequisite remediation, students skip the traditional DE course(s) and move immediately into a foundational college-level course, while also being required to enroll in concurrent DE support in that same semester. Corequisites also call for changes to instruction to better align content in DE with college-level coursework and some models build in opportunities for more personalized support and/or peer support through various design features such as smaller class sizes and the mixing of college ready and DE students.

Read More

Effects of apprenticeship on the short-term educational outcomes of vocational high-school students

José de Amesti & Susana Claro

PDF Version

Apprenticeship increase employment opportunities, but what about academic outcomes?  

Apprenticeship-based high schools increase employment outcomes compared to school-based-only vocational education: countries that implement apprenticeship have eased the school-to-work transition and reduced youth unemployment. Despite these promising benefits of apprenticeship, there is almost no evidence of its impact on students’ educational outcomes. In fact, some worry that spending less time at school could motivate students to leave the academic path to join the workplace instead, causing apprentices to achieve lower educational attainment.

Read More

Unpacking the Treatment Contrast in the Head Start Impact Study: To What Extent Does Assignment to Treatment Affect Quality of Care?

Summary by: Elizabeth Hentschel

PDF Version

Background

Attending high-quality early childhood care and education (ECCE) programs can lead to positive child outcomes across multiple developmental domains including cognitive and social-emotional. Children from low-income families rarely have access to high-quality ECCE – less than one quarter attend center-based childcare rated as high-quality. The National Head Start (HS) Program is one affordable (free) option to low-income families, but data from 2006 find that only 40% of National Head Start Programs are considered high quality. This study seeks to understand if children who were randomly selected and offered an opportunity to participate in Head Start enroll in higher quality ECCE than they otherwise would have.

Read More

Early College, Continued Success: Longer-Term Impact of Early College High Schools

Mengli Song, Kristina L. Zeiser, Drew Atchison, and Iliana Brodziak de los Reyes

PDF Version

What are Early College High Schools?

Early Colleges are small high schools designed to increase the opportunity for students–particularly students traditionally underrepresented in higher education­–to earn a postsecondary credential. To achieve this goal, they partner with colleges and universities to provide high school students with college experience with the expectation that all students will earn an associate’s degree or up to 2 years of college credits during high school at no or low-cost to their families. Early Colleges also provide a rigorous and supportive high school environment to help students navigate and succeed in college coursework.

Read More

The Choice Architecture of School Choice Websites

Summary by: Sophia Li

PDF Version

Displays of school information affect parents’ decisions and decision-making process

Parents increasingly choose between schools for their children to attend. The way in which school data - safety, resources, distance, and academic performance - is presented affects how parents rank and choose schools.

Read More

Gather-Narrow-Extract: A Framework for Studying Local Policy Variation Using Web-Scraping and Natural Language Processing

Kylie L. Anglin

PDF Version

Many education policy decisions are made at the local level. School districts make policies regarding hiring, resource allocation, and day-to-day operations. However, collecting data on local policy decisions has traditionally been expensive and time-consuming, sometimes leading researchers to leave important research questions unanswered.

This paper presents a framework for efficiently identifying and processing local policy documents posted online – documents like staff manuals, union contracts, and school improvement plans – using web-scraping and natural language processing.

Read More

The Methodological Challenges of Measuring Institutional Value-added in Higher Education

Tatiana Melguizo, Gema Zamarro, Tatiana Velasco, and Fabio J. Sanchez

PDF Version

Assessing the quality of higher education is hard but there is growing pressure for governments to create a ranking system for institutions that can be used for assessment and funding allocations.  Such a system, however, would require a reliable methodology to fairly assess colleges using a wide variety of indicators. Countries with centralized governance structures have motivated researchers to develop “value-added” metrics of colleges’ contributions to student outcomes that can be used for summative assessment (Coates, 2009; Melguizo & Wainer, 2016; Shavelson et al. 2016). Estimating the “value-added” of colleges and programs, however, is methodologically challenging: first, high- and low-achieving students tend to self-select into different colleges– a behavior that if not accounted for, may yield to estimates that capture students’ prior achievement rather than colleges’ effectiveness at raising achievement; second, measures considering gains in student learning outcomes (SLOs) as indicators at the higher education level are scant. In our paper, we study these challenges and compare the methods used for obtaining value-added metrics in the context of higher education in Colombia.

How to best estimate value-added models in higher education?

Read More