A Framework for addressing Instrumentation Biases when using Observation Systems as Outcome Measures in Instructional Interventions

Mark White, Bridget Maher, Brian Rowan

PDF Version

Many educational interventions seek to directly shift instructional practice. Observation systems are used to measure changes in instructional practice resulting from such interventions. However, the complexity of observation systems creates the risk of instrumentation biases. Instrumentation bias is bias resulting from changes to the ways that an instrument functions across conditions (e.g., from pre-test to post-test or between control and intervention conditions). For example, teachers could intentionally show off intervention-specific practices whenever they are observed, but not otherwise use those practices. Alternatively, an instructional intervention could shift instruction in ways that increase observation scores without impacting the underlying instructional dynamics that support student learning.

This conceptual paper with a case study exemplar provides a validity framework for using observation systems to evaluate the impact of interventions. Inferences about an intervention’s impact generally involve determining whether a teaching practice has changed within some setting. Observation scores, the evidence for these conclusions, are specific raters’ views of how a rubric would describe observed lessons. The conclusions are far more generalized than the observation scores. The framework (see Figure below) systematically breaks down the processes necessary to operationalize an aspect of teaching practice and sample from a setting to obtain observation scores that can be generalized to draw conclusions.

Using the Framework to Improve Study Design. The framework helps researchers improve the study’s design by explicitly highlighting each important process in measurement. This allows researchers to intentionally engage with each process and carefully consider its challenges. In this way, the framework serves as a cognitive scaffold to support researcher thinking.

Using the Framework to Support Validity Arguments. Each process identified by the framework captures a necessary piece of the validity argument. The validity argument must show that each process did not influence the nature of the construct or the setting to which researchers wish to generalize. Further, the framework highlights many different levels to which results could be generalized, helping researchers be more precise about how generalizable their results are.

 


Full Article Citation:
White, M. C., Maher, B., Rowan, B. (2022). A Framework for addressing Instrumentation Biases when using Observation Systems as Outcome Measures in Instructional Interventions.   Journal of Research on Educational Effectiveness. https://doi.org/10.1080/19345747.2022.2081275.

 

Share this post:

Comments on "A Framework for addressing Instrumentation Biases when using Observation Systems as Outcome Measures in Instructional Interventions"

Comments 0-5 of 0

Please login to comment