A Theory of “Practical Measurement”: Collecting Data to Support Improvements to Educational Practice and Theory

• In Event: Bridging Research and Practice With “Practical Measurement”: The Case of Improving Developmental Mathematics in Community Colleges

Abstract

This presentation will introduce the science of continuous practice improvement and the associated need for “practical measures.”

In general, the new educational R&D infrastructure requires an engineering orientation in which adaptability to local contexts is a direct object of study. In this regard, knowing that a program can work is not good enough; we need to know how to make it work reliably over many diverse contexts and situations (Bryk, 2009). This means accumulating a rigorous knowledge base on practice improvement where the real test of adequacy is its capacity to advance demonstrable, broad-based improvements in teaching and learning.

To achieve this aim, this talk focuses on building a theory of “practical measurement.” In this framework, the data that we collect needs to be organized around some working theory about how various processes and organizing routines interact to affect desired outcomes, and for whom they do so. We need to articulate these causal chains from reform designs to valued outcomes, and we must continually evaluate this logic as new evidence emerges from field use. This means shifting the evaluation question. Instead of a one-time summative assessment, we need to continually interrogate with more detailed, ongoing evidence as to the kind of effects that are likely, for whom, and under what sets of circumstances. Moreover, instead of having a “pet theory” and testing how many outcomes it may predict, it means shifting to understanding the various influences on a set of educational outcomes and attempting to discover methods for improving them for diverse groups of students. Last, instead of estimating only effects on average, this approach entails a focus on examining for whom an effect occurs--and for whom it does not.

More concretely, we argue that traditional measurement approaches have these features that make them impractical for improvement.
● Researchers usually collect too much data--more than they analyze adequately, and more than they use to improve practice.
● They tie measurement approaches to academic theories, not theories of primary drivers of key problems that are recognizable to practitioners and that can motivate their improvements.
● They design measures to understand average effects, not heterogeneity of effects.

By contrast, practical measurement is:
● Tied to theories of solving problems, in addition to academic theories.
● As brief, unobtrusive, and surreptitious as possible
● Recognizable and of high face-validity to practitioners
● Also recognizable to theoretical researchers
● Used to inform practice and understand variability
Moreover, the use of practical measurement is facilitated when it is open and usable by the community for free (e.g., licensed under Creative Commons).

In the talks that follow, we introduce a new practical measure that has all of these features: it is brief; it was co-developed with students, practitioners, and researchers; it is free and open for anyone to use; and it is administered in the context of an ongoing effort to improve community college student success.

Authors

• Anthony S. Bryk, The Carnegie Foundation for the Advancement of Teaching

• Paul LeMahieu, The Carnegie Foundation for the Advancement of Teaching