Practical Measures in Educational Improvement: Overview and the Student Agency Improvement Community Case

• In Event: Hilton Roundtable Session 13
  In Roundtable Session: Practical Measures in Service of Instructional Improvement: Attending to Issues of Rigor

Abstract

This presentation (1) provides an overview of and (2) introduces a case of the use of practical measures in improvement science methods. Improvement science is a method of improving organizational systems that utilizes testing in successive learning cycles to improve the processes that make up a particular system (Bryk et al., 2015; Langley et al., 2009). Improvement science emphasizes the voices of practitioners in identifying an aim, testing change ideas, and using measures to understand whether changes are improvements. Integral to this approach is the use of practical measures that provide timely and specific information to those who are working to improve. Practical measures are “practical” in that they can be collected, analyzed, and used within the daily work lives of practitioners (Bryk et al., 2015; Yeager et al., 2013). Bryk et al. (2015) detail eight qualities of practical measures. Practical measures must 1) operationalize a working theory of improvement; 2) be specific to the work processes that are the object of change; and 3) produce data accessible in a timely manner while placing minimal burden on practitioners. The usefulness of these measures depends on the measures 4) being in a language meaningful to those engaged in the work; 5) being sensitive to change; 6) having predictive power for outcomes; and 7) having formative value in signaling subsequent action. Finally, 8) social routines secure the trust and openness necessary to sustain meaningful change efforts.

Student Agency Improvement Community (SAIC) provides an illustrative case of the use of practical measures in the context of improvement science. SAIC is a community of six networks convened by the Carnegie Foundation for the Advancement of Teaching with the goal of joining academic research from the psychology of learning with improvement science methods to equip students to persist when facing learning challenges. Two central types of practical measures were used by the SAIC community.

The first example is a short student survey on perceptions of agency. The process of survey development included piloting a longer survey to identify a short list of key items using both statistical methods and expert knowledge from the field; conducting cognitive interviews to ensure students understood items in intended ways; developing processes of turning raw data into reports; and conducting predictive validity analyses to link the survey to key student outcomes.

The second type of practical measures are classroom measures of student behavior, such as the rate at which students attend class, complete their assignments, and revise coursework. These measures came from particular classroom processes that were the objects of change in teachers’ improvement efforts. They highlight how student agency manifested in classroom behaviors and therefore where teachers could make changes to influence agentic beliefs. The relationship between these measures and the student agency survey provide insight into the validity of these classroom practical measures.

We will articulate how the development and use of these two sets of measures map on to the eight qualities of practical measures, highlighting steps taken to ensure rigor.

Authors

• Sola Takahashi, WestEd

• Jon Norman, Carnegie Foundation for the Advancement of Teaching