Operationalizing Constructs within Validity Arguments for K-12 Teacher Observational Measures

• In Event: Validity Arguments for K–12 Teacher Observational Measures: Issues and Future Directions

Abstract

Explicit Attention to Concepts (EAC) and Student Opportunities to Struggle (SOS) were identified by Hiebert and Grouws (2007) as clusters of mathematics instructional strategies linked to improved student achievement. In the Researching Order of Teaching project, we developed the EAC & SOS Observation Instrument to quantify teachers’ implementation of EAC and SOS strategies in mathematics lessons. For use in large-scale classroom-based research, this instrument development has important constraints--including relatively few codes and maintaining saliency across a range of lesson content and instructional contexts. In our IUA, we identified two claims related to operationalizing the constructs as suspect: (1) the codes are relevant and representative of the target construct, and (2) the descriptors of performance levels for each code are relevant and appropriate. We next describe our method for operationalizing the constructs, providing an initial positive case for plausibility of the claims (Kane, 2004). This is followed by the results of a conceptual analysis (Machado & Silva, 2007; APA/AERA/NCME, 2014) to provide critical evaluation of plausibility (Kane, 2004).

We accept the definitions of EAC and SOS provided by Hiebert and Grouws (2007) as the universe of content (Cronbach & Meehl, 1955) and operationalized it through cycles of testing, discussion, and revision to ensure construct alignment. The cycle phases included identification of key properties of the constructs (Cronbach & Meehl, 1955), translation into observable indicators, creation of codes with descriptors for three performance levels, and development of explanations of the codes and exemplars for each of the performance levels. After testing the instrument with 12 videos, we added a round of revision for clarity.
The conceptual analysis investigates links between the constructs and the language of the observation instrument (APA/AERA/NCME, 2014; Machado & Silva, 2007). This conceptual analysis involved three phases: (1) making explicit the structure of the instrument and identifying its key conceptual and operational terms, (2) evaluating the conceptual clarity of these terms, and (3) examining the links between the instrument’s codes and the two constructs. The analysis revealed strengths in the operationalization of the constructs. First, representativeness of the codes is demonstrated by direct mapping to their intended constructs, and each property of the constructs is represented in the codes or their descriptions. Second, relevance and appropriateness of descriptors is demonstrated through meanings provided or exemplified in the guide. However, there is ambiguity around terms which have multiple or situated meanings. This leads to some overlap between the constructs (i.e., focus on concepts in EAC and important math in SOS). The instrument would benefit from more clarity around terms such as “concepts,” “connections,” “representations,” and “understanding.”

This conceptual analysis has significance in three ways. First, it provides initial evidence for this operationalization of the constructs. Second, it highlights the importance of situated meanings for conceptual clarity and points to possible revisions to enhance precision in quantifying the constructs. Third, it illustrates the use of conceptual analysis as a source of validity evidence to evaluate claims during instrument development.

Authors

• Angela Rae Crawford, Boise State University

• Michele Carney, Boise State University

• Joe Champion, Boise State University