Exploring Indicators of Consequential Validity Among CRP Measures in PK-12 U.S. Classrooms

• In Event: Use, Analysis, and Development of Classroom Assessments with a focus on Culturally Responsive Educational Practices

Abstract

Background: Well-validated classroom observation measures available to evaluate teachers’ use of culturally responsive practice (CRP) in PK-12 classrooms are relatively few and far between (López, 2018; Franco et al., 2022). In addition, there has been less focus on the measurement of CRP using an argument-based approach to validation (i.e., attention to the purpose and use of the measure, known as consequential validity; AERA et al., 2014; Messick, 1998). Given the scarcity of CRP measures and the lack of attention to examining consequential validity evidence for existing measures, we sought to address these gaps through a systematic review of the literature.

Method: This study reports findings from a systematic review of the research on the quantitative measurement of CRP using the 2020 PRISMA standards to document how CRP is operationalized and measured in PreK–12th grade classrooms in the United States. We use the four stages of establishing a validity argument (Kane, 2006), also known as an interpretation and use argument (IUA; Haertel, 2018; Kane, 2013), to guide the presentation of our results. In Aim 1, we report on evidence supporting the first two stages of IUAs: (1) Scoring and (2) Generalization, including reports on classical test theory metrics (e.g., reliability, content-, convergent-, discriminant-validity) and evidence of measurement invariance. In Aim 2, we shifted our focus to the latter stages of IUAs: (3) Extrapolation and (4) Use or Interpretation.

Results: The search retrieved 4,080 records across six databases. Ultimately, 27 studies were included in the review and assessed for evidence of validity and reliability: 8 observation measures, 4 student-report measures, and 15 teacher-report measures. See Table 1 for a summary of each measure’s response options, scoring procedures, validity, reliability, and measurement invariance reporting and Figure 1 for a visual representation of the psychometric quality analysis. Two of the student-report measures were assessed for measurement invariance. The Kumar (2019) measure demonstrated configural, metric, scalar, and first- and second-order latent factors intercept measurement invariance across the following four student racial groups. The Dickson (2016) measure reported that the non-variance sources in their second-order factor solution were less than 20% parameters, suggesting that the measurement of the CRP construct did not differ significantly between female and male students, Latinx and non-Latinx students, and immigrant and non-immigrant students.

Discussion: The results of the present study help us understand that the current state of the CRP measurement literature. The findings raise important questions regarding the role of classical test theory metrics, such as measurement invariance. The two studies reported invariance across student groups had disparate findings, raising questions about the role of classical test theory metrics, such as measurement invariance, in the measure of CRP. We discuss whether a case could be made that a lack of measurement invariance positions us to gain greater insight into a measure’s target population and setting. Alternatively, when assessing CRP in the context of racial-ethnic heterogeneity and within-class diversity, there is a counterargument for the necessity of measurement invariance to capture the same construct across all types of classrooms.

Authors

• Meredith Dentes Powers, University of Virginia - Charlottesville
  Presenting Author

• Catherine P Bradshaw, University of Virginia - Charlottesville

• Jessika Bottiani, University of Virginia - Charlottesville