Making Valid Instructional Decisions: Teaching Educators to Consider Validity Evidence

• In Event: Teaching on Assessment: Implications for Preparing Teachers

Abstract

Teachers make countless instructional decisions in the course of their work. They use planned formal and informal assessments to give them information about student thinking, content understanding, and possible misconceptions to inform those decisions (Brookhart & Nitko, 2008; Popham, 2003, 2014). Classroom teachers administer and interpret more tests, more regularly, than professionals in virtually any other field. The Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014; henceforth the Standards) define validity as “the degree to which accumulated evidence and theory support a specific interpretation of test scores for a given use” (p. 225; emphasis added). Thus, the Standards apply to instructional decisions, as well. Although teachers are included in its intended audience, the Standards specifically excludes classroom assessments from its scope “due to limitations of resources available for such assessments and also because the results of such assessments had limited consequences for students” (Plake & Wise, p. 6). However, students experience the consequences of classroom assessments on a daily basis. Considered individually, they may have limited consequences for students, but these consequences have aggregate effects that shape instruction and accumulate over time, both within and across classrooms. The Standards cannot easily be applied to classroom assessments, but their principles remain important for classroom teachers to consider (Wiliam, 2014).

Objectives and Purposes
This paper reports the development and validation of the Validity Evidence Instrument (VEI; see Appendix A), a tool that classroom teachers can use when they create or evaluate classroom assessments to consider validity concerns. The VEI is a short tool containing questions that classroom teachers can use to connect the five sources of validity evidence described in the Standards (AERA et al., 2014) to evaluate the validity of classroom assessments (Brookhart, 2003; Moss, 2003) and planned uses for the resulting data (Moss, 2013).

Method, Data Sources, and Findings
Development of the VEI began with a review of relevant literature (e.g., AERA et al., 2014; Fennell, Kobett, & Wray, 2017; Kane, 2016; Lane, 2014; Messick, 1989; Moss, 2003) and structured interviews with a purposefully sampled selection of five subject matter experts. Inductive analysis (Hatch, 2002) was used to generate subthemes and to develop indicators into questions for the VEI. SMEs provided feedback on the initial version of the VEI and minor revisions were made prior to piloting. Eleven K-12 classroom teachers that reported “little to no knowledge of validity” participated in the VEI pilot. Participants attended a 90-minute discussion on validity and its implications for classroom assessment, agreed to use the VEI for four weeks, providing feedback each week. Results indicated that 90% of respondents felt that the VEI was clear for each desired source of validity evidence. Multiple teachers wrote that the instrument was “very clear” and that there were “no changes recommended.”

Significance and Implications
This paper presents one tool to help teachers evaluate classroom assessments to support valid instructional decisions. Findings and implications for future research will be shared.

Authors

• Davis Gerber, Cleveland Metropolitan School District

• Matthew Ryan Lavery, South Carolina Education Oversight Committee

• Jonathan David Bostic, Bowling Green State University