Artifacts as Potential Indicators of Enacted Math Standards: Pilot Study Results

• In Event: 34.051 - Improving Indicators of K–12 STEM Education Policy and Practice: What Have We Learned?

Abstract

Purpose
This study explores the potential of using classroom artifacts as an indicator of the quality of classroom practice (Indicator 5). Classroom artifacts, which can include both assigned tasks from teachers and the responding student work, are useful in providing evidence about the nature of instruction available to students and the expectations that teachers have for their students. This study focuses on the feasibility and utility of an Artifact Indicator Protocol for Math (AIP-M).

Perspective
The study’s first contribution is to examine artifacts as a potential educational indicator. Because the target of inference is the larger institution (e.g., district, state, nation) and not the teacher, there are sampling and analytic implications.

The study’s second contribution is the explicit attention to the College and Career Ready Standards (CCRS) as shaping protocol design. A major intended use of indicators is to be able to track changes in instructional practice coinciding with the establishment of new standards efforts that make explicit the importance of specific content, skills, and disciplinary practices at different grades.

Methods and Data Sources
We design and implement a math artifacts protocol to address the following questions:
● What are the validity argument and the design characteristics of a CCRS-based protocol?
● To what extent does the protocol lead to reliable and valid scores and interpretations?
● To what extent are scores associated with student and teacher characteristics as well as assignment characteristics (e.g., purpose of the assignment and time of year it is given)?
● To what extent is the protocol sensitive to differences in assignment practices that occur antecedent and subsequent to the CCRS?

The protocol focuses on content covered and dimensions of assignment quality that include conceptual understanding, procedural skills, applications, and reasoning. A pilot study resulted in the collection of 185 unique middle school math classroom artifacts, some of which were collected prior to the CCRS movement and others afterwards. Scoring will take place August 2016.
Results
Descriptive analyses will include score distributions, variance components for rating scores by scale and dimension, as well as dimensionality across scales. Variance component analysis will include classroom, artifact type, district, grade level, and time of school year variables. We will also characterize content coverage with respect to grade-level expectations.
Additionally, coversheet data will be analyzed descriptively and narratively to better understand the classroom context in terms of the source of the assignment, the frequency of extended assignments and collaborative work, and modifications to assignments for subgroups (e.g., Students with Disabilities, English Language Learners).
Scholarly and Policy Significance
The findings from the pilot study will provide significant information to stakeholders as to the feasibility of developing an artifact-based system for Indicator 5: how new policies are being interpreted and enacted in classrooms through actual tasks set for students. Clear rubrics along with a library of exemplars can provide much needed support to teachers and their administrators as to how to translate the ideals presented in college and career ready standards into instructional practice.

Authors

• Drew H. Gitomer, Rutgers University - New Brunswick/Piscataway

• Jeanette Joyce, Rutgers University