Scoring Videotaped Performances

• In Event: Technical Issues in Performance Scoring

Abstract

Purpose
This study will determine the impact on scoring quality (consistency, accuracy, and generalizability) of time sampling video segments, varying the length of video segments, and varying the placement of the segment by addressing the following research questions:

1. How consistent, accurate, and generalizable are scores from time-sampled video segments? To what extent do segment scores predict whole video scores?

2. Does varying the length of the segment impact scoring quality or the relationship between segment/whole video scores?

3. Does varying the placement of the segment impact scoring quality, or the relationship between segment/whole video scores?

Theoretical Perspective
Demonstrations are frequently recorded on video to enable scoring of performance quality. However, there is currently little research to support decisions about different video scoring methods. Ho and Kane (2013) demonstrated that the reliability of the first 15-minutes of a videotaped teaching demonstration was 60 percent as large as that for the full performance. However, it is unknown to what extent varying the length of the segment (e.g., 10 minutes versus 15 minutes) or placement of the segment (e.g., at the beginning, middle, or end of the video) might impact scoring quality. These decisions have several implications, including scoring costs and the amount of time needed to report results.

Methods
This study involves 2 parts. In the first part, professional scorers will be randomly assigned to one of three groups that will evaluate an entire video of teacher performance, the first 10 minutes of that video, or the first 15 minutes of that video. The purpose of this part is to estimate the relationship between segment scores and scores from the entire video and to determine the impact on scoring quality of reducing the length of the video segment by as much as 33%. The second part of the study will focus on where the segment appears within the overall performance. Professional scorers will evaluate performance segments taken from the second or third portions of the video. For both studies, scoring quality will be investigated using multivariate generalizability analyses as well as traditional indices of rater agreement.

This study will use videos from the Measures of Effective Teaching (MET) project. Raters will be assigned to one of five groups: whole video, first 10 minutes, first 15 minutes, second portion of the video, or third portion of the video. Each rater will score 100 videos using two dimensions from a modified version of Charlotte Danielson’s Framework for Teaching rubric (Danielson, 2011).
 
Results
Results are not yet available. Scorers will be trained and complete their scoring in fall 2013, with all analyses completed during the winter.

Scholarly Significance
As technology becomes more accessible, videotaped performances will become commonplace, not just for the evaluation of teacher quality. However, scoring of these performances is time-consuming and expensive. Scoring approaches that represent efficiencies and cost savings while still preserving quality will be crucial to the success of such assessment approaches.

Authors

• Emily R. Lai, Pearson

• Catherine A. McClellan, Clowder Consulting