Investigating the Response Length of Video-Based Oral Communication Performance Tasks

• In Event: Developing and Evaluating the Psychometric Quality of Next-Generation Student Learning Outcomes Assessments

Abstract

Theoretical Framework and Study Purpose

Oral communication has been identified as an important skill by higher education (e.g., AAC&U, 2011) and workforce communities (e.g., Hart Research Associations, 2015). Consequently, there is a strong push to evaluate college students’ oral communication skills to determine whether they are prepared to enter the workforce upon graduating college. In developing an oral communication assessment, it is important to authentically measure both oral communication skills and behaviors (Morreale, et al., 2007). Accordingly, we need to ensure that the tasks or prompts associated with an open-ended speaking response are of an appropriate length for students to demonstrate these skills and behaviors and provide valid and reliable scores. This study seeks to determine the appropriate length of oral communication performance tasks in order to obtain valid test scores.

We will investigate the following research questions:
1. Are there significant differences on overall performance when varying amounts of total response time across the performance tasks?
2. Are there significant differences in performance on informational tasks of varying response length? On persuasive tasks of varying response length?
3. Are there significant differences in performance between informational and persuasive tasks of the same length? Of different length?
4. Does student performance ranking change when taking tasks of varying length?
5. Is there variance in student scores attributable to task, task length, and rater?
6. Does changing the task length across individual items impact reliability?

Methods

Four performance speaking tasks were administered to each student using a within-group design, meaning each student participated in each condition. Each of the tasks asks students to construct different messages including an informational and persuasive message, with varying amounts of time for preparation and response (see Table 1). All responses were recorded using a webcam.

Students also completed the Communication Competence Self-Report Questionnaire (CCSR, Rubin, 1985), and a Background Information Questionnaire. Data collection is currently in progress with the goal of collecting responses from 30 undergraduate students.

Once all of the data have been collected, two human raters will score each video-based response using a rubric measuring content, delivery, and holistic performance.

Data Analysis

Descriptive statistics will be calculated across the four tasks, on the holistic score and across the content and delivery dimensions. We will also evaluate the relationship between holistic performance and scores on the CCSR using a Spearman correlation to evaluate concurrent validity evidence. We will next evaluate each of the research questions in turn, using repeated measures ANOVA (RQ 1) and MANOVA (RQs 2 & 3), Wilcoxon’s signed rank test (RQ 4), and generalizability (G) and decision (D) studies (RQs 5 & 6).

Importance of Study

Oral communication is not easily measureable using traditional multiple-choice assessment and instead requires the use of audio- or video-recorded performance. Few studies have evaluated the impact of task length on student performance. This study will be valuable in guiding the development of standardized open-ended tasks for a next-generation oral communication assessment, which would allow for more comparable results across institutions of higher education.

Authors

• Kri Noel Burkander, Educational Testing Service

• Katrina Crotts Roohr, Educational Testing Service

• Liyang Mao, IXL Learning