Designing and Scoring Assessments that Involve Collaboration Among Students

• In Event: Excellence in Education Research: Early-Career Scholars and Their Work

Abstract

Traditional educational tests target a relatively narrow set of constructs compared to the range of competencies required for student success. One way to address this issue is through the use of performance-based assessments that have better fidelity to the situations in which students learn and are expected to demonstrate their knowledge. In this research I focus on one particular performance context -- collaborative problem solving (CPS) -- as an avenue for broadening what we can infer about student ability. The use of collaboration and group work for assessment purposes has a relatively long history, and also features prominently in current initiatives concerning the measurement of ``21st century skills''. However, fundamental questions remain about how to design performance assessments involving collaboration.

One central issue is whether and how to simultaneously measure student performance in a traditional content domain, such as math or science, in conjunction with CPS. For example, the PISA 2015 CPS assessment did not evaluate content domain knowledge, but conceptualized collaboration in a general problem solving context. On the other hand, recent reforms to educational standards have often called for the incorporation of collaboration and other ``non-cognitive skills'' within existing curricula. In this research, I address the problem of designing and scoring assessments that involve CPS but are anchored in a content domain, specifically mathematics.

Regarding task design, it is useful to consider the three-part conceptual framework described by Webb (1995). First, the purpose of the assessment is to measure group productivity (i.e., domain performance). The purpose implies goals of the group work, as understood by the student participants. In this case the goal is also group productivity — students are asked to perform at their maximum vis-a-vis some instructions for evaluating the quality of the group output. Third, the basic group process to be observed during the completion of the group work is the quality of participation among collaborators. There are many other group processes discussed by Webb (e.g., co-construction of ideas, conflict and controversy, giving and receiving elaborated help). While it is hoped that these processes can be facilitated through additional aspects of task design, I start with the issue of participation.

With these objectives in mind, I consider the following question: How can a conventional “one player” mathematics test question be adapted to a collaborative context. In pragmatic terms, the goal is to arrive at a number of ``recipes” for creating collaborative tasks from existing assessment materials. Clearly, this is not an ideal approach to designing group-worthy tasks. However, the reason for starting with existing assessment materials is to retain their strengths (e.g., established psychometric methods for evaluating reliability and fairness), while building towards more authentic and meaningful assessment contexts. The resulting tasks will necessarily represent a compromise between genuine group work and what we can measure well.

For example, the simplest method for incorporating collaboration into an existing mathematics assessment is just to change in the instructions while retaining the assessment materials. Concretely, two students could be presented with one copy of a math test and instructed that their performance will be evaluated based on only what they record on the test form. From the perspective of group-worthy tasks, this would be a worst case scenario. From the perspective of psychometric modeling, this would be the easiest case to deal with.

A slightly more interesting task might incorporate elements of a jigsaw or hidden profile. The basic idea is that each student sees some incomplete portion of the item stimulus, and must share the information they each possess to arrive at a solution. A third task type involves students collaborating to request the information they want to use to answer a question (e.g., in the form of hints). When hints are well devised, this invites students to co-construct the solution path. A fourth type of item involves questions with vector-valued answers. For example, instead of asking students to determine whether a line with a given slope and intercept intersects a certain point, we can turn the problem on its head by providing the point and asking one student to provide the slope and another the intercept. This type of task requires students to collaborate while devising an answer, not simply while obtaining the information to be used to devise an answer.

During the poster session, I’ll provide a live demo each of these item types using a web-based platform that I have been developing over the last 6 months, as well as summarize the results of a pilot study.

Author

• Peter F Halpin, New York University