Measuring Collaboration in Complex Computerized Performance Assessments

• In Event: Developing Innovative Assessments of 21st-Century Skills: Online Inquiry and Collaboration

Abstract

Collaborative problem solving (CPS) has gained increasing attention in the assessment community with particular interest aimed at conceptualizing the construct, determining appropriate methods for making inferences about individuals’ CPS skills, and designing environments that provide sufficient opportunities for individuals to display CPS skills (Hao, Liu, von Davier, & Kyllonen, 2015; OECD, 2017; von Davier & Halpin, 2013). In the Measuring Collaboration in Complex Computerized Performance Assessments (MC3PA) NSF project, we have developed the in-task assessment framework (Authors, 2016) to address these three key areas in assessment. The framework outlines principles for instantiating the task, student, and evidence models in evidence-centered design (Mislevy, Steinberg, & Almond, 2003) when assessment involves more complex constructs in digital environments. In this paper, we describe the components of each of these models in the MC3PA project.

In the project, we designed a collaborative simulation-based task in electronics in which students work in groups of three, each on a separate computer, and each running a fully functional simulation of a portion of an electronics circuit. Each student can measure the voltage across, or the current through, his or her resistor using a simulated digital multimeter, and use an onscreen calculator. The individual simulations are linked together to form a complete circuit and the task levels involve altering that circuit to achieve a particular goal voltage value. Students communicate using a chat window and can “zoom out” to see one another’s circuits, but can only alter or make measurements on their own. As students work to achieve these goals, all of their relevant actions are time-stamped and logged to a database.

A CPS ontology (similar to a concept map) provides a theory driven representation of the targeted skills and their relationships, linking the skills to observable behaviors in the electronics task that would provide evidence of each skill. This representation provides a hierarchical student-evidence model that can connect fine-grained behavioral log data to higher-level constructs. The top level of the ontology provides generalizable construct definitions for CPS (e.g., sharing information). Each lower layer of the ontology becomes more specific with CPS interpreted within a domain (e.g., sharing status updates) and then within a given task environment in the domain (e.g., sharing the status of the resistance in a circuit). Links between the layers describe how behaviors at lower levels combine to imply cognitive behaviors at higher levels.

Pilot testing (n=31 dyads) with community college and university students examined which ontology-based skills were displayed. Two raters coded students’ discourse and actions for the display of nine CPS skills (i.e., maintaining communication, sharing information, perspective taking, negotiating, exploring and understanding, representing and formulating, planning, executing, monitoring). We observed evidence for each of the skills represented in the ontology. We will discuss skill profile differences among more and less successful teams, and the utility of the CPS ontology for other projects as it provides an anchor student model representation which connects to other components of assessment such as behavioral indicators of skills, scoring rules, and task affordances.

Authors

• Jessica Andrews Todd, Educational Testing Service

• Dr. Carol McGregor Forsyth, Educational Testing Service

• Andre A. Rupp, Educational Testing Service (ETS)

• Paul Horwitz, The Concord Consortium

• John Chamberlain, Center for Occupational Research and Development