Construct Validity and Evidence-Centered Design: Using Data From Digital Learning Environments to Evaluate, Build Theory, and Improve

• In Event: 32.079 - What Can Be Inferred From Trace Data? Current Methods to Triangulate and Validate Traces of Learning Behavior

Abstract

Objectives: In many situations, researchers must make due with “found” data—data that is available but not necessarily designed to provide evidence of a student’s knowledge, skill, or ability. When working from found data, validity is of central concern, and in this paper, we describe how Evidence Centered Design (ECD) can be used to build “evidentiary arguments” from data collected from users’ interactions in digital learning environments (Mislevy, Behrens, DiCerbo, & Levy, 2012).
Theoretical framework: Validity comes in a variety of forms, and for the purposes of this paper, we focus on construct validity, which entails assessing the degree to which a measure captures that which it is intended to. To provide a process for assessing construct validity in relation to measures of students’ use of an online system, ECD provides a variety of tools and routines for aligning (1) a latent trait or construct (e.g., persistence), (2) to tasks in which students are engaged, (3) to the data generated from those tasks (Mislevy & Haertel, 2006). Drawing on our work in developmental math courses in community colleges spread throughout the United States, we describe operationalizations of “productive” persistence (e.g., Yeager et al., 2013) and self-regulated learning strategies, such as strategy shifting around resource use over time (e.g., Zimmerman et al., 2011).
Methods: Using trace data, we aligned each data point with specific elements of tasks in which students were engaged and drew on prior research to clarify possible sources of construct irrelevant variance (i.e., other constructs that could be measured along with our focal construct; Messick, 1994). We used the statistical language R and the dplyr package to manipulate data and engineer features, and following this process, we fit multiple hierarchical linear regression models, using the lme4 package, to assess the predictive validity of individual and combinations of features and to evaluate mediating and moderating effects.
Data: Data came from community colleges using the same learning management system, Instructure’s Canvas platform. Data used for feature engineering and regression analyses also included pre- and post-assessments related to the focal domain (e.g., developmental mathematics).
Results: Using ECD, we identified learning behaviors related to productive and un-productive persistence. For example, one measure entailed assessing students’ use of learning resources, such a reading and practice activities, in relation to their assessment taking and performance. Students who consistently logged into an LMS and only took assessments, as opposed to availing him or herself of available resources, took assessments to higher degrees and did less well on individual assessments as well as in the course overall. An ECD tool, referred to as a “design pattern,” aided the research team in specifying key elements of the task and evidence models used to operationalize aspects of persistence and strategy use.
Scholarly significance: The scholarly significance of this work entails making explicit an approach, i.e., ECD, for developing measures that have high construct validity, which is important for practitioners and researchers working together to evaluate learning technologies, build knowledge, and improve learning environments.

Author

• Andrew E. Krumm, SRI International