Psychometric, Computational, and Interactional Issues in Designing Integrated Assessment and Learning Systems

• In Event: From Fixed Test to Endless Data Systems: Analyzing and Communicating Ongoing Formative Assessment

Abstract

In this paper the authors discuss the new challenges and opportunities for constructing digital system for ongoing formative assessment. The paper takes a predominantly technological view by starting with the technical affordances available to current systems designers (Behrens, et al. 2012; Behrens & DiCerbo, 2013) and enumerating corresponding advantages and disadvantages to psychometric, data analytic, data management, and human interaction perspectives. Consider, for instance, the new promise of data persistence. Data persistence is the notion that records of prior performance can exist electronically over time and be accessed for learning and assessment processes during periods removed from the initial data collection. At the psychometric layer persistent data has extremely positive implications including the possibility that earlier data can seed Bayesian priors and improve the efficiency of computer adaptive tests (CAT; Wainer et al., 2000 ) or parallel adaptive learning processes. This could be a significant social benefit if performance records could be transferred across institutional geographies (districts) and levels (secondary to tertiary) thereby breaking the current “dumb” system in which tests are almost universally given based on complete ignorance of prior performance. This assumption would not hold in the type of systems that serve as goal for PARC and Smarter Balance consortia.

On the other hand, however, significant social issues remain regarding the social and legal boundaries that need to be established to explain citizen rights and responsibilities with regard to their data: Who “owns” the data, can the school district or other agency keep in perpetuity? Do I have the right to suppress my earlier data? What are the motivational implications of increase information to establish potential tracking or other patterns? These challenges occur in the data analytic (how do we specify optimal data collection or work assignment strategies), data management (how do we communicate information across disparate systems?), and human computer interaction layers (what is the nature of the interaction being sought and how do we support that with technology).

In the closing section of the paper the authors discuss a process view of such ongoing formative systems and introduce the problems addressed by the subsequent papers.

Authors

• Quinn Nathaniel Lathrop, University of Notre Dame

• John T. Behrens, Pearson