Designing Interactive Assessment Reporting Tools Based on Extended User Feedback

• In Event: From Fixed Test to Endless Data Systems: Analyzing and Communicating Ongoing Formative Assessment

Abstract

In this paper we discuss a 7 month designed-based research (Anderson & Shattuck, 2012) project undertaken with the goal of understanding the affordances and limitations of a learning support system currently in use, and iteratively design and make recommendations for the improvement of the communication of data to students and instructors. We applied approaches from the Human-Computer-Interaction (HCI) literature including Contextual Inquiry Interviews, field observations, and cross-disciplinary Affinity Analysis. These methods have corollaries in the psycho-educational literature in methods such as design-based research (Anderson & Shattuck, 2012) for general approach, protocol analysis (Ericsson, 2006) for expert knowledge elicitation, and the application of grounded theory (Straus & Corbin, 1990) for theme identification.

During Phase I of the research, 20 Contextual Interviews (observation and interviews) were conducted across 6 different university settings in 5 different states to understand how individuals interact with, understand, and use a digital learning support system consisting of practice assignments along with learning aids of various types. Participants included instructors (10), students (5), administrators (3) and support staff. Thematic analysis of interview and observational field notes led to the generation of 780 notes, 11 data flow models, 15 sequence models and 10 cultural models. In the area of support for human judgment, key results include identification of instructor use of analog workarounds, importance of data quality and completeness and the need for data to support otherwise unstructured human decision making. In the area of Human-Technology relationship, key results include identification of points of frustration creation, threats to existing roles, difficulty in data access and use, as well as optimism regarding future application and evolution of the technology. With regard to use of data to improve student success, key findings emphasize importance of early identification to aid retention, the value of self-assessment to motivate students, the increased value of new forms of data, the necessity of multiple types of data and the importance of customizable data. Interviews and collaborative interactions with key stakeholders in a large educational-data organization as well as university-based HCI experts confirmed the value of these findings.

During Phase II we took the findings from Phase I and generated a series of User Experiences through a cyclical process of design, rough creation, customer exploration and feedback, re-design, re-creation, re-exploration and so on. As cycles advanced, interface experiences evolved from illustrative to minimally functional to functionally interactive. The target system is constructed to allow for the rapid navigation across the complex multi-level and multi-source data structures inherent in institutionally embedded instruction, by students, instructors and administrators. Special attention was necessary to address the need to simultaneous understand broad stroke (forest) and detail (tree) level implications. Numerous experiential changes were made in response to user observation and interview. Demonstration of the final interface and enumeration of the design principles inherent in the prototype will be addressed.

Authors

• Stephanie E Butler, LightSide Labs

• Paul Mandel, Carnegie Mellon University

• Auldyn Matthews, Red Privet, LLC

• KeVon Ticer, Carnegie Mellon University

• Nina Xu, Carnegie Mellon University