Iterative Design of a Dashboard to Inform Instructional Choices

• In Event: Scaffolding Self-Regulation, Co-Regulation, and Socially Shared Regulation of Future Learning: Affordances of Learning Analytics Dashboards

Abstract

There is great interest in the use of data by teachers to inform instruction. Research has repeatedly shown that student achievement improves when teachers are provided with information about their students’ learning (Hattie, 2009). However, digital learning environments actually remove the teacher from direct interaction with students’ work products. If we are to find a balance between technology, teachers, and students in the classroom, we must find ways to communicate the information from digital activities to teachers in ways that help them make instructional decisions in the classroom.
In this presentation, we will report the results of a research study focused on how to present results from a variety of students’ digital activities to teachers in ways that support their instructional decision making. The study is part of a larger research project in which digital and non-digital activities were designed aligned to a learning progression on geometric measurement of area. Data were gathered from students’ interactions with software and reported to teachers via a Student Profile, which was designed to help teachers interpret the information on students’ mastery of the stages of the learning progressions in order to support their instructional decisions.
The Student Profile was developed using an iterative design process, including three rounds of testing with 8-12 teachers in each round. An analysis of themes across the rounds revealed three key design challenges:
1) determining granularity of stages (i.e., whether to show individual stages or summarize groups of them). Teachers in focus groups and interviews indicated that coarse-grained summaries prevented them from determining specific actions to take but alternately responded to detailed summaries by saying there was too much information to process;
2) communicating probabilistic data. Teachers consistently struggled to interpret displayed probabilities, often equating information like “a 72% likelihood of mastery” with a score of 72% on an assessment;
3) balancing overviews for quick decision making with access to details about student performance. Teachers simultaneously indicated: a) they had little time and wanted to make quick decisions and b) they wanted to dive into elements of an individual performance like the amount of time a particular student spent on a level.
Overall, most teachers in our sample were interested in sensemaking and impact (Verbert et al., 2013); they wanted answers to instructional questions and to gather support for decisions they needed to make. Three key instructional decisions were identified as areas where teachers could use information to inform instructional decisions: 1) what to review with the whole class (see Figure 1a), 2) how to group students for a small group lesson on a particular topic based on current mastery of particular skills (see Figure 1b, in which students can be placed into homogenous or heterogeneous groups depending on the sorting of the columns) and 3) which students need intervention regarding a particular misconception (see Figure 1c).
This qualitative work in the design process led to a deeper understanding of how tools should be designed to support decisions rather than to simply report data.

Authors

• Kristen E. Dicerbo, Pearson

• Daniel M. Belenky, Pearson Education, Inc.