Misalignment in Data Use Processes: Lessons for Capacity-Building Efforts

• In Event: Sustainable Professional Development for Data-Informed Instructional Improvement

Abstract

Purpose and perspectives

Professional collaboration in schools has changed from focusing on narrow goals to embracing the many purposes of learning and human development (Datnow & Park, 2019; Hargreaves & O’Connor, 2018). However, national curricula vary in the support they provide to teachers in understanding and using standards. Furthermore, psychometric perspectives do not address differences in educational purposes and values across national contexts (Baird & Opposs, 2018). Successful implementation of data use innovations is therefore contingent on contextual factors such as teachers’ knowledge, skills, and attitudes; quality assessments; curriculum and assessment alignment; leadership support; (Kippers, Wolterinck, Schildkamp, Poortman, & Visscher, 2018), and teachers’ sensemaking processes (Bertrand & Marsh, 2015; Vanlommel & Schildkamp, 2019).

The purpose of this paper is to explore how misalignment affects data use practices in schools. Alignment research allows researchers to study the different components of an educational system to compare their content and make judgments about their agreement (Martone & Sireci, 2009). Coburn, Hill, and Spillane (2016) have encouraged research into the alignment of standards with professional development, curriculum materials, and teacher evaluation, arguing for theory building about the conditions that encourage or discourage instructional change. This case study first zooms in (Nicolini, 2009) on contextual factors affecting data use practices before zooming out on issues related to misalignment in data use.

Method

I conducted four hour-long group interviews, observed two data discussion meetings, and wrote field notes in a Norwegian lower secondary school participating in a literacy development project. Guided by the research question How are data use processes affected by alignment issues? and using open coding (Corbin & Strauss, 2008), I constructed descriptors and developed themes until they seemed saturated. Ethical approval was obtained from the Norwegian Centre for Research Data.

Results

Practitioners envisioned data use as a “carrier rocket” – a way to engage in collective capacity building. However, their ability to achieve this goal was limited due to the misalignment of several factors. There was misalignment between data literacy, professional values, knowledge of assessment constructs, the affordances of data management technologies, and curriculum documents. For example, technologies afforded attention to aggregated results and cut scores, leading practitioners to focus more on averages and less on sensemaking about disaggregated results. Furthermore, national curriculum documents represented standards through descriptions of disciplinary skills and knowledge. Misalignment between curriculum constructs and the data used to draw inferences about students was commonplace. Finally, previous experiences with collaboration on developing shared values hindered critical discussions about the meaning of data. This impacted the capacity to use data constructively for instructional change.

Educational Importance

Misalignment of components such as standards, data, and professional development affects the capacity for data use. Given the cost and expertise involved in conventional alignment studies (Martone & Sireci, 2009), building local capacity for evaluating data sets is important. While data literacy may be a key component in capacity-building, data use efforts should also consider the potential misalignment of contextual factors. Moreover, professional development efforts should be sensitive of teachers’ existing purposes and values.

Author

• Henning Fjørtoft, The Norwegian University of Science and Technology