Building Validity Into the Development of the Adaptive Reading Motivation Measures

• In Event: Adolescent Reading Motivation: Building Stronger Interventions

Abstract

Purpose – In this paper, we describe how the Evidence-Centered Design assessment development process and modern psychometric methods have been used to support the development and ongoing validation of a new measure of adolescent reading motivation called the Adaptive Reading Motivation Measures (ARMM), an IES-funded initiative.

Theoretical Framework – Because reading motivation has the potential to influence literacy achievement, learning about and measuring reading motivation is crucial to designing interventions and measuring student responses to those interventions. Currently, measures of reading motivation are targeted to younger children, lack the psychometric properties that show validity and reliability, or have findings of efficacy that are inconclusive or contradictory (Unrau & Schlackman, 2006; Watkins & Coffey, 2004). Evidence-Centered Design is one of a family of highly structured test development approaches that require collection of evidence throughout the development process to support the validity of inferences made from assessment scores (Mislevy et al., 2003; Wilson, 2005). The need for new measures, as well as specification of two Evidence-Centered Design components, the student and task Models, have informed the initial stages of the development of the ARMM.

Data Sources and Methods – To specify the student model and define the construct of interest, we completed a comprehensive review of extant literature on adolescent reading motivation. Literature searches included peer-reviewed journals, conference papers, dissertations, and books. From the literature review, we created a domain model or construct map, showing relationships among the constructs and sub-constructs described in the literature. Building from the student model, initial task models describing what situations would elicit student behaviors relating to reading motivation were specified. In later stages of assessment development, we will fit multi-level item response models and/or cognitive diagnostic models to field test data, comparing and selecting one or more task models to be represented in the final version of the assessment.

Results – In the literature, we found 18 motivation constructs that have been measured and examined previously in the domain of reading. From these constructs, we created a domain model and construct map. We then developed four task models related to reading motivation, two focusing on student behaviors and two on student interests. Specification of student and task models represent the first, but critical, steps in creating a traceable logic path from definition of construct to creation of items, to collection of data to validate our assumptions about structure and measurement of adolescent reading motivation.

Scholarly Significance – While scholarship related to Evidence-Centered Design has been developing rapidly (Ewing et al., 2010; Hansen, Mislevy, & Sternberg, 2008; Shute & Zapata-Rivera, 2008; Bauer et al., 2003) the approach has not been applied extensively to areas outside of achievement and ability testing. This work represents a significant contribution to the understanding of the structure of adolescent reading motivation, as well as to the application of emerging theories related to test development and validation to this area of study.

Authors

• Neal M. Kingston, The University of Kansas

• Gail C. Tiemann, The University of Kansas

• Michael F. Hock, The University of Kansas

• Marcia H. Davis, Johns Hopkins University

• Stephen M. Tonks, Northern Illinois University