Capturing Effects of Knowledge and Metacognition in Scenario-Based Assessments

• In Event: Excellence in Education Research: Early Career Scholars and Their Work

Abstract

The Global Integrated Scenario-Based Assessment (GISA) was developed to evaluate local, global, and deep comprehension, while giving students a realistic purpose for reading (McCrudden, Magliano, & Schraw, 2010). The GISA goes beyond measures of literal and inferential comprehension, assessing students’ higher-order, complex literacy skills (Gordon Commission, 2013; NGA & CCSSO, 2010). The GISA measures five dimensions of literacy, including Print, Verbal, Discourse, Conceptual, and Social-level processes (Sabatini, O’Reilly, & Deane, in press). In the GISA, students read and answer questions about multiple sources of information, varying in modality, format, accuracy, etc., on a single topic (e.g., organic farming). We examined the impact of individual differences in prior topic knowledge and metacognition on GISA comprehension performance.
Theoretical Framework
Research indicates that reading comprehension is supported and constrained by relevant topic knowledge (e.g., McNamara, de Vega, & O’Reilly, 2007), but metacognitive ability might help students compensate for low knowledge (e.g., Schraw, 1998). Measuring students’ prior topic knowledge is increasingly important in scenario-based assessments, which focus on a specific topic domain. To account for the influences of topic knowledge and metacognition on reading comprehension as measured by GISA, these variables were measured with a topical vocabulary test (O’Reilly & Sabatini, in prep), which asked students to identify whether certain words (e.g., “runoff”, “contamination”) were related to the assessment topic. In addition to making a substantive (yes/no) response, students could respond “I don’t know” (IDK) to these items. IDK responses reflect students’ metacognitive awareness—their knowledge about their (lack of) knowledge of a topic. We examined the effects of metacognition on GISA comprehension, after controlling for topic knowledge, to evaluate the degree to which metacognitive ability supports reading comprehension in situations where students have limited knowledge of text topics.
Data Sources
A total of 1,271 sixth and seventh graders were administered the Organic Farming GISA in Fall 2012 (see Table 1). Data from 1,189 students who completed the test was examined. Psychometric analyses (see Table 2) suggested that topic knowledge (29 items, Cronbach’s alpha = 0.75) and reading comprehension (35 items, Cronbach’s alpha = 0.86) were measured reliably.

Author

• Jesse R. Sparks, Educational Testing Service