Usability of New Partnership for Assessment of Readiness for College and Careers Item Types for English Learners

• In Event: Research and Development to Ensure Accessibility for the Partnership for Assessment of Readiness for College and Careers Assessments

Abstract

Purpose
The purpose of this study was to provide qualitative data illustrating accessibility (content and language) and usability performance from English Learners in a PARCC state. The research questions investigated in this study are outlined as follows:
1. Do ELs understand the task content and task format?
2. How do ELs use the pop-up glossary feature, and does this feature improve comprehension?
3. Do EL students report that the instructions and embedded supports were clear and understandable (without increasing linguistic or lexical complexity?
4. Have EL students had an opportunity to learn the content standards being assessed?
Background
While prior research has sought to identify issues (Butler & Stevens, 2001) and establish the validity of standards-based content assessments for ELs (Young et al., 2008), there remains a need to identify performance across current ELs at different levels of English language proficiency, including former ELs (e.g., ELs who are no longer receiving language services given they have been Reclassified Fluent English Proficient, or RFEP). Additionally, with the innovative item types and functionality demands embedded within standards-based content assessments, evidence for usability and performance patterns for ELs are largely missing from the literature base. As such, this study was designed to address these limitations and provide initial evidence to guide maintaining fairness and validity of these innovative item types and assessment implications for test takers who are ELs.
Methodology
Cognitive lab protocols focused on retrospective interview and usability protocols were developed to elicit information on students’ interaction with the test content and overall performance. Purposeful sampling was used to recruit students across grade and language proficiency levels, determined by students’ Overall Composite Proficiency scores on their English language proficiency assessment. See Table 1 for sampling information.
Table 1. English Learner Sample by content assessment, grade, and language proficiency group
ELA Math
3rd grade 7th grade 10th grade 4th grade 6th grade Algebra I
Beginner 3 3 3 2 2 2
Intermediate 4 3 4 3 3 4
Advanced 1 2 2 3 2 2
Reclassified Fluent English Proficient 2 2 1 2 2 2
Total Students (N = 59) 10 10 10 10 9 10

Data was collected in 1:1 cognitive laboratories lasting about 90 minutes. For the first 60 minutes, students took their abbreviated content assessment consisting of approximately 10 items; for the remaining time, students were interviewed to explain and clarify observations.
Results
Across grade level, content assessment, and language proficiency levels, many students reported experiencing some difficulty decoding and comprehending items written to the rigor of the Common Core State Standards. Detailed descriptions of student usability and performance will illustrate answers to the four research questions.
Significance
Implications for assessment developers include considerations for accessibility (i.e., comprehensibility) and innovative item design. School level implications include considerations regarding how to best prepare students to successfully interact with the content and language rigor, in addition to the technology-enhanced nature of the computer-based assessments written to the Common Core.

Author

• Danielle Guzman-Orth, Educational Testing Service