Assessing Breadth of Academic Vocabulary: Testing “New” Words From Across the Fourth-Grade And Fifth-Grade Curriculum

• In Event: Roundtable Session 60
  In Roundtable Session: Depth and Breadth: Approaches to Assessing Partial Vocabulary Knowledge and Supporting Word Learning

Abstract

Goals/Purposes
The assessments described here are an attempt to use what is known about vocabulary learning as an incremental and multifaceted process (Nagy & Scott, 2000). The tests capture elements of the complexity of word learning and are tied to the curriculum being taught in schools. This paper documents a textbook corpus-based process for selecting words that test the breadth of upper elementary students’ academic vocabulary.

Perspectives/Theoretical Framework
In their discussion of vocabulary assessment, Pearson, Hiebert & Kamil (2007) contend that the selection of arbitrary words for norm-referenced standardized tests is driven by
psychometrics, tradition, and convenience, rather than by our knowledge of word learning. They call for research that pays attention to distinctions among the vocabulary of different text genres, pays attention to features of words, and addresses the issue of curriculum mastery and incrementality in word learning. Nagy & Hiebert (2009) identified eight important features in selecting vocabulary for instruction: frequency, dispersion, morphological relatedness, semantic relatedness, familiarity, conceptual difficulty, role in a particular text, and role in the larger curriculum. While some of these aspects are less important for vocabulary assessment (i.e., role in text), basing word selection on an underlying theory of vocabulary acquisition provides better guidelines both for choosing words for instruction and for developing tests.

Methods/Data Sources/Results
Three factors informed word selection for the assessments presented: 1) students’ opportunities to learn new words at school; 2) attributes of words; and 3) psychometric properties of the words in pilot tests. Examining students’ opportunities to learn words in fourth- and fifth-grade classrooms led to a preliminary corpus of over 40,000 words. Selecting only words that were found in two or more source documents reduced the word bank to 1022 math words, 1911 science words, 10567 social-studies words, and 9855 language-arts words that experienced teachers judged to be conceptually “new” for the typical student.

The attributes of these word-bank words determined their eligibility for pilot tests. The factors used to screen the words included dispersion across source texts and subjects, word frequency, morphological family size, and grade level. Applying these constraints, a randomly generated sample of 125 math words, 125 science words, 125 social-studies words, and 250 language-arts words became candidates for pilot tests. The distribution of sample words across subjects roughly reflected the distribution of instructional time across subjects in the typical elementary-school day. In 2010-2011, we pilot-tested 324 of the sample words.

In summer 2011, we will use the psychometric properties of the pilot-tested words in conjunction with additional word attributes (parts of speech, polysemy, concreteness/ abstractness, morphological family frequency, and Spanish-English cognate status) to select words for the penultimate versions of these tests. This process, driven by students’ exposure to words in classrooms, word characteristics, and psychometrics, is one attempt to capture the breadth of students’ academic vocabulary knowledge in a standardized assessment format.

Authors

• Susan Leigh Flinspach, University of California - Santa Cruz

• Judith A. Scott, University of California - Santa Cruz

• Jack L Vevea, University of California - Merced