Using Topic Maps to Survey Vocabulary Breadth in Specific Domains

• In Event: Roundtable Session 60
  In Roundtable Session: Depth and Breadth: Approaches to Assessing Partial Vocabulary Knowledge and Supporting Word Learning

Abstract

Goals/Purposes.
We describe assessments that measure vocabulary breadth across three domains: general academic, history/geography, and biology/ecology, and (i) describe applications of Natural Language Processing (NLP) techniques to sample vocabulary across topics within a subject domain; (ii) present psychometric results from a large-scale study validating estimates of word/topic associations; and (iii) examine variations in word/topic associations among students.

Perspectives/Theoretical Framework.
Vocabulary size is often measured by sampling across word frequencies (Carroll 1970, 1971, 1976; Breland, Jones, & Jenkins, 1994; Zeno et al. 1995; Breland & Jenkins, 1997). This assumption is problematic when applied to specialized vocabulary, (Carroll, 1971). Words are not distributed evenly across texts, but come in intercorrelated sets, particularly in Tier III or specialized domain vocabulary (Beck et al. 2002). We seek to model probabilities of word co-occurrence in a corpus through ‘topics’ (sets of strongly associated words). Sampling across topics (rather than solely by the frequency of words) is likely to yield improved measures of vocabulary breadth, while making it easier to identify specific vocabulary that should be targeted for instruction, given an intended reading domain.

Methods/Techniques.
NLP clustering and corpus analysis; item analysis (proportion-correct [p+] and point-biserial correlations); reliabilities; agreement between raters in Study 1; 3-parameter Item Response Theory (3PL IRT) analyses for the anchor test in Study 2; Equating; Correlation analysis; ANOVA.

Data Sources/Evidence.
Study 1. 30 expert educators were recruited for validation studies in biology/ecology and 46 for history/geography. Data was collected for 314 topics containing 15,700 target words, and 7,850 foil words. Of the target words, 9,066 were core elements in a topic hierarchy built with NLP techniques. Experts were asked to classify topic words as being ‘related,’ ‘somewhat related,’ or ‘not related at all’ to specific topics.

Study 2. 4,543 science and 4,164 social studies students were recruited from 7th, 9th, and 11th grade urban/suburban/rural classrooms in the US and were first administered the depth study tests described in the first study in this symposium. They were then administered a test form comprised of 5 topics and 200 vocabulary items such that they were required to judge whether each word for a given topic was ‘related,’ ‘not related,’ or ‘unknown.’ There were a total of 20 biology/ecology and 33 history/geography forms.

Results. In Study 1 there was a high rate of agreement between the topic hierarchy classifications and human raters (Table 1), and human agreement that foils were unrelated (Table 2). Percentage of agreement between experts was moderately correlated with a statistical association strength measure between words and topics. Study 2 data was collected May-June, 2011 and will be compared to the judgments of domain experts and at each grade level.

Table 1:
Target Words Related Somewhat Related
Biology/Ecology
Core Topic
Vocabulary 88% 9%

Marginal to Topic
Vocabulary 1 61% 28%

History/Geography
Core Topic
Vocabulary 86% 11%

Marginal to Topic
Vocabulary 1 62% 29%

1 Note: As determined by a statistical measure of association with the topic in a corpus of edited texts.

Table 2: Foils
Not Related
Biology 77%
History 71%

Authors

• Paul Deane, Educational Testing Service

• Rene R. Lawless, Educational Testing Service

• Robert Krovetz, Lexical Research

• Isaac I. Bejar, ETS

• Chen Li, ETS

• Tenaha P. O'Reilly, ETS

• Srinivasa Pillarisetti, ETS