Practical Challenges and Some Solutions for Conducting Automated Corpus-Based Analysis of Vocabulary

• In Event: 29.077 - Words for Schools: A Research Project Designed to Identify the Vocabulary That Students Need to Know and Educators Need to Ensure They Learn

Abstract

Purpose

The purpose of this paper is to identify practical challenges encountered when conducting corpus-based analysis of vocabulary and to propose solutions, including ones used in our team’s vocabulary study. Challenges addressed include identifying which sequences of characters should be considered “real words,” associating inflected and derived forms of words to their root-words, and differentiating senses of polysemous words.

Perspective

Early corpus-based vocabulary work involved limited sampling of texts and manual coding (e.g., Thorndike, 1921; Thorndike & Lorge, 1944). As larger corpora are created and analyzed, researchers must increasingly rely on automated approaches. There are important limitations to consider when using such automated approaches; therefore computational linguists sometimes employ manually constructed lexical resources augmented by automated procedures (Manning & Schütze, 1999).

Data Sources

The data sources were those reported by Bowen et al. in their symposium paper—2,175,000 tokens from four current, best-selling, core reading programs.

Method and Results

Here we address three challenges, each of which required a somewhat different method:

The first challenge was identifying which sequences of characters correspond to “real words.” Frequency data from the 450 million word Corpus of Contemporary American English (Davies, 2008) was used to filter out non-words. Any "words" in our data that were not in COCA were deleted.

Next, we had to determine a hierarchical morphological system. In the end, four levels of word family relatedness defined by Bauer and Nation (1993) were identified: Level 1—every word form is counted separately (no families); Level 2—root words and their inflected forms; Level 3—root words with most common derivational affixes; and Level 4—classical roots and classical affixes.

Then, we needed a valid, automated way of identifying word families. After exploring several options, data from CELEX2 (Bayeen, 1995) and an algorithm based on Morfessor (Virpioja, Smit, Grönroos, & Kurimo, 2013) were employed. Reliability was established using expert judgments on 1,000 randomly selected word-forms and was shown to be .95.

The final challenge was how to automatically identify the intended sense of polysemous words. Automatic word sense disambiguation algorithms were explored for this purpose. State-of-the-art performance in automatic word sense disambiguation is relatively poor (Navigli, 2009). One reason for the poor performance is the fine resolution of popular word sense inventories like WordNet (Felbaum, 1998). For example, the word ‘bank’ has 18 senses in WordNet. For practical purposes 4 or 5 is probably more reasonable. Several automated approaches for combining WordNet senses were evaluated and found to be inadequate for our purposes (Snow, 2007). Finally, a crowd-sourced application for combining similar senses within WordNet was used. We are currently evaluating this approach.

Scientific Significance

This paper proposes novel applications of state-of-the-art technology including semi-supervised automatic learning of morphological rules in English and automatic word sense disambiguation from the field of computational linguistics to an important and timely issue in education research. Using such technologies will make it possible to efficiently process very large corpora and determine which words students are likely to face in their reading.

Author

• Jeffrey Elmore, MetaMetrics