Measuring Breadth and Depth of Study With Natural Language Processing and Learning Analytics

• In Event: Equity-Oriented Data Science: Leveraging Computation and Novel Data Sets for Higher Education Research and Practice

Abstract

The perceived importance of both breadth and depth of study motivate the general education and major curricular model that defines much of the academic landscape of the American post-secondary. Breadth is developed in general education coursework while depth is developed through specialization in a major field of study. Despite defining the academic core of most universities in the U.S., little research has explored the relationship between breadth/depth and the desired outcomes of higher education ranging from democratic engagement to labor market success. Failure to define and measure these notions leaves researchers and policy makers without the ability to assess the effectiveness of such an education. In part, this absence occurs because operationalizing fuzzy notions like a student’s breadth of study requires understanding the similarity of courses at a microlevel. All else being equal, the breadth contribution of a math course and a statistics course is less than that of a math course and an art history course because math and statistics typically have more similar course content than math and art history. We cannot assess the role of breadth and depth of study in a student’s educational experience without measures that capture this intuitive notion of similarity. Moreover, we cannot understand which students achieve breadth and depth of coursework and whether this kind of academic experience benefits all students equitably without methods to measure these concepts at scale.
I explore two intuitive definitions of course similarity that can allow us to characterize the breadth and depth of a student’s course enrollment trajectory. First, drawing on a corpus of course descriptions and word embedding models from natural language processing, I develop linguistic measures of course similarity: two courses are similar if they discuss similar content in their course descriptions. Second, drawing on administrative transcript records and learning analytics, I develop measures of course similarity based on course co-enrollment patterns: two courses are similar if they occur in similar locations on student transcripts. Using these measures, I characterize the development of breadth and depth over students’ enrollment trajectories at a large R1 and connect these measures to long term outcomes.

Author

• Annaliese Paulson, University of Michigan - Ann Arbor