Text & Measurement: Critical Approaches to Measuring Text & Measuring Our Field

Session Type: Coordinated Paper Session

Abstract

Amidst fast-moving change, the field of measurement has much to offer in text analysis, as well as much to learn from it. In this session, we share insights from the intersection of natural language processing (NLP) and measurement, with a focus on measuring biases in language. The first paper will seek to measure the biases of ChatGPT-generated text using NLP methods, in the context of a practice-based concern for high school counselors: ChatGPT-drafted letters of recommendation.The second presentation draws on NLP methods to analyze the use of deficit-based language within the measurement field itself in a quant-crit based study of measurement journals. The final presentation will set forth a framework for combining NLP methods with measurement methods to construct scales that measure constructs of interests across large bodies of text, with an application to syllabi data. Assessing and reducing biases in the production and review of text has remained a difficult problem to measure and address. Through our panel, we grapple with what it means to measure constructs, especially those related to equity, within text mediums that have not historically been examined at scale.

Sub Unit

• Individual Papers, Innovation Demonstrations, Coordinated Sessions, or Organized Discussions

Session Organizer

• Brein Mosely

Chair

• Sebastian Muñoz-Najar Galvez, Harvard

Individual Presentations

• Deficit-based Language in Measurement Journals: A Quant-Crit Informed Textual Analysis - Brein Mosely, Harvard; Zach Himmelsbach; Sebastian Munoz-Najar Galvex, Harvard

• Measuring Text: A Framework for Developing and Validating Scales for Text Analysis - Emma M. Klugman, Harvard Graduate School of Education

• ChatGPT as a Writing Assistant: Merit and Adversity Framing in Recommendation Letters - Amy Desiderio, Harvard; Sheridan Stewart, Harvard; Sebastian Muñoz-Najar Galvez, Harvard; Julius DiLorenzo, Harvard; Jen Ha, Harvard; Matthew Nicola, Harvard; Tara P. Nicola, Harvard Graduate School of Education; Megan Richardson, Harvard; Mandy Savitz-Romer, Harvard

Discussant

• Luke Weisman Miratrix, Harvard Graduate School of Education