Text Persistence Project

• In Event: Digital College Access and Success Tools: Scholarly and Practitioner Perspectives on What's Working for Generation Z

Abstract

Artificial Intelligence (AI) chatbots have recently been shown to help universities improve student enrollment outcomes at the pre-matriculation stage in meaningful ways (e.g., by reducing the “summer-melt”) (Page & Gehlbach, 2017). It is still an open question, however, whether such AI-based systems can also successfully support students in resolving various issues that hinder retention and persistence among already enrolled college students. We investigate the effectiveness of an ongoing project at [institution removed for blind submission] that uses a chatbot designed specifically to address such concerns in the continuing undergraduate student population.
 
During the 2018-19 academic year, the [institution removed for blind submission] Office of Enrollment Management and Student Success partnered with AdmitHub to develop and pilot “Pounce”, an AI-based conversational chatbot named for the university mascot. Pounce reached out to selected, currently enrolled undergraduate students via text messages to communicate with students regarding a variety of tasks and activities related to their academic persistence and progress. Students were able to interact with the chatbot by responding to the outreach and/or asking additional questions regarding topics such as registration, financial aid, and advisement. 

We tested the efficacy of this tool via a randomized controlled trial. Our study participants consisted of 13,616 undergraduate students at various stages of their studies whom we randomized in approximately equal shares to either a treatment or control condition. Treatment students were targeted for outreach from Pounce whereas control students received business-as-usual support from the university. In many outreach campaigns, we customized and targeted chatbot outreach based on university-held data indicating those tasks on which students were falling behind. 

The chatbot outreach improved students’ responsiveness to and success with critical milestones that could immediately impact their enrollment status and progress. For example, students in the treatment group were more likely to complete the FAFSA and to resolve academic and financial aid holds at a faster rate, compared to those in the control group. In addition, we observed a strong, positive, and statistically significant effect of the chatbot outreach and support on whether students registered for the Fall 2019 semester, suggesting a positive effect on overall college persistence. Taken together, our findings suggest that the chatbot outreach throughout the year supports students to be more successful in handling necessary administrative and other processes, and in this way facilitates their persistence in college from one year to the next. These findings provide valuable insight into the academic and administrative obstacles in which text-based outreach may best serve to support enrolled students to persist and progress.

Authors

• Jeonghyun Lee, Georgia Institute of Technology

• Lindsay Coleman Page, University of Pittsburgh

• Hunter Gehlbach, Johns Hopkins University