Uneven Automation: ChatGPT’s Impact on Software Tasks Varies by Difficulty and Data Availability

• In Event: 30216 - Emerging Work Practices in the Age of AI

Abstract

While various sectors are integrating LLMs into their work processes, researchers suggest that LLMs' advanced cognitive capabilities can potentially replace human labor, not only simple but also the one that requires high cognitive skills. Despite its existential threat on human labor, the data-driven analyses of the LLMs’ impact remain limited due to its nascent stage. We empirically analyze this problem by tracing changes in software developer’s tasks using digital traces from Stack Overflow, a prominent knowledge-sharing platform for software developers. We focus on two key research questions: (1) does ChatGPT impact software developers’ tasks equally regardless of task difficulty? (2) does ChatGPT impact equally regardless of the data availability? Leveraging 382,677 posts from 211,348 users over a 24-month period spanning one year pre- and post-GPT release, we find difficult tasks are less impacted by ChatGPT than easy ones. Also, tasks about rare topics on software engineering are less impacted than popular topics, revealing that the potential for task automation correlates significantly with the availability of digitized domain knowledge. Our work is significant in that it captures the ongoing process of replacing human labor with the innovative tweaking of existing digital trace data, and suggests two factors of the impact of LLMs: task difficulty and the level of digitization of domain knowledge.

Authors

• Myokyung Han

• Jiwoon Hong, KAIST

• Taegyoon Kim, KAIST

• Jinhyuk Yun, Soongsil University

• Lanu Kim, KAIST