DAO-ISEM-IORA Seminar Series: Hummy Song

Hummy Song

BIZ1 0202

The First Draft Problem: Productivity, Quality, and Physiological Strain in LLM-Assisted Writing

The use of large language models (LLMs) has grown rapidly over the past few years, especially in health care, where documentation burden remains a persistent problem for practicing clinicians. However, despite the deployment of many ambient scribes for documentation, it is still unclear how LLMs impact those who work with them. Across two preregistered studies, we examine how this shift—from writing documentation to editing an LLM-generated draft—impacts workers. We study this “first draft problem”: eliminating the blank page does not remove the work of documentation but relocates it into revising and editing. In Study 1, an online experiment with 16 clinicians producing documentation following a teletherapy patient encounter, we establish the puzzle: counter to our hypotheses, editing an LLM-generated draft of the documentation as opposed to writing it from scratch does not result in time savings or quality improvements on average. Instead, while some clinicians experience no time savings but large improvements in quality after making substantial revisions, others experience meaningful time savings and no change in quality after making limited revisions. In other words, we observe significant heterogeneity in how clinicians engage with the same draft, which leads to variation in productivity and quality. In Study 2, a lab experiment with 100 university students summarizing video interviews, we unpack the mechanisms underlying this heterogeneity using granular keystroke logs, validated self-reports of workload, and micro-level physiological and eye-tracking measurements. In this setting, we find the expected productivity improvements: editing reduces task completion time by roughly 70%, raises quality, and lowers self-reported workload. Yet the physiological evidence tells a more nuanced story. Physiological markers indicate that fatigue accumulates faster when editing an LLM-generated draft than when writing from scratch. Thus, we find that (a) time savings can be real, but are setting dependent; (b) quality improvements are possible, but the relationship between how heavily a draft is revised and the quality of the result differs sharply across our two settings; and (c) while editing an LLM draft lowers reported cognitive load, the body still tires faster over time.

Hummy Song is an Associate Professor of Operations, Information and Decisions at the Wharton School at the University of Pennsylvania. She also holds an appointment as Associate Professor of Health Care Management. She conducted her undergraduate, master’s, and PhD studies at Harvard University.

Professor Song’s research focuses on identifying ways to improve the performance of service systems, with a particular emphasis on the health care sector. For her research, Professor Song has worked with hospitals and health care delivery organizations in the U.S. and in developing countries. She is the recipient of the 2026 MSOM Young Scholar Prize, the 2022 POMS Early Career Research Accomplishments Award, and the M&SOM Service Management SIG Best Paper Award. She was also named by Poets & Quants as one of the Top 50 Undergraduate Business Professors. She currently serves as an Associate Editor of Management Science, M&SOM, Operations Research, and Service Science.