Source-linked AI summary
How AI Impacts Skill Formation
Judy Hanwen Shen, Alex Tamkin
TL;DR
The paper asks whether AI assistance helps workers complete unfamiliar coding tasks without weakening the skills needed to understand and supervise AI-generated work. Using randomized experiments with the new asynchronous Python library Trio, it finds lower conceptual, code-reading, and debugging performance without a significant average efficiency gain. However, cognitively engaged AI-use patterns preserve learning outcomes, while full delegation can improve speed at the cost of learning.
Problem
It is unclear whether workplace AI assistance improves productivity while hindering the development of skills needed to understand and supervise automated work.
Method
The study uses randomized experiments comparing developers who complete unfamiliar Trio coding tasks with or without access to AI assistance, followed by an AI-free skill evaluation.
Results
AI assistance reduced evaluation scores by 17% or two grade points (Cohen’s d = 0.738, p = 0.010) without significantly accelerating completion time on average.
Takeaways & Limitations
AI assistance is not a shortcut to competence: cognitively engaged patterns preserved learning, whereas reliance on code generation weakened skill formation.
Takeaways & Limitations
The study uses a single task with a chat-based interface and does not measure differences in prompting techniques beyond self-report.
Abstract
from arXiv · showhide
AI assistance produces significant productivity gains across professional domains, particularly for novice workers. Yet how this assistance affects the development of skills required to effectively supervise AI remains unclear. Novice workers who rely heavily on AI to complete unfamiliar tasks may compromise their own skill acquisition in the process. We conduct randomized experiments to study how developers gained mastery of a new asynchronous programming library with and without the assistance of AI. We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library. We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance. Our findings suggest that AI-enhanced productivity is not a shortcut to competence and AI assistance should be carefully adopted into workflows to preserve skill formation -- particularly in safety-critical domains.
1 Introduction
The paper examines whether AI assistance helps workers complete unfamiliar coding tasks while undermining the skills needed to understand and supervise automated work. Randomized experiments with a new Python library found reduced learning outcomes, no average significant speedup, and better skill preservation when users remained cognitively engaged.
- Motivation: AI assistance may hinder the conceptual understanding and supervisory skills workers need when technology shifts their role toward overseeing automated tasks.These skills remain important because humans are responsible for final-product quality and error detection.
- Research questions: The study asks whether AI improves productivity on coding tasks requiring new skills and whether it reduces understanding of those skills.Participants learned and used a previously unfamiliar Python library before completing a competency evaluation.
- Results: Additional interaction time, including composing queries, helps explain why AI assistance did not significantly improve productivity on average.Some participants asked up to 15 questions or spent more than 30% of available task time interacting with the assistant.
- Results: Three of six AI interaction patterns preserved skill development by requiring more cognitive effort and independent thinking, such as asking conceptual questions or requesting explanations.These patterns produced higher skill-evaluation scores than more delegating forms of AI use.
2 Background
Prior research documents substantial productivity gains from AI coding assistants, especially for less experienced workers, while raising concerns about cognitive offloading, weak retention, and overreliance. This paper addresses the relatively less studied relationship between AI use and skill acquisition in coding.
- Productivity gains: AI assistants have improved productivity across professional settings, including software development, while benefits often appear larger for novice or less experienced workers.Reported examples include 55.5% faster task completion with Copilot and a 26.8% productivity boost in software-company measures.
- Cognitive offloading and retention: Related studies associate AI assistance with cognitive offloading, weaker critical-thinking or visual-diagnostic development, and performance gains that may not persist on later independent tasks.These findings motivate examining learning outcomes rather than only immediate task performance.
- Overreliance: Because generative AI can produce incorrect or hallucinated content, overreliance can cause human decisions to follow erroneous model outputs.The literature defines overreliance in settings where fallible models assist human decisions.
- Skill measurement: Computer-science learning is commonly assessed through code writing, code reading or explanations, and multiple-choice questions, with code interviews and active discussion also linked to positive learning outcomes.These assessment approaches provide context for evaluating coding-skill acquisition.
- Usage patterns: Observational and user studies describe AI use for code writing, debugging, explanations, and interactive discussion, with interaction quality varying from full delegation to minimal questioning.This variation supports analyzing distinct AI-usage patterns rather than treating assistance as uniform.
3 Framework
The framework focuses on on-the-job learning, asking whether AI assistance creates a tradeoff between immediate productivity and longer-term skill development. It studies this question through new-skill coding tasks and varied patterns of AI interaction.
- Learning framework: The framework connects real-world task completion with learning new concepts and developing skills through experiential-learning approaches.This motivates studying skill acquisition during coding work rather than only after formal instruction.
- AI interaction: Different ways of using AI represent different learning paths to completing a task, including question answering, code generation, and debugging.The study analyzes these usage patterns qualitatively.
- Research questions: The research asks whether AI assistance improves productivity when new skills are required and how it affects development of those skills.The framework explicitly tests both immediate performance and skill formation.
- Research questions: The experiments investigate whether workers can complete unfamiliar tasks without the same learning outcomes when AI assistance is available.The setup targets coding-skill formation during task completion.
4 Methods
The study used a randomized coding experiment in which participants learned the unfamiliar Trio asynchronous Python library, completed timed tasks with or without AI assistance, and then took an AI-free comprehension quiz.
- Task and setting: The experiment centered on Python Trio, an asynchronous-concurrency library introducing concepts beyond general Python fluency.Trio was selected because it was relatively unfamiliar and designed to be easy to use in a learning experiment.
- AI condition: The AI assistant used GPT-4o, had access to participants’ current code, and could generate complete correct solutions when prompted.The interface was an online interview platform with a coding-assistant chat interface.
- Evaluation: Skill mastery was assessed with debugging, code-reading, and conceptual questions covering seven Trio concepts, excluding code-writing questions.The exclusion reduced the influence of syntax errors that could be corrected through AI queries or web search.
- Study design: The study used a between-subjects randomized design with 52 participants, split evenly between control and treatment groups.Participants had Python experience, used Python weekly, had prior AI-assistance experience, and had not used Trio.
- Study design: All participants first completed an AI-free warm-up task, while only the treatment group could use AI during the main Trio task.The warm-up task did not require Trio knowledge and helped calibrate Python familiarity and interface familiarity.
- Task and setting: Participants completed two Trio coding tasks within a maximum of 35 minutes, with treatment participants receiving chat-based AI assistance.All participants were instructed to complete the tasks as quickly as possible.
5 Results
The pilot and main studies found that AI assistance did not reliably improve completion time, while reducing quiz-measured skill formation. This pattern held across prior coding-experience levels, with the largest exploratory gap in debugging questions.
- 5.1 Pilot Studies: 60% of control participants versus 90% of AI participants completed the pilot task within 35 minutes.The pilot included 20 participants and found significant differences in both completion time and quiz score.
- 5.1 Pilot Studies: Cohen’s d=1.11, p=0.03 for pilot task time, while AI participants scored significantly worse on the quiz (Cohen’s d=1.7, p=0.003).The pilot AI group completed Trio tasks faster but showed reduced learning retention.
- 5.2.2 Results: Cohen d=0.738, p=0.01: AI reduced main-study quiz performance by 4.15 points, or 17% and two grade points, without significantly improving completion time.The treatment effect remained significant after controlling for warm-up task time (Cohen’s d=0.725, p=0.016).
- 5.2.2 Results: The main-study results do not conclusively establish either a speedup or slowdown from AI assistance.Four control participants did not finish the second task within 35 minutes, whereas every AI participant did.
- 5.2.2 Results: Control participants scored higher on average than AI participants across all prior coding-experience levels.The authors report that the tasks presented new skills to be acquired regardless of experience level.
- 5.2.2 Results: The largest exploratory quiz-score gap occurred in debugging questions, while the smallest occurred in code-reading questions.The analysis was not pre-registered; the authors relate the debugging gap to independent error resolution in the control group.
- 5.2.2 Results: The AI group reported lower self-reported learning, although both groups reported high enjoyment and the AI group found the task easier.These were further exploratory measures of participants’ study experience.
6 Qualitative Analysis
Qualitative analysis identified six AI interaction patterns with different productivity and learning outcomes. Patterns involving greater cognitive effort and independent thinking generally preserved skill formation better than heavy AI reliance.
- Interaction Time: Participants spent up to 11 minutes interacting with AI, helping explain the absence of a significant overall speedup.Some participants spent substantial time composing queries rather than coding.
- Encountering Errors: Control participants encountered more errors, and independently resolving them likely improved Trio-skill formation.The AI group’s median participant encountered one error, compared with three for the control group.
- Active Time: AI shifted time away from active coding toward interacting with the assistant and understanding generated code.Participants could spend up to six minutes composing a single query during the 35-minute assignment.
- Interaction Patterns: The six interaction patterns produced different completion-time and quiz-score outcomes, separating into low- and high-scoring groups.High-scoring patterns generally involved more cognitive effort and less AI reliance.
- Low-Scoring Interaction Patterns: AI Delegation participants completed tasks fastest but wholly relied on AI and encountered few or no errors.Progressive AI Reliance and Iterative AI Debugging also scored poorly on the quiz.
- High-Scoring Interaction Patterns: Generation-Then-Comprehension participants followed generated code with questions to improve understanding and demonstrated high quiz performance.This pattern resembled AI Delegation but added explicit checking of understanding.
- High-Scoring Interaction Patterns: Conceptual Inquiry was the fastest high-scoring pattern and involved independently resolving many encountered errors.Hybrid Code-Explanation participants requested code alongside explanations, which took more time to read and understand.
- Query Types: Debugging-focused queries correlated with slower completion times and lower quiz scores.The debugging category included pasting errors to AI and asking it to confirm or fix code.
7 Discussion
AI assistance reduced skill formation for unfamiliar coding tasks, but cognitively engaged interaction patterns preserved better learning outcomes. The study’s scope and measurement choices motivate further research on longer-term, more realistic skill development and alternative forms of assistance.
- AI use reduced conceptual understanding, code reading, and debugging skills during acquisition of a new Python library.
- 19.5 minutes versus 23 minutes was the completion-time comparison for the small group that delegated code generation, while the overall AI efficiency improvement was not statistically significant.This group represented around 20% of the treatment participants; other users’ numerous or lengthy queries raised average completion time.
- The authors suggest that aggressive AI incorporation can harm professional development when workers are not cognitively engaged.They connect time constraints and organizational pressures with reliance on AI at the expense of skill development.
- The study measured skill formation for one Python library over one hour, although meaningful skill formation ideally occurs over months to years.
- The participant sample consisted of professional or freelance programmers without the same incentive to learn as workers required to use the library in an actual job.
B Qualitative Analysis Data and Details
The qualitative analysis used screen recordings from nearly all participants to annotate task events and identify themes in AI use.
- 51/52 participants uploaded screen recordings, including recordings from 25 AI-condition and 25 no-AI-condition participants for the main coding task.
- The researchers recorded timestamps for specified events in the task recordings.
- The researchers noted general themes in participants’ AI use based on annotated event codes.
B.2 Data Availability
The appendix provides annotated transcripts and participant-feedback materials supporting the study’s qualitative analysis.
- Annotated participant transcripts are made available through the study’s GitHub URL.
- Table 5 lists the events manually annotated for each main-task video recording.
- Table 6 contains feedback from participants in the AI condition.The AI group left more comments because its average completion time was faster and participants felt they had more time at the end.
C.1 Evaluation Design
The evaluation covered seven core Trio-library concepts using conceptual, code-reading, and debugging questions, while excluding code-writing questions. Figures 18 and 19 examine how query behavior relates to completion time and quiz performance.
- The evaluation used conceptual-understanding, code-reading, and debugging question types.
- Seven core concepts from the Trio library were covered by the evaluation.
- The quiz included questions about async and await keywords, including when await is used inside async functions.
- The quiz covered starting Trio functions, error handling, coroutines, memory channels, nurseries, and sequential versus concurrent execution.
- Figure 18 relates total queries and completion time to the fraction of debugging queries, while Figure 19 relates total queries and quiz score to reliance on AI for debugging.
D Task Details
The evaluation tested conceptual understanding, code reading, and code writing using separate control and AI-assisted conditions on a shared task platform. Control participants were instructed not to use AI, while treatment participants were encouraged to use an AI assistant for speed.
- The evaluation measured three software skills: conceptual understanding, code reading, and code writing.
- Control-group participants pledged not to use AI assistance and received instructions emphasizing avoidance of AI tools.
- Treatment-group participants took a pledge and were encouraged to use the AI assistant to complete the task as quickly as possible.
- Both conditions used a platform with instructions on the left and a coding editor on the right.
- The AI-condition platform included a nudge to use the AI assistant in the left tool plane, with a separate view showing interaction with the assistant.