Source-linked AI summary
A Mixed-Method Empirical Study of LLM Assistance in Software Engineering Workflows
Pamali D. Weerasinghe, Roshan N. Rajapakse, Isuru Dharmadasa, Chamath Keppitiyagama
TL;DR
LLM effects in software engineering are difficult to assess without separating task types, developer seniority, and verification demands. This mixed-method study combines a survey with a task-based quasi-experiment and behavioral analysis, finding widespread adoption but task-dependent benefits and workflow changes shaped by expertise and verification practices.
Problem
Existing discussions often do not distinguish LLM effects by software engineering task type, developer seniority, or verification demands.
Method
The study combines a survey with a task-based quasi-experiment involving first-year and fourth-year undergraduates, using screen recordings to analyze behavior across AI-assisted and non-AI workflows.
Results
LLM use was routine, while AI assistance reshaped workflows and perceived advantages differed by task and cohort, especially for constraint-heavy scenarios.
Takeaways & Limitations
LLM assistance can accelerate routine drafting but does not remove the need for human judgment in constraint interpretation, verification, and architectural reasoning.
Takeaways & Limitations
The quasi-experiment uses purposive grouping, a bounded task suite, undergraduate participants from one institution, and a small sample, limiting causal interpretation and generalizability.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are increasingly integrated into software development workflows, yet their effects are often discussed without distinguishing between task types, developer seniority, and verification demands. This paper presents a mixed-method empirical study of LLM-assisted software engineering with first-year and fourth-year undergraduates. Phase 1 is a preliminary survey (n=157) that characterizes LLM exposure, reliance, and trust calibration among the two groups. Phase 2 is a task-based quasi-experiment with a purposive sample from both cohorts (n=20). Here, we compare AI-assisted and non-AI conditions on a structured set of software engineering tasks spanning implementation, constraint-driven algorithm selection, and architectural reasoning. We then analyze performance outcomes alongside behavioral traces captured via screen recording and a qualitative coding process. Survey results indicate widespread LLM adoption and substantial verification effort, alongside cohort differences in perceived LLM capability for constraint-heavy scenarios. The quasi-experiment further shows that AI assistance changes workflow structure. For example, participants frequently adopt AI-first task entry, copy-transfer integration, and AI-mediated debugging, whereas non-AI workflows rely more on documentation, prior templates, and iterative trial-error refinement. Overall, our findings suggest that the benefits of LLM assistance are task-dependent and mediated by expertise and verification practices, rather than by generation speed alone.
1 INTRODUCTION
LLM assistance is reshaping software engineering, but its value depends on task type, developer expertise, and verification rather than generation speed alone. This study examines those differences through survey, quasi-experimental, and behavioral evidence.
- Motivation: Software engineering tasks span implementation, constraint-driven algorithm selection, performance-sensitive design, and architectural trade-off analysis.These tasks require interpreting incomplete requirements, reasoning about implicit constraints, and validating solutions beyond immediate correctness.
- Motivation: Prior claims that LLMs reduce the expertise gap often overlook task categories, skill levels, and the additional effort required for validation or bug fixing.The paper frames granular investigation of AI assistance as more informative than further demonstrations of coding acceleration alone.
- Study focus: The study combines a preliminary survey, a task-based quasi-experiment, and screen-recording analysis across first-year and fourth-year undergraduates.The task suite covers implementation programming, constraint-driven algorithmic reasoning, and middleware or architecture design under AI-assisted and non-AI conditions.
- Contributions: The paper characterizes LLM usage patterns, trust calibration, verification behavior, and problem-solving strategies across the two cohorts.This contribution connects reported usage and confidence with observed workflow behavior.
- Contributions: Behavioral analysis identifies distinct workflow interaction patterns between AI-assisted and non-AI conditions.The analysis uses participant screen recordings to compare how developers enter, solve, and verify tasks.
2 RELATED WORK
Related work documents broad LLM adoption across software engineering, coding benefits from prompting and code generation, and unresolved concerns about learning, validity, trust, and adaptation to project context.
- LLM adoption: LLM-based tools support software design, code generation, debugging, testing, and deployment, reflecting broad integration across the software engineering lifecycle.Research also examines requirements engineering, system design, and software quality assurance.
- Prompting and code generation: Structured prompt engineering has been associated with improvements in coding velocity and solution quality, with prompt tailoring emphasized for tools such as Claude and Copilot.These studies motivate attention to how prompting strategies interact with tool and model characteristics.
- Evaluation validity: Benchmark performance may reflect memorization or pattern recall when datasets contain publicly available coding problems and solutions.The paper therefore emphasizes previously unseen tasks to reduce the likelihood of data contamination.
- Learning and performance: Prior studies report mixed outcomes: AI-heavy use can improve productivity while accompanying reduced learning or weaker conceptual understanding.Other student studies found more debugging activity and more error messages in ChatGPT-facilitated programming.
- Education: Research on student use identifies perceived usefulness alongside challenges in prompt formulation and adapting generated artifacts to project-specific needs.These findings support pedagogical strategies that preserve learning while using LLM capabilities.
3 Methodology
The study uses a two-phase mixed-method design combining a survey with a task-based quasi-experiment on complex, open-ended software engineering tasks. Screen recordings and thematic coding capture both outcomes and workflow behavior.
- Design: The methodology consists of a preliminary survey followed by a task-based quasi-experiment measuring performance, approaches, and behavior patterns on COSTs.The survey profiles participants and informs experimental candidate selection.
- Participants: The survey sample comprised 131 first-year and 26 fourth-year undergraduates from the University of Colombo School of Computing.The study population is undergraduate students from a single institution.
- Phase 1: Survey: The survey includes demographics, baseline technical skills, LLM familiarity and reliance, verification-related confidence, and scenario-based comparisons.Scenarios address complex tasks, rapid prototyping, implicit regulations, and interactive refinement.
- Phase 2: Quasi-Experiment: The experiment formed with-LLM and without-LLM groups using cohort-specific criteria based on LLM experience, programming proficiency, and preferred problem-solving approach.Ten participants were selected from each cohort for the task-based experiment.
- Data collection: Participants completed COSTs without time restrictions while continuous screen recording captured real-time technical behavior.The task specifications and survey instruments were made available online.
- Analysis: Survey data were analyzed with descriptive statistics, nonparametric cohort comparisons, categorical exact tests, and effect sizes, while videos underwent transcription and thematic analysis.Behavioral codes covered task entry, prompting, code transfer, debugging, and verification before clustering into higher-level themes.
4.1 Preliminary Survey
The survey shows widespread, routine LLM use alongside substantial verification effort and cohort differences in perceived capability for complex or constraint-heavy software engineering tasks.
- LLM exposure and usage: 56.06% (88/157) used LLMs daily for coding, while 33.76% (53/157) used them occasionally for targeted errors and 10.19% (16/157) relied heavily on them.Fourth-year daily use was 84.62% (22/26), compared with 50.38% (66/131) among first-year respondents.
- Technical domains: 78.34% (123/157) selected Algorithms & Data Structures as a primary LLM-use domain, with substantial use also reported for frontend, backend, and database development.The domain question allowed multiple selections.
- Perceived code reliability: 63.69% (100/157) believed AI-generated code worked immediately “Often (50–80%)” for COSTs, while 22.93% (36/157) selected “Rarely.”Only 1.27% (2/157) selected “Never.”
- Verification: 54.78% reported significant verification and 38.22% reported minimal execution checks, while only 5.10% reported no verification.Among respondents who viewed AI code as often or almost always workable, 54.62% still reported significant verification; perceived reliability correlated negatively with verification effort (Spearman ρ = −0.21, p = 0.008).
- Engagement with generated code: 71.97% (113/157) asked LLMs to explain complex solutions line-by-line before use, compared with 20.38% who rewrote the logic and 7.64% who copy-pasted and executed it unchanged.The response pattern indicates that many students use LLMs as comprehension aids as well as code generators.
- Perceived weaknesses: 54.78% (86/157) identified debugging complex logic errors in large projects as the area where LLMs perform poorly.This contrasts with heavy reported use for algorithms and data structures, despite 14.01% (22/157) identifying LeetCode-style algorithmic problems as a weakness.
4.1.3 Scenario judgments: perceived advantage of humans vs. LLMs
Respondents generally reserved human expertise for ambiguity and implicit constraints, while viewing LLMs as especially advantageous under tight time limits. Judgments about iterative refinement were more balanced across both cohorts.
- 58.60% of respondents favored human experts for unclear instructions and interpretation, with a stronger preference among Fourth-Year students.Fourth-Year preference was 69.23%, compared with 56.49% among First-Year students.
- 84.08% favored LLMs under a 30-minute constraint, rising to 92.31% among Fourth-Year respondents.
- 52.23% favored human experts for unspoken domain rules and constraints, including 80.77% of Fourth-Year respondents and 46.56% of First-Year respondents.
- 43.31% favored LLMs for iterative refinement, compared with 35.03% favoring humans and 21.66% perceiving parity.The cohorts showed similar splits, with effectiveness depending on feedback quality and task nature.
4.1.4 Between-cohort differences and within-sample associations
Fourth-Year respondents reported stronger programming proficiency, confidence, and LLM experience, while proficiency and industry experience were associated with reserving human expertise for constraint-heavy tasks. Comparisons remain suggestive because the Fourth-Year subgroup was small.
- Because the Fourth-Year subgroup was small (n = 26) relative to First-Year (n = 131), between-cohort comparisons should be interpreted as suggestive.
- Fourth-Year respondents reported higher programming proficiency, confidence without external help, LLM usage history, and daily-workflow LLM use than First-Year respondents.Programming proficiency medians were 4.0 versus 2.0, and daily-workflow use was 84.62% versus 50.38%.
- Fourth-Year students showed a stronger human preference for domain-rule tasks, while First-Year students with industry experience also preferred human-generated solutions more strongly.The between-cohort difference was U = 839.0, p < 0.001, r = 0.51; within First-Year students, U = 612.5, p = 0.029, r = 0.33.
- Confidence without external help was negatively associated with assignment reliance on LLMs (Spearman ρ = −0.19, p = 0.017).
- Higher proficiency was associated with stronger human preference for constraint-heavy tasks and stronger LLM preference for speed-focused drafting.The corresponding associations were ρ = 0.35, p < 0.001, and ρ = −0.30, p < 0.001.
- Verification effort increased with programming proficiency (ρ = 0.21, p = 0.009), while perceived reliability was negatively associated with verification effort (ρ = −0.21, p = 0.008).The authors interpret this pattern as consistent with more proficient respondents applying stricter validation standards.
4.1.5 Implications from the Survey results
The survey indicates that LLM use is embedded in student workflows and that users differ in how they calibrate trust across task types. The findings motivate studying adoption intensity, verification depth, prompting strategies, and error detection rather than treating AI use as binary.
- LLM use is embedded in everyday workflows for most students, particularly senior cohorts, making a simple no-AI baseline potentially ecologically limited.
- Future work should measure adoption intensity, verification depth, prompting and iteration strategies, and error-detection competence.
- Multiple indicators point to a trust-calibration gradient in which novices are more likely to treat LLMs as capable when implicit constraints matter.
- More experienced users verify more and reserve human expertise for constraint-heavy correctness.
4.2 Task-Based Quasi-Experiment
AI assistance reorganized task entry, information seeking, code integration, debugging, and architecture work. AI-assisted sessions often emphasized rapid generation and transfer, whereas non-AI sessions relied more on deliberate triage, documentation, templates, and execution-based refinement.
- Task entry and triage: AI-assisted participants frequently began with AI-first task entry, prompting for complete solutions before producing independent decomposition artifacts.
- Task entry and triage: Without AI, participants more often read requirements carefully, rechecked constraints, and consulted documentation before coding or selecting tools.Fourth-Year participants showed the clearest deliberate triage and structured sequencing.
- Information seeking: LLMs often replaced documentation and conceptual scaffolding by explaining unfamiliar frameworks, constraints, and algorithm choices through conversation.
- Information seeking: Non-AI planning relied more on web searches, official documentation, and reusable templates, with senior participants showing an expert-verification approach.
- Construction and reuse: AI-assisted sessions commonly used copy-transfer integration, with near-complete templates pasted into projects and minimally modified.
- Verification and debugging: First-year AI sessions often showed low verification intensity, including cursory review and limited execution or independent edge-case validation.
- Verification and debugging: AI-mediated debugging involved pasting raw errors into an LLM and following procedural fixes, while non-AI debugging used web-supported trial-error or systematic test-and-verify routines.
- Environment constraints: Generated solutions frequently assumed unavailable dependencies, causing environment failures that exposed limits of AI scaffolding.
5 THREATS TO VALIDITY
The study’s validity is constrained by quasi-experimental grouping, self-reported measures, a bounded task suite, transcription risks, and a small, single-institution undergraduate sample.
- Purposive grouping means observed AI-condition differences may also reflect prior proficiency, tool fluency, motivation, or LLM experience.
- Self-reported AI literacy, verification effort, and confidence may not fully correspond to participants’ observed behavior.
- The bounded task suite spans multiple abstraction levels but excludes broader industrial practices such as long-term maintenance and team coordination.
- Video transcription and experimental timeline construction may introduce measurement errors.
- Undergraduate participants from one institution and a small quasi-experiment sample limit generalizability.
6 CONCLUSION
The study combines survey and quasi-experimental evidence to show that LLM assistance reshapes software-engineering workflows rather than uniformly eliminating expertise gaps. Its benefits appear strongest for routine drafting, while judgment and validation remain necessary for complex tasks.
- The mixed-method design combines a survey with a task-based quasi-experiment.
- LLM use is routine among students, but trust calibration and verification practices vary with seniority and experience.
- AI availability shifts effort from planning and documentation navigation toward prompting, integration, and AI-mediated debugging.
- LLMs can accelerate routine implementation drafting but do not remove human judgment in constraint interpretation, toolchain troubleshooting, and validation.
- The findings temper broad claims that LLMs uniformly bridge expertise gaps.