Source-linked AI summary
Developer Task Delegation to LLMs and the Trust Factor: A Mixed Methods Study
Samuel Ferino, Rashina Hoda, John Grundy, Christoph Treude, Hashini Gunatilake
TL;DR
The paper addresses limited evidence about which software-development tasks developers delegate to LLMs, to what extent, and why. It combines interviews with a survey to identify delegation modes and examine associated factors; trust is significantly associated with delegation. The study’s survey scope was limited to 17 coding tasks.
Problem
How much and why developers delegate coding-related tasks to LLMs remained unclear despite prior studies of selected tasks and factors.
Source: §I · p. 1Method
The researchers interviewed 22 practitioners and surveyed 160 practitioners about task delegation and associated factors.
Source: §I · p. 1Results
Trust was significantly associated with full or partial task delegation, and the study identified four delegation modes: full delegation, partial delegation, self-delegation, and task avoidance.
Source: §I · p. 2Takeaways & Limitations
For software organizations moving toward autonomous task delegation, the authors describe balanced trust in LLM capabilities as essential.
Source: §VII · p. 11To keep the survey concise, the researchers narrowed its task scope from the full software-development life cycle to 17 coding tasks.
Source: §VI · p. 10
Abstract
from arXiv · showhide
With the rise of Large Language Models (LLMs), software developers are presented with a powerful new approach: delegating software development tasks to LLMs. Existing studies have explored task delegation for a subset of tasks and factors. Critically, what tasks developers delegate to LLMs, to what extent, and why they delegate them, is not yet well understood. We explore these questions through 22 interviews and a survey with 160 software practitioners. Developers reported employing four modes of task delegation to LLMs: full delegation, partial delegation, self-delegation, and task avoidance. Drawing on our qualitative findings, we discuss the rationales behind these choices and explore how task difficulty, task risk, developer motivation, and trust appear to be associated with software development task delegation to LLMs. Our statistical analysis reveals that trust is a significant factor associated with task delegation to LLMs. Finally, we share actionable implications for human-AI collaboration in software research, and for software practitioners to make the most of their newly found powers of delegation.
I. INTRODUCTION
The study addresses limited understanding of which software-development tasks developers delegate to LLMs, how extensively, and why. It examines delegation modes and their associated factors through interviews and a practitioner survey.
I. INTRODUCTION
- Prior work reports that LLM automation can reduce effort on some tasks, while also raising concerns about fragmented codebase mental models and weakened software-engineering skills.The introduction also notes concerns that excessive delegation may reduce developers’ ability to review and correct LLM errors.Source: §I · p. 1
- Prior work examined delegation for selected tasks and factors, but how much and why developers delegate coding-related tasks remained unclear.The study frames this gap against reported benefits of automation and concerns about developers’ mental models and skills.Source: §I · p. 1
- Interviews with 22 practitioners followed by a survey of 160 practitioners identified full delegation, partial delegation, self-delegation, and task avoidance.Full delegation involves reviewing only the LLM’s final output; partial delegation requires closer, active management.Source: §I · pp. 1–2
- Task difficulty, task risk, developer motivation, and trust were identified as factors shaping delegation choices; trust was significantly associated with full or partial delegation.The authors describe these factors as associated with delegation, not as causes.Source: §I · p. 2
II. STUDY DESIGN
The study uses an exploratory sequential mixed-methods design, beginning with interviews and following with a practitioner survey. Interview recruitment and questioning focused on practitioners experienced with LLMs in software development.
II. STUDY DESIGN
- The study presents four delegation modes and examines how task difficulty, risk, motivation, and trust relate to delegation decisions.The authors describe the modes as identified through exploratory interviews and complemented by a survey.Source: §II · p. 2
- An exploratory sequential design used qualitative interviews first, then a mixed-data practitioner survey to address the research questions.The interviews informed hypotheses, while survey results addressed parts of RQ1 and all of RQ2.Source: §II · p. 2
1) Interview Data Collection:
- Interviewees had LLM software-development experience and varied in professional experience, geography, and domain; interviews lasted 34–58 minutes.The first author conducted semi-structured interviews in three rounds between October 2024 and September 2025; two pilot interviews were excluded.Source: 1) Interview Data Collection · p. 2
2) Interview Data Analysis:
Interview analysis used socio-technical grounded theory techniques to derive four task-delegation modes across the software-development life cycle. The modes distinguish LLM use, developer involvement, and whether a task is avoided.
2) Interview Data Analysis:
- The interview analysis used open coding, constant comparison, and memoing, with co-author review and workshops to align and critically examine emerging concepts.The authors focused on collaborative alignment rather than inter-rater reliability metrics.Source: 2) Interview Data Analysis · p. 3
- Interview analysis identified four task delegation modes adopted by interviewees across the software development life cycle (SDLC), including fully delegating tasks to LLMs and partially delegating tasks to LLMs.Source: 2) Interview Data Analysis · p. 3
- Self-delegation bypasses LLMs in favor of other resources, whereas task avoidance means evading the task despite LLM assistance being available.Source: 2) Interview Data Analysis · p. 3
3) Hypotheses Formulation:
Interview findings shaped hypotheses about factors associated with delegation and motivated a survey focused on coding-related tasks. The survey gathered mixed data from practitioners and included 17 selected tasks.
3) Hypotheses Formulation:
- Interview memoing identified task difficulty, task risk, developer motivation, and trust as potential factors associated with delegation.The researchers formulated hypotheses about relationships between these factors and delegation behavior.Source: 3) Hypotheses Formulation · p. 3
- The survey focused on coding-related tasks because developers mostly delegate these tasks to LLMs.Source: 3) Hypotheses Formulation · p. 3
- Interview accounts linked over-trust and unsafeguarded vibe coding with full delegation, while moderate trust and output review entailed greater developer engagement.A participant also described self-delegating emergent topics based on prior experience and distrust of LLMs.Source: 3) Hypotheses Formulation · p. 3
B. Phase 2: Survey
- The 15-minute survey combined open- and closed-ended questions and used a modified TDAI framework to assess difficulty, risk, motivation, and trust.Five pilot rounds narrowed the task list to 17 coding-related tasks.Source: B. Phase 2: Survey · p. 3
1) Survey Data Collection:
- After initial recruitment yielded one response in almost a month, the researchers used Prolific and applied attention and authentication checks.Source: 1) Survey Data Collection · p. 3
- Figure 2 depicts the emergence of “Task Delegation Modes” from raw data through codes and concepts to a category using constant comparison.Source: Figure 2 · p. 4
- The final dataset contained 160 responses: 155 from Prolific, three from LinkedIn, and two from the pilot survey.Manual review accepted 155 of 187 Prolific submissions (82.8%).Source: 1) Survey Data Collection · p. 4
2) Survey Data Analysis:
Survey analysis combined qualitative coding of open-ended responses with quantitative descriptive statistics and sequential mixed-effects models. The models separated adoption of any LLM assistance from the choice to delegate fully rather than partially.
2) Survey Data Analysis:
- Open-ended survey responses were analyzed using STGT4DA, open coding, and constant comparison, while closed-ended items received descriptive statistics.The quantitative analysis examined associations of difficulty, risk, motivation, and trust with delegation.Source: 2) Survey Data Analysis · p. 4
- Within-person reliability was modest for difficulty (ωwithin=0.544) and trust (ωwithin=0.512), but acceptable for motivation (ωwithin=0.785) and risk (ωwithin=0.628).Between-person reliability was strong across subscales (ωbetween=0.744–0.947); ICCs were 0.476–0.692.Source: 2) Survey Data Analysis · p. 4
- Task avoidance was excluded from regression because it was marginally selected, with only 21 observations.The ordinal outcome was decomposed because preliminary testing found the proportional-odds assumption violated.Source: 2) Survey Data Analysis · p. 4
- Sequential binary mixed-effects logistic regression modeled adoption and autonomy as separate transitions, with participant random intercepts for repeated task ratings.Adoption compared self-delegation with partial or full delegation; autonomy compared partial with full delegation, conditional on LLM adoption.Source: 2) Survey Data Analysis · p. 5
III. FINDINGS
Survey responses showed partial delegation was the most common mode across nearly all surveyed tasks, while interviews linked delegation choices to time savings, motivation, trust, and task characteristics. Participants described delegating a range of tasks, often while retaining human oversight.
A. Participants
- Participants’ ATAI acceptance scores were 6.0 in interviews and 6.81 in the survey, indicating balanced attitudes towards AI.The study included 22 interview participants and 160 survey participants.Source: A. Participants · p. 5
B. Task Delegation with LLMs (RQ1)
- Interviewees associated delegation with time savings on low-stakes tasks and low motivation, while enjoyment, low trust, or perceived LLM limitations favored manual control.The reported fully or partially delegated tasks included system design, diagram and code generation, code translation, test and document generation, and code review.Source: B. Task Delegation with LLMs (RQ1) · p. 5, 1) Tasks developers fully or partially… · p. 5
- Partial delegation (n=516) dominated across almost all tasks; full delegation (n=209) occurred for every task but was less common, and task avoidance was least common.Self-delegation totaled n=132; daily task frequency was not collected, so patterns may also reflect participants’ exposure or experience.Source: B. Task Delegation with LLMs (RQ1) · p. 5
1) Tasks developers fully or partially delegate:
- Survey participants delegated code representation in 82% of cases, including 53% fully and 29% partially; code translation delegation reached 93%, with 60% partial and 33% full.Interview examples included supplying requirements for system design and requesting architecture diagrams.Source: 1) Tasks developers fully or partially… · pp. 5–6
- For code generation, 70% partially delegated and 11% fully delegated; interviewees described using LLMs for tedious or repetitive coding.For documentation, 90% delegated code-comment generation, split between 48% partial and 42% full delegation.Source: 1) Tasks developers fully or partially… · pp. 5–6
- Interviewees described test generation as tedious and low-stakes, while one survey respondent drafted a test suite with an LLM and then reviewed and refined it.Participants also described automated PR review as an initial review or summary, with a person performing final approval.Source: 1) Tasks developers fully or partially… · pp. 6–7
2) Full delegation vs Partial delegation:
Participants distinguished full from partial delegation mainly by the degree of human involvement, while both modes included human review.
2) Full delegation vs Partial delegation:
- Partial delegation involves close, continuous management of LLM actions; full delegation requires minimal input, but both modes retain human review.Participants also associated the distinction with control, ownership, accountability, effort, mental models, and requirements.Source: 2) Full delegation vs Partial delegation · p. 7
3) Tasks developers self-delegate:
Developers self-delegate some coding, testing, approval, and debugging tasks, while others avoid LLM assistance despite its availability. Interview findings identify task difficulty, risk, motivation, and trust as factors shaping delegation.
3) Tasks developers self-delegate:
- Self-delegation was reported for code generation, test generation, pull request approval, and debugging, with concerns including bugs, confidence, and trust.One participant said testing was a critical safeguard against bugs; others cited low confidence in LLM-based approval and poor debugging experiences.Source: 3) Tasks developers self-delegate · p. 7
4) Task avoidance (despite LLMs availability):
- Some developers avoid LLMs even when available, citing security concerns, sensitive information, accountability, limited expertise, distrust, task characteristics, or a desire for control.Survey responses identified code generation (n=4), code understanding (n=4), code completion (n=3), and data analysis (n=2) among these tasks.Source: 4) Task avoidance (despite LLMs… · p. 7
1) Emerging factors distilled from interviews:
- Interviewees associated delegation choices with task difficulty, perceived individual or organisational risk, personal interest, and trust in LLMs.Examples include reluctance to use LLMs for complex tasks, concern about code leakage, enjoyment of coding over debugging, and over-trust despite unsuccessful interactions.Source: 1) Emerging factors distilled from… · pp. 7–8
2) Descriptive results:
Across 17 coding-related tasks, delegation scores were tightly bounded around partial delegation despite variation in the four measured factors. The authors interpret this pattern as systematic preference for partial delegation over either extreme.
2) Descriptive results:
- Across 17 coding-related tasks, delegation scores ranged from 1.84 to 2.48, with every mean close to 2, indicating partial delegation rather than either extreme.Source: 2) Descriptive results · p. 8
- Difficulty peaked at 2.82 for control flow graph generation, risk at 2.78 for API entity and relation extraction, motivation ranged from 2.32 to 3.15, and trust reached a low of 2.66 for code generation.Source: 2) Descriptive results · p. 8
- The authors interpret the scores near partial delegation as developers rejecting both full delegation and no delegation in favor of a balanced mode.Source: 2) Descriptive results · p. 8
3) Testing our Hypotheses:
In the sequential regression analysis, trust was the only factor significantly associated with both LLM adoption and the subsequent autonomy decision. The discussion relates this result to prior work on task-specific adoption and trust in software development.
3) Testing our Hypotheses:
- Trust was the only significant factor across adoption and autonomy; a one-standard-deviation increase was associated with 2.01 times the odds of adoption and 3.65 times the odds of full rather than partial delegation.Task risk, difficulty, and developer motivation did not meet the stated significance threshold in either stage.Source: 3) Testing our Hypotheses · p. 8
IV. RELATED WORK
- Lambiase et al. [17] described LLM adoption as potentially domain-specific, while Sergeyuk et al. found task-difficulty correlations that were not statistically significant across five software activities.The authors say their analysis evaluates these dynamics across a larger pool of experienced tasks.Source: §IV · p. 9
- The findings align with Choudhuri et al. on trust and actual LLM use, and extend that connection to both adoption and the later autonomy stage.Source: §IV · p. 9
V. DISCUSSION
The practice implications favor retaining human control over LLM delegation while improving trust; the research agenda calls for deeper study of trust, context, review, and organizational factors.
A. Implications for Practice
- For code generation, 70% partially delegate, 16% self-delegate, and 11% fully delegate; it is the least-trusted task at 2.66 out of 4.The authors advise against expecting developers to delegate code generation blindly.Source: A. Implications for Practice · p. 9
- Across analyzed tasks, partial delegation is most frequent at 60%, despite requiring more developer engagement than full delegation.The authors interpret this as developers keeping code under active human control.Source: A. Implications for Practice · p. 9
- Risk perceptions vary by task, and teams should align delegation levels because developers may misclassify tests as low-stakes.The suggested alignment balances productivity gains with reliability.Source: A. Implications for Practice · p. 9
- Trust is significantly associated with delegation; tools could add trust-supporting features, while greater trust may move near-full delegation toward automation.The authors identify aggregate usage dashboards as an initial step toward designing for trust.Source: A. Implications for Practice · pp. 9–10
B. Implications for Research
- Research should examine domain and interaction-mode differences, trust and trustworthiness, review aids for large outputs, and organizational-policy effects on delegation.The authors also propose contextualized studies spanning human role, AI specifics, and collaboration practice, evolving role, or artifact.Source: B. Implications for Research · p. 10
VI. THREATS TO VALIDITY
The study concludes that trust is significantly associated with delegation, while noting limits involving self-reports, task exposure, and survey scope. Its findings identify delegation modes and factors for research and organizational use.
VI. THREATS TO VALIDITY
- The modes and hypotheses derive from exploratory interviews, and self-reported data leave findings subject to self-reporting bias.The study used the TDAI framework to examine associations involving risk, difficulty, motivation, and trust.Source: §VI · p. 10
- Voluntary participation creates selection bias, and unmeasured daily task frequency means some delegation differences may reflect task exposure rather than task characteristics.Source: §VI · p. 10
- Ethics approval prevents sharing interview and survey raw data, while the authors document analysis and provide research artifacts, scripts, and spreadsheets.Source: §VI · p. 10
- The survey covers 17 specific coding tasks rather than the entire SDLC, though participant diversity supports reasonable transferability to similar contexts.Source: §VI · p. 10
VII. CONCLUSION
- The study identifies full delegation, partial delegation, self-delegation, and task avoidance, offering organizations vocabulary to audit delegation patterns and risk exposure.The study interviewed 22 practitioners and surveyed 160 practitioners.Source: §VII · p. 10
- Developer trust is significantly associated with task delegation, and the authors regard balanced trust as essential for organizations seeking more autonomous delegation.Source: §VII · p. 11