Source-linked AI summary
Walking the Tightrope of LLMs for Software Development: A Practitioners' Perspective
Samuel Ferino, Rashina Hoda, John Grundy, Christoph Treude
TL;DR
Prior research has examined LLMs in software engineering, but evidence on managing their impacts remains needed. The authors interviewed 22 practitioners across three rounds and analysed responses using socio-technical grounded theory. They report benefits and challenges across four levels, with recommendations to help leaders assess fit to context.
Problem
Although studies examine LLMs in software engineering, further investigation is needed into managing their impacts.
Source: §1 · p. 1Method
The authors interviewed 22 software practitioners across three rounds and analysed the interviews using socio-technical grounded theory.
Source: §2 · p. 2Results
Practitioners described LLM benefits and challenges across individual, team, organisation, and society levels, including reduced effort, mentorship losses, security and privacy concerns, and erosion of social trust.
Source: §8 · p. 14Takeaways & Limitations
The findings can support team leaders and IT managers in evaluating whether LLMs fit their context and needs.
Source: §7 · p. 14The fast-paced evolution of LLMs may quickly make some findings outdated.
Source: §7 · p. 14
Abstract
from arXiv · showhide
Background: Large Language Models emerged with the potential of provoking a revolution in software development (e.g., automating processes, workforce transformation). Although studies have started to investigate the perceived impact of LLMs for software development, there is a need for empirical studies to comprehend how to balance forward and backward effects of using LLMs. Objective: We investigated how LLMs impact software development and how to manage the impact from a software developer's perspective. Method: We conducted 22 interviews with software practitioners across 3 rounds of data collection and analysis, between October (2024) and September (2025). We employed Socio-Technical Grounded Theory for Data Analysis (STGT4DA) to rigorously analyse interview participants' responses. Results: We identified the benefits (e.g., maintain developer flow, improve developer mental models, and foster entrepreneurship) and challenges (e.g., damage to developers' reputation) of using LLMs at individual, team, organisation, and society levels; as well as actionable guidances into how mitigate these challenges. Conclusion: Critically, we present the trade-offs that software practitioners, teams, and organisations face in working with LLMs. Our findings are particularly useful for software team leaders and IT managers to assess the viability of LLMs within their specific context.
1 INTRODUCTION
The paper addresses the need to understand how to manage LLMs’ impacts in software engineering, asking how they affect practitioners and how to use them in a balanced way. It reports benefits and challenges and offers recommendations for software development.
1 INTRODUCTION
- Prior reviews report benefits and risks, but identify gaps around communication, human collaboration, and mentorship in LLM-supported software engineering.The authors therefore call for investigations focused on managing LLM impacts.Source: §1 · p. 1
- The study examines how LLMs affect software practitioners and how developers can achieve balanced use through benefits, challenges, and actionable recommendations.Its research questions ask how LLMs help developers, create imbalances, and can be used more evenly.Source: §1 · p. 1
- Interviews surfaced benefits including reduced effort, maintained flow, a safe space for interaction, and LLM entrepreneurship, alongside challenges to flow, skills, agency, and mentorship.The study involved 22 practitioners interviewed across three rounds.Source: §1 · p. 2
- The paper categorizes benefits and challenges across individual, team, organisation, and society levels.It also presents recommendations for using LLMs in software development tasks.Source: §1 · p. 2
2 METHODOLOGY
The authors conducted three rounds of interviews with software practitioners and used socio-technical grounded theory to analyse responses iteratively. The study design, sampling, and analysis were adapted as data collection progressed.
2 METHODOLOGY
- An interview-based study involved 22 software practitioners across three data-collection rounds (six, thirteen, and three participants); the final round aimed to consolidate findings, and data were analysed iteratively using STGT4DA.Source: §2 · p. 2
2.1 Study Design and Piloting
- The study paired an adapted five-point ATAI questionnaire with interviews, and pilot feedback informed the interview structure and use of a pre-interview questionnaire.The pilot interviewee without software-development LLM experience was excluded from analysis.Source: §2.1 · p. 2
2.2 Sampling
- Sampling shifted from purposive recruitment of novice developers to convenience sampling across experience levels, yielding 22 practitioners from five continents.Most participants had five or fewer years of professional experience (n=13), and ChatGPT was the most recurrent tool (n=18).Source: §2.2 · p. 3
2.3 Data Analysis
- The analysis used iterative open coding, constant comparison, and memoing to develop codes into concepts, subcategories, and categories and refine the interview guide.Researchers maintained an assumption list to make potential biases transparent.Source: §2.3 · pp. 4–5
- A memo on developer intuition notes that novices should do most of the work when learning with LLMs, supporting skill development and intuition that can help identify misleading outputs.The memo draws on participant accounts of LLMs filling knowledge gaps and of ChatGPT being misleading.Source: §2.3 · p. 5
3 FINDINGS
The findings describe LLM impacts across individual, team, organisation, and society levels, with benefits and challenges at each scale. They also report conventions for indicating how much evidence supports each theme.
3 FINDINGS
- At the individual level, participants reported time savings and reduced effort alongside friction and overhead from LLM-assisted development.The paper notes that most participants reported these individual-level effects.Source: §3 · p. 5
- At the team level, LLMs were described as mitigating interruptions but reducing mentorship opportunities.Most participants mentioned both effects.Source: §3 · p. 5
- At the organisation level, participants identified cost savings as a benefit and security and privacy as concerns.These are presented as organisation-level impacts.Source: §3 · p. 5
- At the society level, participants associated LLMs with entrepreneurship and erosion of social trust.The paper defines this level as extending beyond developer communities to daily life in society.Source: §3 · p. 5
- The authors use “few” for less than 25% and “many or majority” for over 50% to indicate the evidence supporting a theme.They clarify that the study is not quantitative.Source: §3 · p. 5
3.1 RQ1. How do LLMs help software developers move forward?
Practitioners reported benefits from LLM use at individual, team, organisational, and societal levels, including reduced effort, learning opportunities, fewer interruptions, cost savings, and support for entrepreneurship.
3.1.1 Individual Level
- LLMs reduce effort on simple, repetitive, or tedious tasks and save time by automating corrections and shortening the search for answers.Participants described handing control to LLM automation and avoiding time-consuming searches across sites such as Stack Overflow.Source: §3.1.1 · pp. 5–6
- Fourteen participants described LLMs as educational tools that support learning by examining incorrect suggestions and considering alternative solutions.Participants said this could help them assess whether a wrong answer reflected insufficient prompting and expand their solution options.Source: §3.1.1 · p. 6
- LLMs can help developers focus on logic rather than typing and support code understanding, including why a method is implemented in a particular way.Autocomplete was described as reducing writing time and leaving more space to think about development flow.Source: §3.1.1 · p. 6
- Some participants described LLMs as a nonjudgmental space for asking repeated questions, sharing ideas, and receiving conversational support.They also described companionship and empathetic responses; one participant linked using LLMs to avoiding disturbing busy colleagues.Source: §3.1.1 · pp. 6–7
3.1.2 Team Level
- At team level, participants described using LLMs to reduce interruptions, including by helping new team members investigate questions before approaching experienced coworkers.Fifteen participants described fewer interruptions and a preference for asking LLMs rather than disturbing colleagues.Source: §3.1.2 · p. 7
3.1.3 Organisation Level
- At organisational level, participants said LLM-assisted debugging can save time and reduce costs by helping locate and mitigate production problems faster.A participant from a small company described the cost of time spent searching for production bugs.Source: §3.1.3 · p. 7
3.1.4 Societal Level
- At societal level, participants saw LLMs as supporting entrepreneurship through faster prototyping, smaller teams, and access to basic business information.They described non-coders showcasing working front ends and LLMs providing basics in marketing, accounting, and law.Source: §3.1.4 · p. 7
3.2 RQ2. How do LLMs create an imbalance in software development?
LLM-related challenges span individual, team, organisational, and societal levels, combining operational friction and cognitive overhead with risks to code quality, developer capability, and reputation. Participants also described team mentorship, organisational adoption constraints, and broader concerns about trust and jobs.
3.2.1 Individual Level
- LLM interactions can disrupt flow through repeated errors, context switches, miscommunication, prompt breakdown, extra effort, and tasks that take longer than doing them directly.Participants described restarting conversations, breaking work into smaller prompts, and spending effort on inaccurate suggestions.Source: §3.2.1 · pp. 7–9
- Code-quality risks include weakened skills, missing project context, and incorrect or incompatible suggestions; generated-code errors can also damage developers’ professional credibility.Participants noted that their name, rather than the LLM’s, appears with code committed to a repository.Source: §3.2.1 · pp. 8–9
- Overreliance may erode novice and experienced developers’ skills, reduce learning opportunities, and fragment code understanding and confidence.Participants linked delegated work to missed skill practice, declining coding and review abilities, and weaker familiarity with generated code.Source: §3.2.1 · pp. 8–9
3.2.2 Team Level
- At team level, novices turning to LLMs first may reduce opportunities for mentorship from senior developers..Source: §3.2.2 · p. 9
3.2.3 Organisation Level
- Organisations face code-licensing, security and privacy concerns, alongside token costs that can constrain high-volume use.Some organisations prohibit LLM use or wait for licensing arrangements before adoption.Source: §3.2.3 · p. 9
3.2.4 Societal Level
- Societal concerns include misuse that may erode trust and automation that may replace workers in simpler roles while demand remains for experts.Participants raised interview cheating and misinformation as examples of misuse.Source: §3.2.4 · pp. 9–10
3.2 RQ2. How do LLMs create an imbalance in software development?
- Challenges span all four levels, with individual-level findings most numerous (n = 10), followed by organisational (n = 3), societal (n = 2), and team (n = 1) findings.Across levels, adoption can introduce operational friction, cognitive overhead, and code-quality risks, while reliance may erode skills, codebase familiarity, flow, and professional reputation.Source: §3.2.4 · p. 10
3.3 RQ3. How can software developers achieve a balanced use of LLMs?
The recommendations pair workflow and governance changes with human oversight and continued learning. They include visualising code, tailoring model choice and integration, setting review and usage policies, and upskilling developers.
3.3 RQ3. How can software developers achieve a balanced use of LLMs?
- Visualising LLM-generated code with diagrams can help developers understand and assess how it works, while also supporting review of its workflow and mental models.Participants described generating architecture diagrams; the recommendation links visualisation to code-quality review and reducing fragmented code mental models.Source: §3.3 · p. 10
- Smaller language models can serve as efficient alternatives in some solution components and reduce overall computational, memory, energy, or token costs.One participant suggested combining smaller models; the recommendation connects them to lowering resource requirements.Source: §3.3 · p. 10
- Keeping humans accountable for code entering the main branch means requiring developers to understand generated code before production use.The recommendation is intended to address code-quality and reputational risks through human review.Source: §3.3 · p. 11
- Established coding conventions may reduce communication problems with LLMs and support code quality, though the cited rationale does not explain how conventions protect developer skills.Participants reported that misleading or nonstandard variable names can confuse Copilot.Source: §3.3 · p. 11
- Clear LLM-for-software-engineering policies can guide tool use, code sharing, licensing, and handling of privacy concerns.Four participants (18.1%) reported that their organisations lacked an AI policy or that they were unaware of one.Source: §3.3 · p. 11
- Participants recommend continued upskilling, self-hosted models for privacy concerns, and workflow-integrated solutions for more contextualised assistance.The recommendations associate upskilling with maintaining market relevance, local deployment with addressing hosted-service privacy concerns, and integration with reducing interruptions.Source: §3.3 · pp. 10–11
4 DISCUSSION
The discussion uses McLuhan’s Tetrad to consider how LLMs enhance, displace, retrieve, and reverse software-development practices. It also identifies research questions about delegation, social effects, intuition, burnout, job titles, and organisational adoption maturity.
4.1 Implications for Practice
- The discussion applies McLuhan’s Tetrad of Media Effects to examine what LLMs enhance, make obsolete, retrieve, or reverse in software development.The framework is used to explore implications of both benefits and challenges.Source: §4.1 · p. 11
- Using LLMs for code improvement rather than generation, alongside Test-Driven Development, is presented as a way participants maintain control amid potential cognitive debt.The discussion links incomplete code mental models to possible loss of control over software over time.Source: §4.1 · p. 11
- Third-party LLMs may pose unacceptable data risks for company onboarding, prompting exploration of open-source models or vendor data-protection agreements..Source: §4.1 · p. 11
- LLMs can reduce effort on repetitive tasks, but the discussion says humans should remain reviewers because the models are stochastic; exploratory learning may also benefit from other developers sharing knowledge..Source: §4.1 · p. 11
- Because effort-saving automation may also reduce skills practice, participants describe limiting AI use at times to exercise their coding skills; environmental concerns were raised by only two participants.The discussion notes that those participants’ concerns focused on energy consumption and overlooked water consumption.Source: §4.1 · pp. 11–12
4.2 Implication for Research
- Further research is proposed on how task risk and developer motivation shape delegation, and on contested uses such as test and code generation and debugging.Participants disagreed about these uses; the discussion notes concerns about false assurance in test tasks and that LLMs cannot run code for debugging.Source: §4.2 · p. 12
- The discussion identifies open questions about LLMs and tight deadlines, social intelligence, developer intuition, agentic-workflow burnout, job-title inflation, and organisational maturity stages.It calls for research on supervisory burden and sustainable concurrent agent supervision, and longitudinal validation of maturity levels’ returns and team stability.Source: §4.2 · p. 12
5 RELATED WORK
The study builds on prior work on LLM benefits and challenges while extending the comparison across software-development levels and identifying practices to balance adoption’s effects.
5 RELATED WORK
- The study corroborates that LLMs can maintain developer flow, but long suggestions can disrupt it, as also reported by Barke et al..Source: §5 · p. 12
- Findings align with prior reports of reduced development time and improved code quality, while also identifying code degradation and tasks being dragged by LLMs.The authors use these contradictions to argue for a balanced approach to adoption.Source: §5 · p. 13
- The findings share with prior work the potential for LLMs to improve developers’ mental models, and add impacts at other levels, including teams.One team-level example is mitigating interruptions.Source: §5 · p. 13
- Compared with prior research on individual and organisational impacts, the study also reports team- and society-level effects, including lost mentorship opportunities and entrepreneurship.Source: §5 · p. 13
- The study extends prior impact research by identifying best practices for balancing the forward and backward effects of adopting LLMs.Figure 5 distinguishes overlap with existing literature from findings presented as new.Source: §5 · p. 13, Figure 5 · p. 13
6 EVALUATION
The evaluation addresses both the conduct of the STGT4DA application and the qualities of its outcomes, using criteria of rigour, credibility, originality, relevance, and density.
6.1 Evaluating STGT4DA Application
- The application is evaluated for credibility and rigour through reporting on recruitment, sampling, iterative data collection and analysis, memoing, coding, constant comparison, and interview evidence.Source: §6.1 · p. 13
6.2 Evaluating STGT4DA Outcomes
- The STGT4DA outcomes are assessed for originality, relevance, and density, with findings compared to related work and presented through concepts, sub-categories, codes, and interview quotes.Source: §6.2 · p. 13
7 THREATS TO VALIDITY
The authors discuss study quality across credibility, analyzability, transparency, and usefulness, noting boundaries in recruitment, analysis, reporting, and the changing technology context.
Credibility
- The participant pool spans varied demographics and roles, but future work could include IT or project managers and professional social-media data; changing ATAI questions from nine to five points may threaten validity.The authors also describe recruiting participants as arduous, while considering the achieved sample size suitable.Source: §7 · p. 14
Analyzability
- Qualitative analysis is intrinsically subjective; the authors sought to mitigate this through pilot open coding and discussions of emerging analytical categories.Source: §7 · p. 14
Transparency
- The authors intend the reporting to be sufficiently clear and complete to support replication or transfer to other contexts, with further information in appendices and supplementary materials.Source: §7 · p. 14
Usefulness
- Rapid LLM evolution may quickly make some findings outdated, so further studies can compare them with capabilities of newer models.The findings are intended to help team leaders and IT managers assess fit to their context and needs.Source: §7 · p. 14
8 CONCLUSION
The study presents practitioners’ accounts of LLM benefits and challenges across individual, team, organisational, and societal levels, alongside recommendations to help mitigate imbalances in use.
8 CONCLUSION
- Practitioners report benefits and challenges at four levels: time savings and frictions for individuals, fewer interruptions and less mentorship for teams, cost savings and security or privacy concerns for organisations, and entrepreneurship and eroding social trust for society.The authors also provide actionable suggestions to help mitigate the resulting imbalance and support LLM integration into software development.Source: §8 · p. 14