The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study

1 Introduction

• The review addresses how LLM-assistants affect developer productivity, which includes efficiency and quality as well as human-centered factors.The introduction identifies satisfaction, collaboration, and cognitive load as part of productivity.Source: §1 · p. 2
• The study maps 39 peer-reviewed studies published from 2014 through December 2024 and analyzes their methods, evaluation practices, and productivity dimensions.It also synthesizes reported benefits and risks.Source: §1 · p. 2
• Reported benefits include reduced task-initiation overhead and support for code-adjacent tasks, while risks include over-reliance and flow disruption.The review analyzes these effects across the primary studies.Source: §1 · p. 2
• The review applies the SPACE framework and uses McLuhan’s Tetrad to consider broader socio-technical implications.It also offers recommendations and a replication package containing study data, selection decisions, and exclusion rationales.Source: §1 · p. 2

2 Background

• Software development productivity is also social and organizational: as projects scale across modules and teams, communication bottlenecks and coordination overhead can arise.Informal communication becomes harder to sustain, requiring deliberate information-sharing and decision-tracking mechanisms.Source: §2 · p. 3
• Developer productivity has no universal definition or measurement consensus, and simple output measures do not capture its broader human, social, and organizational dimensions.Examples of traditional proxies include lines of code and task completion time.Source: §2 · p. 3
• The DevEx model organizes developer experience around feedback loops, cognitive load, and flow state.It is presented as a practical framework for assessing productivity from the developer’s point of view.Source: §2 · p. 3
• SPACE characterizes productivity through Satisfaction and well-being, Performance, Activity, Communication and collaboration, and Efficiency and flow.These dimensions cover outcomes, observable work, coordination, and uninterrupted progress.Source: §2 · p. 3

3 Systematic Literature Review Methodology

• The review seeks to synthesize fragmented knowledge, characterize research methods and instrumentation, and identify gaps to guide future research.Its overall aim is to establish the current state of evidence on LLM-assistants’ effects on developer productivity.Source: §3 · p. 4
• The review examines primary studies’ methods and instruments, reported benefits and risks, and the productivity dimensions investigated.These are the three dimensions specified for its analysis.Source: §3 · p. 4
• The methodology is grounded in Kitchenham and Charters’ systematic-review guidelines, adapted for software engineering.The paper introduces its research questions as the guide for the review.Source: §3 · p. 4

software developer productivity?

• RQ0 contextualizes the emerging research landscape surrounding LLM-assisted software development.The review examines the temporal distribution of publications, publication venues and disciplinary focus, and authorship patterns.Source: software developer productivity? · p. 4
• The landscape analysis considers when studies were published and where and by whom they were conducted.These are the three angles named for RQ0.Source: software developer productivity? · p. 4
• The next research question concerns the methods, procedures, and instruments used in peer-reviewed studies.The supplied text introduces this focus as RQ1.Source: software developer productivity? · p. 4

• RQ1 examines the empirical strategies used to study the relationship between LLM-assistants and software developer productivity.Source: that investigate the impact of… · p. 4
• The review examines the procedures and study designs used to conduct these investigations.Source: that investigate the impact of… · p. 4
• It also examines the instruments and metrics used to evaluate software developer productivity.Source: that investigate the impact of… · p. 4

• RQ2 summarizes findings on LLM-assistant effects on software development practices and reported benefits and risks across diverse study settings.Source: RQ2: What is the impact of… · p. 4
• The question aims to provide a structured understanding of effects on developers in practice and the trade-offs these tools introduce.Source: RQ2: What is the impact of… · p. 4
• RQ3 asks which dimensions of developer productivity are investigated.Source: RQ2: What is the impact of… · p. 4

the SPACE framework?

• The review maps each study’s main focus to SPACE dimensions—Satisfaction and well-being, Performance, Activity, Communication and collaboration, and Efficiency and flow—to identify operationalizations and underexplored areas.The mapping is intended to guide future research.Source: the SPACE framework? · p. 4

3.1 Pre-review mapping

• Pre-review mapping followed Kitchenham and Charters, defining research questions and eligibility criteria, identifying control papers, and iteratively refining search strings.Detailed steps, including control-paper identification and query refinement, are in the supplemental appendix [20].Source: §3.1 · p. 5

3.1.1 Control papers identification.

• A pilot search manually screened titles and abstracts and used one backward-and-forward snowballing round to identify 17 control papers for validating the search string.The inclusion criteria required an English, accessible full-text study published in 2014 or later that investigated AI or LLM effects on software developer productivity.Source: §3.1.1 · p. 5

3.1.2 Query formulation and refinement.

• The finalized search string, refined over five iterations against control papers, successfully retrieved all 17 control papers.Searches covered six digital libraries and were restricted to titles, abstracts, and keywords.Source: §3.1.2 · p. 6

3.2 Primary Study Selection Process

• From 9,756 initial records, title-and-abstract screening retained 228 for full-text review; full-text screening excluded 189, and snowballing added five to reach 44 studies.The 44-study set preceded the quality assessment that produced the final 39-study set.Source: §3.2 · pp. 7–8

3.3 Quality Assessment

• After quality assessment excluded five studies, the final set contained 39 primary studies, most rated above 3.Studies were assessed against 11 criteria using a five-point Likert scale and a 50% average-score threshold.Source: §3.3 · p. 8

3.4 Data Extraction and Synthesis

• Over three months, the authors used qualitative synthesis and an initial thematic analysis to extract study goals, tools, designs, tasks, settings, and key results.The synthesis followed Kitchenham’s guidelines.Source: §3.4 · p. 9
• The review used an initial thematic analysis followed by targeted iterations for methods, benefits and risks, and mapping studies to SPACE, with findings jointly validated against source texts.The first and last authors cross-checked citations against original text to ensure accuracy and traceability.Source: §3.4 · p. 9

4.1 Publication years

• Research interest rose in 2022, and 77% of included studies were published in 2024; only four studies appeared from 2014 through 2022 [50] [51].The authors note that the 2022 rise coincided with ChatGPT’s release.Source: §4.1 · p. 9

4.2 Author distributions

• Of 154 authors, 147 had one publication, six had two, and Igor Steinmacher was the most prolific with three, reflecting a topic just starting to build momentum.Source: §4.2 · p. 9

4.3 Publication venues

• Software Engineering and Computer Science venues published 46% of primary studies, while Human-Computer Interaction venues accounted for 18% (7 of 39).The remaining studies were distributed across specialized areas including AI for Software Engineering, education, information systems, and human aspects.Source: §4.3 · p. 10

5.1 Distribution of the research strategies

• Laboratory experiments were most common, used by 38% (15 of 39) primary studies; field studies ranked second at 23% (9 of 39), followed by surveys at 15% (6 of 39).Experimental simulations accounted for 13% (5 of 39).Source: §5.1 · pp. 11–12

5.2 Distribution of the research procedures

• Self-reported data appeared in 90% of studies, 41% used experimental settings, and 69% adopted mixed methods.The most common method pairing was a user experiment with a survey, used to triangulate perceptions and measured performance.Source: §5.2 · p. 13
• Formative goals characterized 59% of studies and summative goals 41%, while 67% used both quantitative and qualitative analysis.Qualitative-only analysis appeared in 21% of studies and quantitative-only analysis in 13%.Source: §5.2 · p. 14

5.3 Evaluation instruments

• Self-reported instruments remained predominant, while 15 of 39 studies used validated instruments such as SPACE, NASA-TLX, or TAM.Behavioral and performance metrics were mostly associated with controlled studies; field and sample studies primarily relied on surveys, interviews, and open-ended feedback.Source: §5.3 · pp. 14–15

5.3.1 Time to completion.

• Task completion time was the most frequently used performance metric, appearing in 31% (12 out of 39) of empirical studies, mostly laboratory experiments.One field study compared throughput and cycle time before and after Copilot integration, while other studies used person-days or issue-resolution duration.Source: §5.3.1 · p. 15

5.3.2 LLM suggestions acceptance rate.

• Accepted-suggestion frequency strongly correlated with self-reported productivity, but acceptance rate alone may favor common languages and routine tasks without improving developer outcomes.Studies also measured the number of accepted suggestions and the proportion of code authored by assistants.Source: §5.3.2 · p. 16

5.3.3 Mental effort and cognitive load.

• None of the identified studies used biomedical sensors to measure cognitive load; one used eye-tracking, and six used NASA-TLX questionnaires.NASA-TLX studies used comparative experimental settings.Source: §5.3.3 · p. 16
• Cognitive-load findings were mixed: studies reported improvements [62] [72], others neutral effects [51] [57], and only one significantly worse experience in frustration level [61].Reported variability also spans participant expertise, task design, and assistant capabilities across studies.Source: §5.3.3 · p. 16

5.3.4 Econometric analysis.

• Coding exhibited the highest reported gains: average throughput improvement was 24%, and 26% was also reported without a specified measure; throughput negatively correlated with code quality (r = –0.45).LLM-assistant effectiveness depends heavily on organizational readiness and the ability to balance speed with software quality.Source: §5.3.4 · pp. 16–17

• The authors conduct a thematic analysis of findings reported in each primary study to identify recurring themes.Source: §6 · p. 17
• Multiple studies identify recurring benefits and risks associated with LLM-assistants.Source: §6 · p. 17
• Figure 6 summarizes theme frequencies as benefits on the left and risks on the right.Source: §6 · p. 17, Figure 6 · p. 17

6.1.2 Minimize online code search.

• LLM-assistants reduce information-retrieval effort and aid syntax recall and unfamiliar APIs, but comparative findings vary: ChatGPT performs better on algorithmic and library tasks, while Stack Overflow performs better on debugging [54] [60].The comparison found no statistically significant difference in task completion time.Source: §6.1.2 · p. 19

6.1.3 Automate trivial/ repetitive tasks.

• LLM assistance reduces trivial tasks through boilerplate generation [51] [54] [60], reduces keystrokes and typing effort [60], and supports test generation, with unit tests viewed as repetitive and well-suited to assistance.These tools could automate all routine tasks, freeing developer time for more complex tasks.Source: §6.1.3 · p. 19

6.1.4 Support knowledge acquisition.

• LLM-assistants support learning: 69% reported enhanced learning, expert consultation accounted for 62% of analyzed conversations, and 75% of survey respondents called ChatGPT a helpful learning tool [51] [56].They can also lower the barrier to learning new frameworks by letting developers start without navigating extensive tutorials [64].Source: §6.1.4 · pp. 19–20

6.1.5 Support code-adjacent tasks.

• LLM-assistants support code-adjacent work including ideation, requirements, documentation, quality assurance, emails, meeting minutes, and onboarding materials [54] [58] [60].AI-assisted teams were also reported to document more GitHub issues and coordinate more effectively.Source: §6.1.5 · p. 20

6.1.6 Reduce task initiation overhead.

• LLM-assistants lower task-initiation barriers by providing starting points and scaffolding; 55% of participants used ChatGPT primarily to generate initial code scaffolding before shifting to more independent workflows [51] [57].They can build early momentum by reducing the time and cognitive effort required to start a project [51].Source: §6.1.6 · p. 20

6.1.7 Improve code quality.

• Reported code-quality benefits include an 18% improvement across metrics in projects developed with LLM-assistants, fewer code smells in three of five teams, and fewer defects in all five teams.Developers also report using assistants to rewrite and improve code.Source: §6.1.7 · p. 20
• ChatGPT users showed higher code quality than Stack Overflow users in algorithmic and library tasks, but the control group performed better in debugging.Source: §6.1.7 · p. 20
• 51% lower error rate was reported for LLM-assisted code translation [51].The study evaluated translation-related metrics including Translation Error, Language Error, and Spurious Error.Source: §6.1.7 · pp. 20–21

6.1.8 Support troubleshooting / debugging.

• LLM-assistants support troubleshooting by explaining error messages, suggesting fixes, accelerating bug identification, detecting early defects, and reducing documentation searches.Source: §6.1.8 · p. 21

6.2.1 Fail to meet requirements.

• LLM-assistants may fail functional and non-functional requirements, producing out-of-context or generic suggestions that require further refinement [60].Fifty percent of participants cited missing or misunderstood requirement context as key issues with ChatGPT 3.5; code-generation correctness scores ranged from 31.1% for Amazon CodeWhisperer to 65.2% for ChatGPT.Source: §6.2.1 · p. 22

6.2.2 Promote over-reliance and cognitive offloading.

• Heavy reliance on LLM-assistants raises concerns about eroding critical thinking and independent thought among novices, students, and professional developers [61] [64].Students using an AI tutor still worried that reliance could hinder learning progress, while survey respondents across experience levels reported diminished ability to think independently [64].Source: §6.2.2 · p. 22

6.2.3 Disrupt the flow.

• LLM-assistants may distract developers when rapid suggestions impede code understanding, compete for attention, or increase notification fatigue [54].Open-source human-bot teams may experience notification fatigue from increased automated activity.Source: §6.2.3 · p. 22
• Developers spend an average of 51.5% of coding-session time in LLM interaction states, including verifying suggestions, crafting prompts, and deferring thought.The studies attribute flow disruption to interruptions such as unwanted suggestions, interface switching, and verbose answers.Source: §6.2.3 · p. 22

6.2.4 Limit code quality.

• LLM-assistants can generate inaccurate, inefficient, vulnerable, or context-insensitive code, and whether they improve or degrade code quality remains unresolved.A study of 70 global companies found a moderate negative correlation (r = −0.45) between reported productivity and code-quality gains; variation in study scope and evaluation methods makes direct comparisons difficult.Source: §6.2.4 · p. 23

6.2.5 Reduce team collaboration.

• Excessive use can lead developers to consult chatbots instead of colleagues, reducing team discussions, communicative learning, and organic idea-sharing [56] [64].Conversational assistants primarily support one-on-one interactions and are not well-suited to effective team coordination.Source: §6.2.5 · p. 23
• Satisfaction studies examined developer experience, self-efficacy, trust, and cognitive load, but none of the empirical studies examined well-being.Developer experience was the main focus, while cognitive load was the second most commonly investigated satisfaction sub-dimension.Source: §6.2.5 · p. 26
• Performance studies commonly assessed quality through unit tests, functional correctness, and code smells, while only three studies examined business-related product impact.Impact measures included cost savings, product-quality improvements, and delivery speed.Source: §6.2.5 · p. 26
• Efficiency studies examined task timing, automation of repetitive work such as boilerplate code [58] [60], and interruptions or flow, with findings including both fewer interruptions and new distractions.Studies assessed temporal efficiency using task-completion metrics or developer perceptions.Source: §6.2.5 · p. 26
• Activity was examined in 12 of 39 studies, often alongside Efficiency and Performance, using counts such as actions taken or tasks completed [54, 55, 65, 73].Measures could include acceptance rates, suggestions shown, and completions changed or unchanged.Source: §6.2.5 · p. 26
• Only 3 of 10 Communication studies examined human–human collaboration with an LLM in the loop, leaving team communication and coordination effects insufficiently understood.The review recommends more investigation of team dynamics in LLM-assisted workflows, amid concerns that over-reliance may reduce team collaboration.Source: §6.2.5 · pp. 26–27
• Satisfaction was studied most often (77%, 30 out of 39), followed by Performance (64%, 25 out of 39) and Efficiency (59%, 23 out of 39); Activity (31%, 12 out of 39) and Communication (26%, 10 out of 39) were least studied.Satisfaction was mainly measured through self-reports; Performance studies often examined quality, and Efficiency studies often examined task timing or automation.Source: §6.2.5 · p. 27

8.1 In-depth synthesis across studies using the McLuhan Tetrad

• The Tetrad extends measurement-focused productivity frameworks to interpret shifts in development practices, decision-making, and team dynamics through enhancement, obsolescence, retrieval, and reversal.Figure 9 maps these four dimensions to the implications of LLM-assistants for developer productivity.Source: §8.1 · pp. 27–28, Figure 9 · p. 28

8.1.1 Enhance.

• LLM-assistants can accelerate development, lower barriers to complex tasks, support knowledge acquisition and debugging, and suit boilerplate, syntax recall, scaffolding, and prototyping.The discussion advises strategic use rather than expecting uniform gains across development activities.Source: §8.1.1 · p. 28

8.1.2 Reverse.

• Uncritical reliance can shift developers from code production to reviewing generated output, weaken validation and collaboration, and negatively affect code quality.The discussion links reduced peer communication to turning to the assistant instead of teammates for feedback.Source: §8.1.2 · pp. 28–29

8.1.3 Obsolesce.

• Conversational assistants can displace online search and Q&A use, but may diminish independent verification, comparison of solutions, and search and validation skills.The discussion notes these skills remain essential when generated outputs are incorrect or incomplete.Source: §8.1.3 · p. 29

for traditional information-seeking. They should continue maintaining proficiency in search and validation

• Developers should retain information-seeking proficiency and cross-check LLM-assistant suggestions against reliable sources.Source: for traditional information-seeking.… · p. 29

8.1.4 Retrieve.

• LLM-assistants can revive documentation, requirements elicitation, and legacy-system work, while human expertise remains important.The review notes that legacy-system support is currently limited, but these tools may extend productivity beyond coding speed.Source: §8.1.4 · p. 29

8.2 Implications and recommendations for software developers

• Developers should calibrate trust by validating generated code, checking unfamiliar APIs against official documentation, and periodically coding without assistance.Excessive trust can foster complacency, while insufficient trust can cause frustration and underuse; hallucinations and incorrect outputs can erode trust.Source: §8.2 · p. 30
• LLM-assistant use shifts developers toward reviewing and refining outputs; one study found participants spent over 50% of their time on evaluation activities.Time saved generating code can be lost to evaluation and refinement, particularly on complex tasks.Source: §8.2 · p. 30
• Developers should reserve time for systematic output validation, while teams preserve pair programming, reviews, and shared decision records to protect collaboration and knowledge sharing.Customizing suggestion frequency can limit interruptions; the extent to which adoption reduces knowledge sharing and pair programming remains unclear.Source: §8.2 · p. 31
• Organizational readiness and governance matter: industry-level analysis found a moderate negative correlation (r = −0.45) between throughput and code quality.The review highlights accountability and quality-assurance processes to balance speed with quality.Source: §8.2 · p. 31
• Organizations should align productivity gains with code quality through calibrated-trust training, task policies, enhanced reviews for high-risk code, and monitoring of team knowledge sharing.The recommendations also emphasize collaboration and continuous learning alongside automation, given the reported negative correlation between throughput and quality.Source: §8.2 · p. 32
• Organizations should disclose AI-assisted contributions, define accountability and traceability, and integrate ethical review and bias testing into quality assurance.Opaque systems may omit code sources, hindering validation and creating accountability gaps when generated code introduces defects, vulnerabilities, or copyrighted material [58].Source: §8.2 · p. 32

8.3 Implications and recommendations for researchers

• No primary study directly examines developer well-being, and human-human collaboration remains understudied, with only 3 of 10 Communication studies addressing it.The review identifies unresolved questions about pair programming, knowledge sharing, and collective code ownership.Source: §8.3 · p. 32
• Long-term effects remain largely unknown, while code-quality findings conflict across study contexts and evaluation criteria.Short-term laboratory studies cannot establish whether gains persist or reveal risks such as skill erosion and cumulative technical debt.Source: §8.3 · pp. 32–33
• Cognitive-load findings are mixed, and the review recommends shared evaluation frameworks and longitudinal, field, and team-based studies.Recommended approaches include validated cognitive-load measures, extended tracking of productivity, quality, and skills, and quantitative-qualitative triangulation.Source: §8.3 · p. 33
• 90% of studies examine at least two SPACE dimensions, but only 15% extend beyond three; Communication remains underexplored alongside human-human collaboration.Satisfaction, Performance, and Efficiency are most frequently investigated, while human-human collaboration is the least investigated Communication sub-dimension.Source: §8.3 · p. 33
• Researchers should account for developer expertise, task complexity, and domain context because these factors can alter LLM-assistant productivity outcomes.Novices may gain immediately but risk over-reliance and skill erosion [61], while experts may achieve sustained efficiency through selective, critical use [60].Source: §8.3 · p. 34
• Researchers should model expertise, task complexity, and domain context as covariates and report them to improve comparability across studies.The review also recommends defining task complexity, documenting organizational context, and replicating studies across diverse populations, tools, and settings.Source: §8.3 · p. 34

9.1 Study Selection and Classification Rigor

• The search may omit relevant studies through eligibility rules and difficulty identifying human-centered work amid broad LLM research and technical applications.The review excludes shorter and non-peer-reviewed work and notes that broad search terms can retrieve studies focused on technical applications rather than developer productivity.Source: §9.1 · pp. 34–35
• Open-ended questions and SPACE mapping require interpretive decisions that may introduce subjectivity, despite multi-author validation and collaborative coding discussions.The authors report validation meetings and adapted prior definitions for coding findings to SPACE sub-dimensions.Source: §9.1 · p. 35

9.2 Limitations of the Primary Evidence Base

• Formative and laboratory studies strengthen internal validity but can limit ecological validity because results often come from isolated tasks and controlled settings.The review reports 59% formative studies and 38% laboratory experiments; such results may not capture large-scale industry development.Source: §9.2 · p. 35
• Methodological diversity hinders comparability, with no standardized metrics for contested concepts such as code quality and cognitive load.Code quality measures include cyclomatic complexity, code coverage, defect density, and functional correctness.Source: §9.2 · p. 35
• The review may become temporally incomplete as LLM capabilities, evaluation methods, and developer practices evolve and new findings emerge quickly.Search and extraction ended in 2024, and 77% of included studies were published that year; the authors encourage periodic updates.Source: §9.2 · p. 35

• The review synthesizes 39 peer-reviewed studies, analyzing their methods, evaluation practices, and productivity dimensions through established conceptual frameworks.Source: §10 · p. 36
• Reported benefits include reduced task-initiation overhead, accelerated development, and code-adjacent support, while risks include over-reliance, disrupted flow, and reduced team communication or collaboration.Source: §10 · p. 36
• Code-quality findings are mixed, with improvements and degradations depending on context, task design, and evaluation criteria.Source: §10 · p. 36
• 90% of studies consider multiple productivity dimensions, but relatively few extend beyond three, and communication and human-human collaboration remain underexplored.Source: §10 · p. 36
• The authors recommend team-based studies that capture software development’s dynamic socio-technical nature and provide a publicly available replication package.Source: §10 · p. 36