PaperlayerView exact revision on arXiv

Source-linked AI summary

Navigating the Complexity of Generative AI Adoption in Software Engineering

Daniel Russo

arXiv:2307.06081v2cs.SE

TL;DR

The paper asks what influences Generative AI adoption in software engineering, a gap in research that has largely emphasized tools rather than adoption factors. It combines mixed methods, theory generation, and model validation to study individual, technological, and social influences. The central result is that compatibility with existing development workflows is the pivotal adoption factor.

  • Problem

    Research has largely focused on Generative AI tools themselves, leaving adoption factors across individual, organizational, technological, and environmental dimensions insufficiently examined.

  • Method

    The study combines theory-guided qualitative analysis using the Gioia Methodology with PLS-SEM validation of the Human-AI Collaboration and Adaptation Framework.

  • Results

    Compatibility with existing development workflows is the pivotal factor associated with Generative AI adoption in software engineering.

  • Takeaways & Limitations

    AI tools designed to fit existing development workflows may better support adoption than tools whose benefits are considered in isolation.

  • Takeaways & Limitations

    The qualitative analysis was conducted by a single researcher, creating potential bias and subjectivity despite the structured Gioia approach.

Abstract

from arXiv · show

In this paper, the adoption patterns of Generative Artificial Intelligence (AI) tools within software engineering are investigated. Influencing factors at the individual, technological, and societal levels are analyzed using a mixed-methods approach for an extensive comprehension of AI adoption. An initial structured interview was conducted with 100 software engineers, employing the Technology Acceptance Model (TAM), the Diffusion of Innovations theory (DOI), and the Social Cognitive Theory (SCT) as guiding theories. A theoretical model named the Human-AI Collaboration and Adaptation Framework (HACAF) was deduced using the Gioia Methodology, characterizing AI adoption in software engineering. This model's validity was subsequently tested through Partial Least Squares - Structural Equation Modeling (PLS-SEM), using data collected from 183 software professionals. The results indicate that the adoption of AI tools in these early integration stages is primarily driven by their compatibility with existing development workflows. This finding counters the traditional theories of technology acceptance. Contrary to expectations, the influence of perceived usefulness, social aspects, and personal innovativeness on adoption appeared to be less significant. This paper yields significant insights for the design of future AI tools and supplies a structure for devising effective strategies for organizational implementation.

1 INTRODUCTION

The paper examines what influences Generative AI adoption in software engineering and frames the issue as consequential for how software development may evolve. It uses a convergent mixed-methods investigation grounded in individual, technological, and social perspectives.

  • The study asks what influences the adoption of Generative AI tools in software engineering.
  • The investigation applies a convergent mixed-methods approach to Generative AI and Large Language Model adoption.
  • TAM, DOI, and SCT frame individual, technological, and social influences on AI adoption.
  • AI tools may reshape software development as they become more refined and widely adopted.
  • The article induces theory, develops a framework and hypotheses, and validates the model with PLS-SEM.

2 RELATED WORK

Related work shows that Generative AI and Large Language Models offer productivity and code-generation benefits while raising concerns about quality, trust, integration, usability, and education. It also identifies limited empirical evidence on adoption factors and on several code-generation tools.

  • Existing studies examine AI-generated code accuracy, quality, productivity, usability, and effects on software development processes.
  • Developers report integration difficulties, security worries, workflow changes, legal concerns, and time costs when using AI tools.
  • Online communities and shared discussions shape developers’ trust in AI assistance.
  • AI programming assistants reduce keystrokes and speed task completion, but users struggle with control, output alignment, understanding, editing, and debugging.
  • Empirical research evaluating code generators beyond Copilot is scarce, despite numerous tools being acknowledged in the field.
  • Research largely focuses on AI tools themselves rather than the individual, organizational, technological, and environmental factors influencing adoption.

3 THEORY GENERATION

The theory-generation phase combines TAM, DOI, and SCT to capture individual perceptions, technological attributes, and social influences on language-model adoption. It operationalizes these dimensions through questionnaire constructs and analyzes qualitative responses using the Gioia Methodology.

  • TAM, DOI, and SCT jointly address user perceptions, innovation attributes, and social-environmental influences on adoption.
  • The study analyzes individual, technology, and social-level factors in the acceptance and use of LLM-powered tools.
  • Questionnaire constructs include usefulness, ease of use, behavioral intention, compatibility, relative advantage, complexity, social influence, environmental factors, and self-efficacy.
  • Qualitative data came from 100 software engineers answering nine open-ended, theory-based questions through Prolific and Qualtrics.
  • Participants underwent prescreening and competence screening based on professional, educational, programming, and approval criteria.
  • The Gioia Methodology identifies first-order concepts, groups them into broader themes, and supports systematic theoretical progression.

4 RESULTS

The qualitative results identify multiple influences on LLM usefulness and ease of use, including efficiency, task-specific benefits, learning, experience, individual differences, interface design, and task complexity. They also document limits involving applicability and output quality.

  • Perceived Usefulness: Efficiency improvement is the most frequently mentioned perceived-usefulness dimension, accounting for 55%.
  • Perceived Usefulness: Task-specific benefits such as debugging, learning features, and generating code snippets account for 26% of aggregate dimensions.
  • Perceived Usefulness: LLMs are viewed as complementary to human expertise and judgment rather than substitutes for human oversight.
  • Perceived Usefulness: Concerns about limited applicability affect 15% of responses, while quality concerns affect 9%.
  • Perceived Ease of Use: Learning, prior experience, individual differences, intuitive interfaces, and task complexity shape perceived ease of use.
  • Perceived Ease of Use: The perceived ease of using LLMs varies with the difficulty and nature of the software-engineering task.

4.3 Behavioral Intention of LLMs in Software Engineering

Software engineers intended to adopt LLMs for code improvement, automation, learning, problem-solving, and specialized tasks, although cost, third-party dependence, and ethical concerns remained barriers. Compatibility themes emphasized efficiency, assistance, similarity to current practices, and adaptation as conditions shaping integration.

  • Software engineers intended to use LLMs to improve and maintain codebases through refactoring, design-pattern adherence, and SOLID-principle implementation.
  • LLMs were perceived as useful for automating repetitive tasks and increasing efficiency in software development.
  • Respondents intended to use LLMs for learning and problem-solving, including documentation searches, code clarification, and programming questions.
  • Respondents identified specialized uses including writing basic functionality, defining tasks, and composing emails.
  • Adoption Barriers and Concerns: Cost, dependence on third-party services, and ethical issues were reported as barriers to LLM adoption.
  • Compatibility: Compatibility was associated with improved efficiency, assistance and support, similarity to current practices, and adaptation and learning.

4.5 Complexity of LLMs in Software Engineering

Perceived complexity of LLM adoption involved job-security fears, dependence concerns, privacy and security risks, code-quality uncertainty, ethical and legal issues, bias, explainability, and integration challenges. These concerns were linked to slower diffusion or resistance and were presented as factors organizations should address when adopting LLMs.

  • 25% of respondents expressed fear of job loss and skill devaluation from LLM automation.The paper links perceived threats to job security with resistance under Diffusion of Innovation theory.
  • 16% of respondents worried that junior programmers’ overreliance on LLMs could reduce code understanding and increase bugs.
  • 15% of respondents raised data-security and privacy concerns about training LLMs on sensitive or personal data.
  • 13% of respondents questioned the quality and accuracy of LLM-generated code because errors could be introduced.
  • 8% of respondents identified ethical and legal concerns involving authorship and intellectual-property rights.
  • 7% of respondents cited output bias and limited explainability, while 6% reported integration and compatibility challenges.

4.6 Relative Advantage of LLMs in Software Engineering

The qualitative analysis identifies several dimensions of LLM relative advantage in software engineering, led by time efficiency and extending to code quality, user experience, learning, and customization.

  • Time Efficiency: 42% of respondents reported time efficiency as an LLM advantage, including faster task completion, information searches, and coding solutions.Respondents linked this benefit to communicating needs in natural language and receiving tailored solutions rapidly.
  • Code Quality: 14% of respondents highlighted improved code quality, citing clearer, more understandable code that reduced errors and strengthened robustness.The reported benefit involved more organized code and fewer errors.
  • User Experience: 11% of respondents reported enhanced user experience through easier communication and human-like interaction with LLMs.Respondents described language models as easier and faster to use because they could communicate through messages.
  • Learning and Skill Development: 9% of respondents reported that LLMs facilitated learning and skill development by simplifying technical learning and reducing mastery time.Respondents valued contextual guidance that made technical concepts easier to learn.
  • Customization and Personalization: 8% of respondents highlighted customization and personalization, especially digestible information and responses adapted to user preferences.Users could request shorter explanations or more detailed responses depending on their needs.
  • Summary: Overall, the analysis characterizes LLM relative advantage through time efficiency, code quality, user experience, learning, and customization.These dimensions are presented as benefits over existing methods that can facilitate adoption.

4.8 Self-efficacy of LLMs in Software Engineering

The analysis identifies self-efficacy and environmental factors as varied influences on LLM adoption, spanning developers’ orientations toward cutting-edge technology and organizations’ support positions.

  • Self-efficacy: The self-efficacy analysis identifies cutting-edge orientation, practicality and efficiency, and low emphasis on novelty as three aggregate dimensions.Together, these dimensions are described as shaping developers’ self-efficacy and likelihood of adopting LLMs.
  • Self-efficacy: 57% of respondents emphasized the importance of being seen as users of cutting-edge technology in their work.This orientation was associated with confidence in capabilities and staying current with technology.
  • Self-efficacy: 35% of respondents prioritized practicality and efficiency, favoring technologies that solve problems and meet client needs over novelty alone.This reflects a task-oriented focus on selecting the most appropriate tool for the job.
  • Self-efficacy: 8% of respondents assigned low importance to being seen as cutting-edge, prioritizing stability and effectiveness instead.These developers focused on reliable or task-appropriate tools rather than adopting the latest technologies.
  • Environmental Factors: Organizational environmental attitudes ranged from supportive and neutral positions to conditional, limited, and absent support for LLM adoption.The study frames these attitudes as contexts shaping organizational integration of LLMs.
  • Environmental Factors: Supportive attitudes were most prevalent at 42%, while neutral stance accounted for 20%, conditional support 19%, limited support 12%, and lack of support 15%.Conditional support depended on clear benefits or alignment with organizational objectives; limited support restricted use to selected tasks, often amid security or privacy concerns.

4.10 Key Insights of the Qualitative Study

The qualitative findings describe LLM adoption as shaped by perceived usefulness and ease of use, compatibility, social and personal factors, and organizational context, alongside implementation concerns requiring human oversight.

  • Key Insights: LLMs were associated with automating repetitive tasks, improving problem-solving, supporting learning, enhancing code quality, assisting debugging, and increasing efficiency.These findings summarize the reported potential benefits of LLMs in software engineering.
  • Key Insights: Adoption depends on perceived ease of use, including integration with existing tools and workflows, documentation, customization, adaptability, and developer-community support.These factors are described as facilitating seamless integration into software engineering.
  • Key Insights: Behavioral intention to adopt LLMs is shaped collectively by perceived usefulness, perceived ease of use, social influence, and facilitating conditions.The study presents these factors as creating an environment conducive to integration.
  • Key Insights: Compatibility includes efficiency, assistance, similarity to current practices, and adaptation, while concerns involve dependency, privacy, security, displacement, accuracy, reliability, and explainability.The findings present adoption as involving both perceived fit and concerns about the consequences and dependability of LLM use.
  • Key Insights: Relative advantage was associated with time efficiency, code quality, user experience, learning and skill development, and customization and personalization.These dimensions represent perceived benefits over traditional methods.
  • Key Insights: Peer influence ranged from none to high, while self-efficacy and organizational factors varied with developers’ orientations and workplace support or resistance.The study emphasizes diverse social and environmental contexts surrounding LLM adoption.
  • Key Insights: Overall, LLM adoption is characterized as multifaceted, involving interacting individual, technological, social, and organizational factors.The study presents these dimensions as shaping LLM integration in software engineering.

5 THE HUMAN-AI COLLABORATION AND ADAPTATION FRAMEWORK (HACAF)

The HACAF is a tailored framework for understanding LLM adoption by extending established acceptance theories with constructs reflecting social, personal, organizational, and workflow-related complexity.

  • Framework: HACAF is designed to understand and predict adoption of Generative AI tools in software engineering.Its components draw on TAM, DOI, SCT, UTAUT, and personal innovativeness.
  • Framework Novelty: The framework evolves TAM, DOI, and SCT because the qualitative investigation indicated that a more nuanced model was necessary.HACAF incorporates additional facets revealed by the research rather than simply combining existing theories.
  • Framework Novelty: HACAF adds UTAUT constructs for social influence and facilitating conditions and incorporates personal innovativeness to represent individual differences in adoption behavior.These additions address qualitative findings about social context, organizational support, and variability among engineers in similar environments.
  • Framework Contribution: The framework combines established theories with empirical findings to represent the multifaceted dynamics of LLM adoption in software engineering.It is intended to provide a tailored basis for deeper understanding and further empirical scrutiny.
  • Framework Constructs: Perceptions about the technology cover usefulness, ease of use, and relative advantage, whereas compatibility captures fit with existing values, experiences, needs, and workflows.The qualitative evidence emphasizes practical benefits and alignment with current software development practices.
  • Framework Constructs: Social factors combine social influence with computer self-efficacy, while personal and environmental factors combine personal innovativeness with organizational support.The model links these constructs to peer approval, confidence in mastering LLMs, experimentation, and perceived facilitating conditions.
  • Framework Structure: HACAF operationalizes four determinants of intention to use LLMs: perceptions about the technology, compatibility factors, social factors, and personal and environmental factors.These determinants are translated from the theoretical foundation into hypotheses represented in Figure 1.
  • Hypotheses: The hypotheses propose that technology perceptions affect compatibility and social factors, which, alongside those constructs and personal-environmental moderation, relate to intention to use LLMs.HACAF specifies positive paths from perceptions to compatibility and social factors, and from compatibility and social factors to intention.

6 THEORY VALIDATION

The study quantitatively validated a theory-informed model of Generative AI adoption using survey-based PLS-SEM. Results supported measurement quality and identified compatibility factors, alongside technology perceptions, as especially important for intention to use.

  • Quantitative validation: The quantitative phase translated qualitative themes into measurable scales and survey hypotheses, then evaluated constructs and structural relationships using PLS-SEM.The survey instruments and model relationships were grounded in the initial qualitative study.
  • Survey design: The survey used adapted prior instruments and uni-dimensional items rated on a 7-point Likert scale, after a three-respondent pre-test refined usability and phrasing.Minor issues identified during pre-testing were addressed before the survey proceeded.
  • Measurement evaluation: All construct coefficients were below the predefined HTMT threshold, indicating that the model’s constructs represented distinct phenomena.The reported HTMT criterion was below 0, although the passage appears to truncate the intended threshold value.
  • Measurement evaluation: Cronbach’s Alpha, rho_a, and rho_c exceeded the required 0.60 threshold, supporting the reliability of the measurement items.Cross-loading analysis also removed weak items and improved AVE, reinforcing the model’s measurement robustness.
  • Structural and predictive results: Four hypothesized relationships were statistically significant at the 5% level, with p-values below 0.05 and T statistics above 1.96.The model’s predictive evaluation used PLSpredict with ten data partitions and ten repetitions.
  • Structural and predictive results: Compatibility factors had the strongest effect on intention to use LLMs, while technology perceptions and compatibility factors were the most significant constructs influencing intention.Technology perceptions also showed a reported total effect of 0.540 on intention to use, illustrating their high importance.

7 DISCUSSION

The HACAF findings place compatibility with existing workflows at the center of Generative AI adoption, while social, personal, and environmental influences show weaker direct relationships with intention to use. The discussion also frames these results as early evidence requiring further validation and attention to ethical, sampling, and longitudinal limitations.

  • Technology Perceptions: The quantitative findings contradicted traditional Technology Acceptance Model expectations by showing that technology perceptions did not directly influence intention to use LLMs.Compatibility with existing work processes significantly shaped perceptions about the technology.
  • Social Factors: Social Factors did not significantly contribute to intention to use LLMs, despite perceptions about the technology positively influencing Social Factors.The authors contextualize this unexpected relationship within the nascent stage of Generative AI transformation.
  • Personal and Environmental Factors: Personal and Environmental Factors influenced technology perceptions but did not directly affect intention to use, limiting the direct role of personal innovativeness and organizational support.The finding suggests that shaping perceptions does not necessarily translate into adoption.
  • Compatibility Factors: Compatibility Factors emerged as primary drivers of Generative AI adoption, especially when tools fit established software engineering workflows.Seamless integration into coding, debugging, testing, and deployment processes is associated with greater adoption, whereas disruption may prompt resistance.
  • Implications: The study provides early practical guidance for designing AI tools around developer needs and existing workflows, while warning that compatibility should not eclipse ethical and security risks.The paper notes harmful outputs, bias, and security vulnerabilities as concerns raised by a minority of informants and prior research.
  • Limitations: The findings remain an initial snapshot because HACAF requires long-term longitudinal evaluation, and the sample is not representative of the software engineering population.Additional limitations include single-researcher qualitative analysis, self-reported single-informant measures, and restricted generalization from the Prolific community.

8 CONCLUSION

The conclusion presents HACAF as a mixed-methods framework for understanding Generative AI adoption during the field’s early transformation. Its central finding is that compatibility with existing development workflows is more pivotal than perceived benefits alone, informing user-focused tool design while motivating longitudinal research.

  • Contribution: The study develops HACAF to provide a nuanced account of Generative AI adoption in software engineering’s nascent transformation.
  • Findings: Compatibility Factors play a pivotal role in adoption because AI tools need to fit existing development workflows rather than rely solely on perceived benefits.
  • Implications: The findings offer early guidance for designing and refining user-focused AI tools by addressing developer concerns and optimizing workflow compatibility.
  • Future Research: Comprehensive assessment of HACAF requires long-term longitudinal studies to examine how adoption factors evolve as Generative AI matures and gains wider adoption.

A APPENDIX A (SURVEY INSTRUMENT)

The appendix documents survey items, noting that items marked with an asterisk were removed because they loaded insufficiently onto their latent variable.

  • Survey Instrument: Table 20 describes the survey items and identifies items dropped because of insufficient loading onto their latent variable.

B APPENDIX B (OVERFITTING ANALYSIS)

The appendix evaluates whether the model is correctly specified and generalizes without overfitting. Residuals were mostly random, while test-set and cross-validation errors were reasonably close.

  • The notably high effect size between ‘Perception about the Technology’ and ‘Compatibility Factors’ raised a potential overfitting concern requiring further investigation.
  • Residuals for ‘Intention to Use’ were mostly randomly scattered around zero, suggesting random errors and appropriate model specification.
  • The overfitting analysis used an 80% training and 20% testing split, evaluating predictions with mean squared error (MSE).
  • 0.523 was the approximate test-set MSE, where lower values indicate better fit.
  • 0.676 was the approximate mean cross-validation MSE across five folds, reasonably close to the test MSE and suggesting limited overfitting.
Loading 2307.06081v2…