PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards an Understanding of Large Language Models in Software Engineering Tasks

Zibin Zheng, Kaiwen Ning, Qingyuan Zhong, Jiachi Chen, Wenqing Chen, Lianghong Guo, Weicheng Wang, Yanlin Wang

arXiv:2308.11396v3cs.SE

TL;DR

Software-engineering applications of LLMs lack systematic research on their integration and effectiveness. This paper reviews 123 post-2022 works, categorizes them across seven tasks, and synthesizes evaluations. It finds stronger performance on syntax-oriented tasks than semantics-oriented tasks, while model iterations continue to improve.

  • Problem

    Systematic evidence about how LLMs integrate with software engineering and how effectively they handle its tasks remains limited.

  • Method

    The paper screens literature from multiple databases, selects 123 works, categorizes them into seven software-engineering task types, and compiles evaluation studies.

  • Results

    LLMs perform well on syntax-oriented tasks such as code summarization and repair but less satisfactorily on semantics-oriented tasks such as code generation and vulnerability detection.

  • Takeaways & Limitations

    LLMs currently serve as assistants for software developers, while continued model and version improvements indicate potential for better performance.

  • Takeaways & Limitations

    The review’s cross-task conclusions may be biased because evaluated models and software-engineering tasks vary across studies.

Abstract

from arXiv · show

Large Language Models (LLMs) have drawn widespread attention and research due to their astounding performance in text generation and reasoning tasks. Derivative products, like ChatGPT, have been extensively deployed and highly sought after. Meanwhile, the evaluation and optimization of LLMs in software engineering tasks, such as code generation, have become a research focus. However, there is still a lack of systematic research on applying and evaluating LLMs in software engineering. Therefore, this paper comprehensively investigate and collate the research and products combining LLMs with software engineering, aiming to answer two questions: (1) What are the current integrations of LLMs with software engineering? (2) Can LLMs effectively handle software engineering tasks? To find the answers, we have collected related literature as extensively as possible from seven mainstream databases and selected 123 timely papers published starting from 2022 for analysis. We have categorized these papers in detail and reviewed the current research status of LLMs from the perspective of seven major software engineering tasks, hoping this will help researchers better grasp the research trends and address the issues when applying LLMs. Meanwhile, we have also organized and presented papers with evaluation content to reveal the performance and effectiveness of LLMs in various software engineering tasks, guiding researchers and developers to optimize.

1 Introduction

Software engineering spans costly, expertise-dependent tasks, motivating automation; existing task-specific methods face generalization, reliability, and coverage problems, while LLMs offer potential through learned code syntax and semantics.

  • Motivation: Software engineering includes requirements, design, development, testing, and maintenance, requiring expertise to produce reliable and maintainable software.This dependence increases human-resource costs and complicates development and maintenance.
  • Motivation: Automation is pursued to make software production and maintenance more efficient across tasks such as code generation, test generation, and vulnerability detection.
  • Challenges: Existing task-specific automation methods may not generalize universally and can introduce vulnerabilities or incomplete test coverage.The paper cites potentially vulnerable generated code and test cases that often fail to achieve comprehensive coverage.
  • LLM Opportunity: LLMs trained on code-containing corpora can learn code syntax and semantics, supporting tasks such as code completion and code summarization.
  • Study Scope: The study reviews the LLM–software-engineering intersection using screened literature published after 2022 and defines LLMs as transformer-based models with at least 0.8 billion parameters.The supplied passage states that the literature was gathered from six databases and screened using card sorting.

RQ1: What are the current works focusing on combining LLMs and software engineering?

The review organizes LLM–software-engineering research into seven task categories to describe current integrations and their applications.

  • Task Categories: The 123 selected papers are categorized into Code Generation, Code Summarization, Code Translation, Vulnerability Detection, Code Evaluation, Code Management, and Q&A Interaction.

RQ2: Can LLMs truly help better perform current software engineering tasks?

The review finds that LLM performance differs by software-engineering task: syntax-oriented tasks are stronger, while semantics-oriented tasks remain less satisfactory, despite continued model improvement.

  • Findings: LLMs excel in tasks requiring code-syntax understanding, including code summarization and code repair.
  • Findings: LLMs perform less satisfactorily on tasks requiring code-semantics comprehension, including code generation and vulnerability detection.
  • Findings: LLMs continue improving across model and version iterations, retaining potential for better future performance.
  • Review Scope: The review systematically examines 123 relevant works and categorizes software-engineering tasks into seven types.
  • Evaluation Synthesis: The paper compiles evaluation results across software-engineering tasks and analyzes reasons for variation among studies.

2 Background

Transformers provide the architectural basis for mainstream LLMs through attention, parallel computation, and scalable sequence modeling, with several decoder and encoder–decoder variants.

  • Transformer Architecture: Mainstream LLMs are based on Transformers, whose attention mechanism supports parallel computation and captures long-range dependencies in large-scale text.
  • Transformer Architecture: A Transformer is a sequence-to-sequence model composed of stacked encoder and decoder blocks.
  • Attention Mechanism: Self-attention weighs the importance of tokens while modeling contextual relationships without sequential operations.
  • Architecture Types: LLM architectures include encoder–decoder, causal-decoder, and prefix-decoder structures.
  • Architecture Types: Encoder–decoder models map input sequences to output sequences and suit tasks such as translation, summarization, and dialogue generation.
  • Architecture Types: Causal decoders use a diagonal mask so each token accesses only preceding tokens during autoregressive generation.
  • Architecture Types: Prefix decoders combine bidirectional attention over prefix tokens with unidirectional attention over generated tokens.
  • Scaling and Emergence: Scaling model parameters and training data increases capacity and can produce emergent abilities absent from smaller models.

3 Methodology

The review followed a three-step process of literature search, screening, and data analysis, beginning with searches across six academic databases using six targeted keywords.

  • The methodology comprised literature search, literature screening, and data analysis.
  • Six databases were searched: ACM Digital Library, IEEE Xplore, dblp, ScienceDirect, Google Scholar, and arXiv.
  • The search used six keywords covering software engineering, LLMs, and code-oriented language models.
  • Manual screening was required because searches produced duplicate and irrelevant results across databases and keyword queries.
  • Screening excluded duplicate or irrelevant studies and applied explicit eligibility criteria, including language, topic, degree, and publication-date restrictions.

Exclusion Criteria

The review excluded studies outside its contemporary LLM–software-engineering scope, emphasizing that LLM research in software engineering became prominent around 2022.

  • Studies not written in English, doctoral or master’s theses, and papers unrelated to LLMs or software engineering were excluded.
  • 19 relevant language-model articles appeared in selected top software-engineering conferences between 2017 and 2022.
  • Transformer-based pretrained language-model applications in software engineering began around 2021, while LLM applications gained attention around 2022.
  • Pre-2022 studies emphasized pretrained language models, whereas code generation became more prominent in later LLM research because decoder-only architectures differ from encoder-decoder models.

Inclusion Topics

The review focused on papers at the intersection of LLMs and software engineering, classifying 123 studies by task and examining evidence about LLM performance.

  • Inclusion Topics: The inclusion topics covered LLMs in software engineering, applications such as code generation, and empirical studies of LLMs on software-engineering tasks.
  • Inclusion Topics: Closed card sorting used predefined relevant and irrelevant categories to systematically evaluate papers by title and abstract.
  • Inclusion Topics: Six experienced researchers independently reviewed results and discussed their classifications, yielding 123 relevant papers.
  • Inclusion Topics: The review organized papers around two questions: current LLM–software-engineering applications and LLM performance on software-engineering tasks.
  • Inclusion Topics: ICSE 2024 provided a validation set of 40 LLM-related papers that fit the seven selected categories.
  • Inclusion Topics: Performance analysis examined papers offering critical or explicit opinions, particularly instance studies and survey articles.

4 LLMs in Software Engineering Tasks

The paper organizes LLM–software engineering research into seven task categories and reviews how LLMs are applied, evaluated, and improved across these areas. The surveyed tasks include code generation, summarization, translation, vulnerability detection and repair, evaluation, management, and Q&A interaction.

  • Seven categories structure the review: Code Generation, Code Summarization, Code translation, Vulnerability Detection, Code Evaluation, Code Management, and Q&A Interaction.The categories are based on the software engineering tasks addressed in the selected papers.
  • Code Generation: LLM-based code generation research evaluates generated-code capabilities and develops methods to improve correctness, efficiency, and related properties.Code completion, refactoring, and augmentation are included within the code generation category.
  • Code Summarization: Code summarization uses LLMs to generate comments and documentation at line, function, or module granularity, with studies evaluating models and developer-oriented applications.Applications include documentation construction and test-case generation, alongside work on robustness and developer interaction.
  • Code Translation: Code translation converts code between programming languages while preserving functionality, but current LLM research remains limited and faces correctness, data, and evaluation challenges.Multiple implementations can express the same functionality, complicating performance evaluation.
  • The review also covers vulnerability detection and repair as a reliability and security task, alongside broader code evaluation, management, and Q&A interaction research.The supplied passages define vulnerability detection and repair as identifying and fixing code vulnerabilities, while Q&A research addresses developer questions and prompt engineering.

5 Performance of LLM in SE Tasks

The review synthesizes evaluation evidence across software engineering tasks and finds that LLM performance varies by task: syntax-oriented work is stronger, while semantic tasks remain challenging. Overall, LLMs are improving but are not yet substitutes for professional programmers.

  • Evaluation approach: The review screened evaluation studies, extracted the models and tasks assessed, and compiled their datasets and evaluation metrics.Because evaluation methods and criteria differ across papers, the review converts findings into authors’ confidence levels rather than directly standardizing performance.
  • Cross-task findings: ChatGPT performs well on log summarization, referential resolution, and code summarization, but struggles with vulnerability detection and test case generation.The review also notes challenges identifying subtle code differences between syntactically similar program versions.
  • Cross-task findings: Authors usually report high confidence for program repair and Text-to-SQL, but low confidence for program vulnerability detection.These confidence patterns summarize heterogeneous studies rather than a standardized cross-paper benchmark.
  • Task-specific findings: Code summarization and code explanation generally perform well but lack robustness, while code generation often remains below a ready-to-use level.The review separately reports positive attitudes toward the future potential of LLMs in test case generation despite contradictory conclusions.
  • Overall conclusion: LLMs perform better on syntax-understanding tasks such as code summarization and repair than on semantic tasks such as code generation and vulnerability detection.The review concludes that newer model versions improve performance and that LLMs retain substantial potential.
  • Overall conclusion: LLMs still fall short of professional human programmers but can assist developers by answering development questions and providing code examples.Evaluations also report significant advancement from GPT-3 to GPT-4 and positive feedback in programming education settings.

6 Related work

Related work spans broad reviews of LLMs, generative AI, deep learning, and specialized applications, alongside studies focused on individual software engineering tasks. The paper distinguishes its contribution by systematically compiling LLM research specifically in software engineering and organizing it across task categories.

  • Positioning of this work: This paper systematically investigates, analyzes, and compiles research progress on LLMs in software engineering tasks.Its stated distinction from related work is the systematic focus on the LLM–software-engineering intersection.
  • General LLM reviews: Prior surveys review LLM backgrounds, development, applications, performance evaluation, reasoning, hallucination mitigation, compression, and ethical issues.These works address LLMs broadly rather than focusing exclusively on software engineering tasks.
  • Domain-specific reviews: Other reviews examine generative AI by input-output format or study LLM applications in domains such as text generation, biomedicine, and recommendation systems.Their scope is organized around model taxonomies or application domains rather than a comprehensive software engineering task review.
  • Software engineering reviews: Several studies focus on individual software engineering areas, including natural-language-to-code generation, AI-assisted programming, and deep learning across software engineering.These works cover selected tasks or broader AI methods, not the full intersection of LLMs and software engineering examined here.
  • Earlier PLM-based work: Previous research applied pretrained language models to variable naming, bug classification, source-code generation, code comprehension, adversarial robustness, and merge-conflict repair.Some studies combine pretrained models with program analysis to improve the reliability of generated code.

7 Threats to validity

The review identifies threats from subjective manual screening and analysis, evolving research, ambiguous task boundaries, and variability or potential bias in the reviewed evaluations.

  • Internal Validity: Manual screening and content analysis may introduce subjectivity, although multiple experienced participants independently analyzed results and resolved disagreements.The protocol included non-coauthors with more than three years of software engineering research experience.
  • Internal Validity: The software engineering task classification is an early-stage structure that may not fully encompass all task aspects or reflect practice.The authors state that it nevertheless captures a significant portion of important software engineering tasks.
  • Internal Validity: Evaluation results vary across studies because they use different datasets, software engineering tasks, and evaluation perspectives.The review organizes evaluation details to help readers understand this variability.
  • External Validity: The review cannot encompass future work because LLM technology and software engineering research are evolving rapidly.The authors frame their categorization as applicable within the current research scope.
  • External Validity: Subtle boundaries between tasks such as code cloning and code smells create potential ambiguity in classification.The authors address this by providing explicit task definitions and clarifying classification scope.
  • External Validity: The literature review may contain bias in RQ2 because verifying the accuracy of every reviewed work is impractical.The authors provide each work’s dataset, evaluation emphasis, target, and results to mitigate this threat.

8 Conclusion and Future Work

The paper reviews 123 studies on LLM applications in software engineering, classifies them by task, and synthesizes evaluations to characterize capabilities, limitations, and future directions.

  • Conclusion and Future Work: The review collected and screened 123 works, categorized them by software engineering task, and identified research focuses and integration deficiencies.It also examined evaluation studies to summarize LLM capabilities and limitations.
  • Conclusion and Future Work: Evaluation synthesis provides directions for future research and optimization by exposing LLM capabilities and limitations across software engineering tasks.The review is intended to support researchers and developers working on LLM-based software engineering.
  • Conclusion and Future Work: Future work should examine why many proposed task-specific improvement methods lack detailed performance evaluations or discussions.The authors suggest this may reflect insufficient or unstable current performance, but present it as an issue for future investigation.
  • Conclusion and Future Work: Detailed evaluations of code-centric models such as Codex remain limited compared with evaluations based on general large models such as ChatGPT.The paper also raises whether task-specific fine-tuning is needed for tailored software engineering products.

9 Conflict of Interests

The authors state that the manuscript presents original research approved by all authors and declare no commercial or associative conflict of interest.

  • Conflict of Interests: The authors declared no conflict of interest and stated that all authors approved the manuscript for publication.They also stated that the work had not been previously published or submitted elsewhere.
  • Conflict of Interests: The authors declared no commercial or associative interest representing a conflict in connection with the submitted work.
Loading 2308.11396v3…