PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models for Software Engineering: A Systematic Literature Review

Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, Haoyu Wang

arXiv:2308.10620v6cs.SEcs.AI

TL;DR

Existing reviews provide only narrow or preliminary coverage of LLM applications in software engineering, leaving the broader relationship between LLMs and SE insufficiently understood. This systematic literature review examines LLMs, data practices, optimization and evaluation strategies, and applications across SE tasks, highlighting dataset quality and generalization challenges.

  • Problem

    Existing reviews narrowly target individual SE scopes or earlier ML/DL models, leaving the broader relationship between LLMs and software engineering insufficiently understood.

  • Method

    The study conducts a systematic literature review using searches across six top SE venues, automated scanning, manual verification, and quality assessment.

  • Results

    The review characterizes LLMs, data practices, optimization and evaluation strategies, and applications spanning 85 SE tasks.

  • Takeaways & Limitations

    Well-curated datasets are significant to successful LLM applications in SE, while generalization remains challenging across projects, languages, domains, and datasets.

  • Takeaways & Limitations

    The review may omit relevant papers despite combining manual search, automated search, and snowballing techniques.

Abstract

from arXiv · show

Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a systematic literature review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We select and analyze 395 research papers from January 2017 to January 2024 to answer four key research questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, preprocessing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and flagging promising areas for future study. Our artifacts are publicly available at https://github.com/xinyi-hou/LLM4SE_SLR.

1 INTRODUCTION

LLMs are increasingly applied to Software Engineering because many SE challenges can be reframed as data, code, or text analysis tasks. This review addresses limitations in prior surveys by systematically mapping LLM4SE research and its applications.

  • LLMs in Software Engineering: LLMs have expanded from traditional language modeling into applications across Software Engineering tasks.SE challenges can often be reframed as data, code, or text analysis tasks.
  • Research gap: Prior reviews often focused on a single SE scope, ML/DL models, or preliminary performance analyses, leaving newer LLM applications insufficiently covered.Examples include software testing, natural-language-to-code, and applications such as ChatGPT.
  • Review contributions: The review summarizes challenges in applying LLMs to SE and proposes potential directions for future LLM4SE research.Its stated aim is to provide researchers and practitioners with a guide to the convergence of LLMs and SE.
  • Review scope: The review analyzes 395 papers published between January 2017 and January 2024 on LLM-based solutions for SE challenges.It examines publication trends and the distribution of included research.
  • Review contributions: The review classifies LLMs, describes data-processing stages, examines optimization and evaluation techniques, and maps 85 SE tasks across six core activities.The six activities are requirements engineering, software design, software development, software quality assurance, software maintenance, and software management.

2 APPROACH

The review follows a Kitchenham-based systematic literature review process, combining manual, automated, and snowballing searches with predefined criteria. Its questions cover LLMs, datasets, optimization and evaluation, and addressed SE tasks.

  • SLR methodology: The SLR methodology consists of planning the review, conducting the review, and analyzing basic review results.These stages correspond to Sections 2.1–2.5.
  • Research questions: The four research questions examine employed LLMs, dataset collection and preprocessing, optimization and evaluation, and SE tasks addressed with LLM4SE.The questions respectively define the review’s scope across models, data, techniques, and applications.
  • Search strategy: The search strategy used a Quasi-Gold Standard, automated database searches, manual searches, and snowballing to improve coverage.The QGS search string combined SE-task keywords with LLM-related keywords.
  • Search items: Manual searching crawled 4,618 papers from six top SE venues and identified 51 relevant papers for constructing the QGS.The selected venues included ICSE, ESEC/FSE, ASE, ISSTA, TOSEM, and TSE.
  • Search items: The automated search covered seven databases and focused on publications from 2017 onward because the Transformer architecture underlying LLMs was published in 2017.Two authors independently performed the automated search before merging and deduplicating results.

2.3 Study Selection

Study selection progressively filtered the search results using page-length, relevance, publication-type, and quality criteria. The final review set contained 382 papers before snowballing added 13 more.

  • Study selection: The initial search produced 218,765 potentially relevant papers before inclusion and exclusion screening.The selection process then applied predefined criteria to identify papers addressing the research questions.
  • Study selection: Automated filtering removed papers shorter than eight pages, reducing the set to 80,611 papers.Later screening examined titles, abstracts, and keywords for LLM- and SE-related content.
  • Study selection: Full-text screening excluded 190 grey, workshop, or doctoral-symposium publications and identified 382 papers directly relevant to the review.The process also excluded papers that mentioned LLMs without directly applying them.
  • Quality assessment: The final quality thresholds retained papers scoring at least 16.8 for published studies and 14.4 for unpublished arXiv papers.After quality assessment, the final set contained 382 papers.

2.4 Snowballing Search

Snowballing supplemented the screened study set, while the review then summarized publication distributions, topics, extracted data, and LLM usage. The resulting corpus comprised 395 relevant research papers with rapid growth after 2020.

  • Snowballing search: Forward and backward snowballing collected 3,964 and 9,610 papers, respectively, before deduplication and full study selection.The process began from 382 quality-assessed papers and left 5,152 papers after initial deduplication.
  • Snowballing search: Snowballing added 13 papers, producing a final corpus of 395 relevant research papers.The additional papers were obtained after applying the full study-selection process to snowballing results.
  • Publication distribution: The corpus included 154 papers in peer-reviewed venues and 241 papers on arXiv.ICSE contributed 41 peer-reviewed papers, followed by TSE with 14, ESEC/FSE with 12, and TOSEM with 11.
  • Publication distribution: 273 relevant papers were published in 2023, compared with 56 in 2022 and 46 during January 2024.The publication count increased rapidly after 2020, when seven papers were identified, and 2021, when 13 were identified.
  • Topics and LLM distribution: The collected-paper abstracts most frequently emphasized code, LLMs, language, models, tasks, generation, performance, software, and programs.The word-cloud analysis reflected the review’s focus on LLM applications in software engineering.
  • Data extraction and analysis: Data extraction recorded SE tasks, activities, LLM categories, model characteristics, and applicability to support systematic analysis of LLM4SE.These extracted items were used to address the review’s research questions.

3 RQ1: WHAT LLMS HAVE BEEN EMPLOYED TO DATE TO SOLVE SE TASKS?

The review identifies more than 70 LLMs used for SE tasks and groups them into decoder-only, encoder-decoder, and encoder-only architectures. Usage shifted over time toward decoder-only models, while different architectures served distinct SE purposes.

  • Encoder-only models use bidirectional context for comprehensive understanding, whereas specialized models incorporate program structure, new pre-training tasks, or additional modalities.
  • GPT-series and related decoder-only models include specialized systems such as CodeGPT, Codex, and Copilot, alongside open-source models including GPT-J, GPT-Neo, GPT-NeoX, LLaMA, and Vicuna.
  • Trend Analysis: From 2020 to 2024, decoder-only, encoder-decoder, and encoder-only architectures underwent notable shifts in preference and application across SE tasks.
  • Trend Analysis: 51.41% of 2022 studies centered on decoder-only LLMs, compared with 36.62% for encoder-only and 11.97% for encoder-decoder models.
  • Trend Analysis: The review concludes that LLM4SE transitioned from encoder-only toward decoder-only architectures, reflecting changing approaches to SE tasks.
  • More than 70 LLMs used for SE tasks are classified as decoder-only, encoder-decoder, or encoder-only models.

4 RQ2: HOW ARE SE-RELATED DATASETS COLLECTED, PREPROCESSED, AND USED IN LLMS?

LLM4SE datasets come from open-source, collected, constructed, or industrial sources and undergo extraction, filtering, deduplication, segmentation, and formatting. The literature strongly favors text, code, and token-based inputs, while alternative representations remain uncommon.

  • Dataset Sources: Dataset sources are categorized as open-source, collected, constructed, and industrial, with source choice governed by project requirements and constraints.
  • Dataset Sources: Industrial datasets can support real-world business scenarios but often involve business confidentiality, privacy concerns, non-disclosure agreements, and legal safeguards.
  • Dataset Sources: Open-source datasets were used by 235 studies, making them the dominant collection strategy.
  • Data Types: Programming tasks/problems were the most prevalent training-data type, with 42 instances, while source code was the most abundant code-based type, with 60 instances.
  • Input Formats: Token-based inputs accounted for approximately 97.75% of studies, whereas tree/graph-based inputs appeared in seven studies, or approximately 1.4%.
  • Input Formats: Pixel-based and hybrid-based inputs each appeared in one study, contributing approximately 0.28% each, while combined datasets often paired programming tasks with test suites or cases.
  • Input Formats: The review finds a strong preference for token-based inputs, while tree/graph, pixel-based, and hybrid representations remain areas of ongoing exploration.
  • Preprocessing: Common preprocessing includes data extraction, unqualified-data deletion, duplicate-instance deletion, and segmentation before data is formatted for LLM inputs.

5 RQ3: WHAT TECHNIQUES ARE USED TO OPTIMIZE AND EVALUATE LLM4SE?

LLM4SE optimization combines task-specific tuning and prompt engineering, while evaluation uses metrics matched to regression, classification, recommendation, and generation tasks. Parameter-efficient methods and diverse prompting strategies are increasingly used to adapt models without full parameter updates.

  • Optimization: General-purpose LLMs can perform code generation, summarization, and program repair without fine-tuning, but task-specific tuning is often used to exploit their potential more fully.
  • Optimization: Eight studies used LoRA to enhance LLM performance in SE tasks, including adapting StarCoder for multilingual-to-Python code translation.
  • Prompt Engineering: Prompt engineering improves model performance through task-specific instructions without modifying core model parameters; 88 studies used few-shot prompting.
  • Prompt Engineering: The surveyed prompting techniques include few-shot, zero-shot, Chain-of-Thought, and Modular-of-Thought prompting, alongside additional prompt-design strategies.
  • Evaluation: Evaluation is organized around regression, classification, recommendation, and generation tasks, with metrics selected according to problem type.
  • Evaluation: 35 studies used Precision, 34 used Recall, and 33 used F1-score for classification tasks.
  • Evaluation: MRR was the most frequent recommendation metric, appearing in 15 studies.
  • Optimization: PEFT methods such as LoRA, prompt tuning, prefix tuning, and adapter tuning are gaining prominence while minimizing computational complexity.

6 RQ4: WHAT SE TASKS HAVE BEEN EFFECTIVELY ADDRESSED TO DATE USING LLM4SE?

LLMs have been applied across a broad range of software engineering tasks, with research concentrated in software development and generation. Applications span requirements engineering, design, development, quality assurance, and maintenance, including code generation, testing, vulnerability detection, program repair, and debugging.

  • 56.65% of studies address software development, followed by software maintenance at 22.71% and software quality assurance at 15.14%.
  • 70.97% of studies focus on generation tasks, while 21.61% address classification and 6.77% address recommendation tasks.
  • Requirements engineering: In requirements engineering, BERT, SpanBERT, and ChatGPT have been used to address anaphoric ambiguity, with ChatGPT accurately identifying antecedents across ten requirement specifications.
  • Software development: Software development applications include code generation, completion, summarization, comment generation, and method name generation, with GPT-4 prominent in code-generation studies.
  • Software development: LLM-supported code completion improves reported efficiency and accuracy, while programming-thinking techniques that sketch code before implementation improve synthesized-code accuracy and robustness.
  • Quality assurance and maintenance: Quality assurance and maintenance applications include vulnerability detection, test automation, fault localization, program repair, code review, code clone detection, and debugging.

7 THREATS TO VALIDITY

The review identifies paper-search omission and study-selection bias as threats, mitigating them through combined search strategies, screening criteria, manual verification, and secondary review.

  • Paper search omission: Paper-search omission may arise from incomplete keyword coverage when collecting LLM4SE studies across publishers.The review combines manual, automated, forward-snowballing, and backward-snowballing searches to reduce this risk.
  • Study selection bias: Automated selection can mislabel papers because BibTeX records may contain incomplete or ambiguous information.Papers that cannot be confidently excluded are retained for manual verification.
  • Study selection bias: Manual verification may introduce researcher subjectivity and bias into paper quality assessment.Two experienced SE and LLM reviewers performed a secondary review to improve selection accuracy.

8 CHALLENGES AND OPPORTUNITIES

LLM4SE faces challenges in deployment, data dependency, generalization, evaluation, interpretability, and trustworthiness, while opportunities include specialized models, richer inputs, broader SE coverage, domain datasets, and automation.

  • Challenges in LLM Applicability: LLM scale creates storage, memory, computational, and deployment barriers, especially in resource-limited or real-time scenarios.A 176B-parameter BLOOM training run is estimated at 1,082,880 GPU hours.
  • Challenges in LLM Applicability: LLMs rely heavily on large, high-quality, diverse datasets, while limited or biased data can reduce generalizability and produce biased or inaccurate predictions.Domain-specific data for fine-tuning is also a bottleneck.
  • Challenges in LLM Applicability: LLM performance may not generalize across projects, languages, domains, or semantically transformed inputs without careful fine-tuning and validation.The review reports significant performance decreases for CodeBERT after semantic-preserving transformations.
  • Challenges in LLM Evaluation: Accuracy, recall, and F1-score may miss qualitative properties such as interpretability, robustness, and sensitivity to specific errors.Recent LLM4SE studies therefore customize evaluation metrics for particular tasks.
  • Challenges in LLM Evaluation: Black-box behavior and uncertain training-data quality can make LLM-generated code difficult to explain and trust, including when low-quality data produces vulnerabilities.The review identifies interpretability and trustworthiness as important adoption concerns.
  • Opportunities: Code-specialized LLMs such as GitHub Copilot, CodeWhisperer, Code Interpreter, and Code Llama create new avenues for code understanding, generation, and efficiency.These models are fine-tuned on code-specific datasets.
  • Opportunities: Task-specific training offers a choice between commercially available models and open-source frameworks with different levels of control and customization.The review contrasts GPT-4 with Llama 2, Gemma, and Mistral.
  • Opportunities: Scarce graph-based datasets and predominant code- and text-based inputs leave room for spoken, diagrammatic, and multimodal inputs.New input forms could broaden how LLMs process diverse user requirements.

9 CONCLUSION

This SLR examines LLM use in software engineering across four dimensions: models, data, optimization and evaluation, and SE tasks. It also identifies challenges and proposes a research roadmap.

  • The review analyzes LLMs used in SE tasks, characterizing their features and applications.
  • It investigates data collection, preprocessing, and usage, emphasizing well-curated datasets for successful LLM4SE applications.
  • It examines strategies for optimizing and assessing LLM performance on SE tasks.
  • It reviews the SE tasks where LLMs have been applied and their practical contributions.
  • The authors summarize existing LLM4SE challenges and outline promising directions in a research roadmap.

A DATA TYPES

The review classifies datasets used in LLM4SE into five data-type categories, covering code, text, graphs, repositories, and combinations of these types.

  • Datasets in prior LLM4SE studies are classified into five categories: code-based, text-based, graph-based, software repository-based, and combined.

B INPUT FORMS

The review identifies four input formats used to represent data for LLMs in LLM4SE research: token-, tree or graph-, pixel-, and hybrid-based inputs.

  • LLM4SE studies use four input formats: token-based, tree or graph-based, pixel-based, and hybrid-based input.

C PROMPT ENGINEERING

The review identifies eight prompt engineering techniques used across 395 LLM4SE studies, spanning few-shot, zero-shot, chain-of-thought, and automated or structured variants.

  • Eight prompt engineering techniques appear in 395 LLM4SE studies, including few-shot, zero-shot, CoT, APE, CoC, Auto-CoT, MoT, and SCoT prompting.

D EVALUATION METRICS

LLM-supported Software Engineering tasks are grouped into four task types, each associated with commonly used evaluation metrics presented in Table 16.

  • The review categorizes LLM4SE tasks into regression, classification, recommendation, and generation.
  • Each task type has commonly used evaluation metrics.These metrics are presented in Table 16 and its continuation tables.
  • Table 16 spans three table segments covering the evaluation metrics.

E SE TASKS

The review organizes Software Engineering tasks from 395 studies according to six software development lifecycle activities and lists the relevant papers in Table 17.

  • The 395 studies are organized into Requirements engineering, Software design, Software development, Software quality assurance, Software maintenance, and Software management.
  • Table 17 presents the papers applying LLMs to the six categorized Software Engineering activities.
  • Table 17 spans multiple table segments covering the distribution of Software Engineering tasks.
Loading 2308.10620v6…