Large Language Models for Software Engineering: A Systematic Literature Review

• LLMs can address SE challenges reframed as data, code, or text analysis tasks; Codex solved 72.31% of complex Python programming challenges posed by humans [43].Source: §1 · p. 2
• Prior reviews were limited in scope or method, and a systematic review of LLM applications across SE was absent.Existing work focused on narrow SE tasks, ML/DL models, preliminary empirical exploration, or Code LLMs without fully considering general LLMs.Source: §1 · pp. 2–3
• The review analyzes 395 papers on LLM-based solutions to SE challenges, including publication trends and venues.Source: §1 · p. 3
• The review classifies LLMs used for SE and summarizes reported data collection, categorization, preprocessing, and representation stages.Source: §1 · p. 3
• The review discusses LLM4SE tuning techniques, prompt-engineering approaches, and commonly used evaluation metrics.Source: §1 · p. 3
• The review groups 85 specific SE tasks into six core activities and identifies challenges and potential research directions for LLM4SE.The activities are requirements engineering, software design, software development, software quality assurance, software maintenance, and software management.Source: §1 · p. 3

2 APPROACH

• The review follows Kitchenham et al., organizing work into planning, conducting the review, and analyzing basic results.Planning covers Sections 2.1–2.2, conducting the review covers Sections 2.3–2.4, and basic-results analysis is in Section 2.5.Source: §2 · pp. 3–4

2.1 Research Questions

• The four RQs map LLM architectures used in SE, dataset collection and preprocessing, optimization and evaluation methods, and SE tasks addressed.They examine models employed, how SE datasets are assembled and used, how LLM4SE is optimized and evaluated, and which SE tasks have been tackled.Source: §2.1 · p. 4

2.2 Search Strategy

• The QGS strategy combined manual identification of relevant studies, an extracted search string, automated searching, snowballing, and strict relevance filtering.Its five steps covered venue and database selection, QGS screening, search-string definition, automated search, and snowballing.Source: §2.2 · pp. 4–5
• Manual searching crawled 4,618 papers from six leading SE venues and identified 51 relevant studies as the QGS basis.The venues were ICSE, ESEC/FSE, ASE, ISSTA, TOSEM, and TSE.Source: §2.2.1 · p. 5

2.2.1 Search Items.

• Search items paired extensive SE-task terms with LLM terms, deliberately including broad ML, DL, and related keywords to reduce omissions.The SE terms covered tasks including code generation, testing, program repair, and requirements; the LLM-related terms included model families and techniques.Source: §2.2.1 · pp. 5–6
• From 2017 onward, two authors independently searched IEEE Xplore, ACM Digital Library, ScienceDirect, Web of Science, Springer, arXiv, and DBLP, then merged and deduplicated results.Source: §2.2.2 · p. 6

2.3.1 Study Inclusion and Exclusion Criteria.

• Screening reduced 218,765 papers to 1,172 after page-length, keyword, language, and venue filtering, then to 810 after deduplication and full-text exclusions.The keyword stage expanded beyond LLM terms to include ML and DL; the review retained 2023–2024 arXiv papers, excluded 190 grey, workshop, or doctoral-symposium publications, and identified 382 directly relevant papers.Source: §2.3.1 · pp. 6–7

2.3.2 Study Quality Assessment.

• Ten quality assessment criteria evaluated relevance, clarity, validity, and significance, with staged scoring and early exclusion for failures on any of the first three criteria.The first three criteria used scores of -1, 0, or 1; QAC4–QAC10 used scores of 0–3, with a maximum total of 21 for published papers.Source: §2.3.2 · p. 7
• The quality assessment retained 382 papers, using thresholds of 16.8 for published papers and 14.4 for unpublished arXiv papers.The published-paper threshold was 21 × 0.8; for arXiv papers, QAC4 scored 0 and the maximum QAC5–QAC10 score was 18, yielding a threshold of 18 × 0.8.Source: §2.3.2 · p. 8

2.4 Snowballing Search

• Snowballing identifies potentially relevant studies through papers’ reference lists (backward) and citations to those papers (forward).The authors note that it can also be complemented by systematically examining where papers are referenced and cited.Source: §2.4 · p. 8
• 13 additional papers were obtained after snowballing yielded 3,964 forward and 9,610 backward papers, initial deduplication left 5,152, and full selection was conducted.The 5,152 papers were also deduplicated against the 382 papers in the initial list.Source: §2.4 · p. 8
• The initial snowballing list comprised 382 papers remaining after quality assessment.Source: §2.4 · p. 8

2.5 Data Extraction and Analysis

• Of 395 included papers, 241 appeared on arXiv and 154 in peer-reviewed venues; ICSE led these venues with 41 papers, after quality assessment across all papers.Other notable peer-reviewed venues were TSE (14 papers), ESEC/FSE (12), and TOSEM (11).Source: §2.5 · p. 9
• Publications grew from 7 in 2020 and 13 in 2021 to 56 in 2022, 273 in 2023, and 46 in January 2024.The authors interpret this rapid increase as evidence of growing research interest in LLM4SE.Source: §2.5 · p. 9
• A word cloud of 395 abstracts foregrounded code/software, LLM/model, generation/program/task, and performance, reflecting code generation and evaluation among the reviewed SE themes.The authors describe the word cloud as visual evidence that the collected literature closely relates to their research topic.Source: §2.5 · p. 9
• Full-text extraction captured SE-task classifications and activities plus LLM categories, characteristics, and applicability, enabling systematic analysis for the review questions.The extracted data were gathered to support comprehensive responses to the RQs.Source: §2.5 · p. 9

3.1 Large Language Models (LLMs)

• Encoder-only models such as CodeBERT and GraphCodeBERT support code understanding, with uses including code review, bug-report understanding, and code-entity recognition.CodeBERT uses token prediction, while GraphCodeBERT models relationships among code elements as a graph.Source: §3.1 · p. 11
• Encoder-decoder models including PLBART, T5, and CodeT5 combine input representation and output generation, supporting SE tasks such as code summarization.The review describes this architecture as suited to summarization, translation, and question-answering.Source: §3.1 · p. 11
• Decoder-only models, including GPT-series and specialized coding models, suit code generation and completion and can perform downstream tasks from examples or instructions without fine-tuning.The review notes that GPT-4 and ChatGPT were integrated into SE tasks, while newer models such as Gemini, Llama 2, and Claude had not yet been widely applied.Source: §3.1 · pp. 11–12

3.2 Trend Analysis

• In 2023, decoder-only LLMs appeared in 195 papers (70.7%); in January 2024, they appeared in 77 papers (64.17%), remaining the most-used architecture.The 2023 count includes 432 instances of decoder-only model use across the 195 papers.Source: Figure 5 · p. 12, §3.2 · p. 13
• In 2021, decoder-only models featured in 9 of 19 papers (47.37%); in 2022, they featured in 73 of 142 papers (51.41%), as architecture use diversified.In 2020, eight papers predominantly focused on encoder-only models; in 2022, encoder-only models appeared in 52 papers (36.62%) and encoder-decoder models in 17 (11.97%).Source: Figure 5 · p. 12, §3.2 · pp. 12–13
• LLM selection for SE should consider code-context understanding, relevant generation, fine-tuning responsiveness, and SE-benchmark performance; task-specific fine-tuning can improve performance.The review cites fine-tuning on task-tailored datasets for bug detection and code review as examples.Source: §3.2 · p. 13

3.1 Large Language Models (LLMs)

• The review identifies more than 70 LLMs across encoder-only, encoder-decoder, and decoder-only architectures; over 45 are decoder-only models studied in 195 papers.Encoder-only models emphasize comprehensive understanding, encoder-decoder models pair input understanding with content generation, and decoder-only models suit generation tasks.Source: §3.1 · p. 10, §3.2 · p. 14

4.1 How are the datasets for training LLMs sourced?

• 235 studies used open-source datasets, but only six used industrial datasets, suggesting an academic–industry data mismatch; open-source data comprised 62.83% of 374 explicit reports.Industrial datasets can include proprietary data and sensitive information, making access difficult because of confidentiality and privacy constraints.Source: §4.1 · p. 15, §4.4 · p. 21

4.2 What types of SE datasets have been used in existing LLM4SE studies?

• Text-based datasets were most common (151); programming tasks/problems (42) and prompts (33) led within them, while source code was the most abundant code-based type (60).Combined datasets most often paired programming tasks with test suites or cases.Source: §4.2 · pp. 16–17

4.3 How do data types influence the selection of data-preprocessing techniques?

• Text preprocessing extracts, segments, filters, normalizes, deduplicates, tokenizes, and partitions data; code preprocessing adds compilation checks and token-, tree-, or graph-based representation.Code processing also extracts, filters, and deduplicates code; the order of preprocessing steps may vary with the research task and dataset.Source: §4.3 · pp. 18–19

4.4 What input formats are the datasets for LLM training converted to?

• Token-based inputs dominated: text in tokens appeared in 150 studies (42.25%) and code in tokens in 118 (33.24%).Tree/graph-based inputs appeared in seven studies (1.4%); pixel-based and hybrid-based inputs each appeared in one study (0.28%).Source: §4.4 · p. 20

5.1 What tuning techniques are used to enhance the performance of LLMs in SE tasks?

• Full fine-tuning adapted LLMs to downstream SE tasks in 83 studies, but its computational, data, and per-task deployment costs are high.General-purpose LLMs can also be applied directly without fine-tuning, while in-context learning avoids parameter updates but offers limited downstream improvement.Source: §5.1 · p. 21
• Parameter-efficient fine-tuning reduces computational complexity by updating a subset of parameters; eight studies used LoRA, three used prompt tuning, and two used prefix tuning.Other approaches include adapter tuning, reinforcement learning, supervised fine-tuning, syntax fine-tuning, knowledge-preservation fine-tuning, and task-oriented fine-tuning.Source: §5.1 · pp. 21–22

5.2 What prompt engineering techniques are applied to improve the performance of LLMs in SE tasks?

• Few-shot prompting appeared in 88 studies, zero-shot prompting in 79, and chain-of-thought prompting in 18; one in-context approach outperformed supervised methods for comment generation [104].Zero-shot methods rely on inference-time prompts without task-specific training, while CoT guides reasoning through linked steps.Source: §5.2 · pp. 22–23
• Other techniques include APE, CoC, Auto-CoT, MoT, and SCoT, while 76 studies used other carefully designed prompts or prompt-based strategies.PromptCS generates continuous prompts; Automatic.ART selects reasoning steps and tools; MoT decomposes tasks into subtasks; and SCoT uses program structure to guide reasoning and improve code-generation performance.Source: §5.2 · pp. 23–24

5.3 How are evaluation metrics utilized to assess the performance of LLM4SE tasks?

• Evaluation uses metrics tailored to regression, classification, recommendation, and generation; classification most often uses Precision (35 studies), Recall (34), and F1-score (33).MAE is used for regression; the review reports 19 different generation metrics and nine classification metrics.Source: §5.3 · pp. 24–25
• For recommendation, MRR appeared in 15 studies, while Precision@k and F1-score@k appeared in six each; generation most often used BLEU variants (62 studies) and Pass@k (54).Generation studies also used ROUGE/ROUGE-L, METEOR, Exact Match, and Edit Similarity.Source: §5.3 · pp. 24–25

6.1 What are the distributions SE activities and problem types addressed to date with LLM4SE?

• Software development accounts for 56.65% of studies, followed by maintenance (22.71%), quality assurance (15.14%), requirements engineering (3.9%), software design (0.92%), and management (0.69%).The review describes this distribution as a concentration on development and maintenance, with comparatively limited research in design and management.Source: §6.1 · p. 26
• Generation tasks comprise 70.97% of studies, classification 21.61%, recommendation 6.77%, and regression 0.65%.The review characterizes generation and classification as the most emphasized problem types.Source: §6.1 · pp. 26–27

6.2 How are LLMs used in requirements engineering?

• In requirements engineering, LLMs address anaphoric ambiguity, requirements classification, term identification, coreference detection, and traceability, alongside elicitation and specification tasks.ChatGPT accurately identified antecedents in analysis of ten requirement specifications; BERT transfer learning classified functional and non-functional requirements, and T-BERT transferred code-search knowledge to traceability with limited training data [246].Source: §6.2 · p. 28

6.3 How are LLMs used in software design?

• Software design applications include BERT-based GUI retrieval, prompt-pattern support for rapid prototyping, and SpecSyn for synthesizing specifications from natural language.SpecSyn exceeded the previous state-of-the-art tool by 21% in F1 score and handles specifications expressed in single or multiple sentences.Source: §6.3 · p. 29

6.4 How are LLMs used in software development?

• Code generation dominates software-development studies, with GPT-series models—particularly GPT-4—prominent; sketch-first prompting and tool-integrated evaluation are reported strategies for improving generated code.The review also notes differing performance between method- and class-level generation and identifies EvalPlus as an evaluation framework.Source: §6.4 · pp. 29–30
• LLMs support software-development assistance through code completion, summarization, search, understanding, and program synthesis, using code context and natural-language/code semantic representations.Code completion models suggest context-appropriate snippets; summarization can support documentation, maintenance, search, and classification, while code-search studies indicate that pretraining alone may be insufficient without multimodal understanding.Source: §6.4 · pp. 30–31
• Other development applications include API recommendation and synthesis, code representation, code-comment generation, agile story-point estimation, API documentation-smell detection, and extraction of API entities and semantic relationships.AERJE achieved F1 scores of 96.51% for API entity extraction and 81.2% for API relationship extraction. GPT2SP was comparable to Deep-SE in within-repository evaluations and surpassed it in 62.5% of cases, with improvements of 3% to 46% across projects.Source: §6.4 · pp. 32–33

6.5 How are LLMs used in software quality assurance?

• Quality-assurance applications include vulnerability detection, test generation, bug localization, verification, and test automation.Source: §6.5 · p. 35
• Quality-assurance applications include vulnerability detection, test generation, formal verification, mutation testing, and fuzzing.Reported approaches include combining LLMs with bounded model checking for software repair and generating diverse inputs to identify bugs and vulnerabilities.Source: §6.5 · pp. 35–36
• Difference testing with ChatGPT increases the probability of finding correct fault-inducing test cases, while augmented bug reports improve BERT-based bug localization.The approach synthesizes programs that amplify subtle code differences; LLMs are also used for defect detection, GUI testing, static analysis, and related tasks.Source: §6.5 · pp. 35–36

6.6 How are LLMs used in software maintenance?

• ChatGPT stands out in program repair, where interactive feedback can refine contextually apt patches and domain-specific integration can further improve repair performance.LLMs generate syntactically correct, contextually relevant patches and can be fine-tuned on repair data; comparisons across baselines highlight ChatGPT’s adaptability.Source: §6.6 · pp. 36–37
• LLMs support code review, clone detection, debugging, bug reproduction, and duplicate bug-report detection in software maintenance.Examples include models that generate review suggestions, form debugging hypotheses, or extract reproduction steps from bug reports using GUI state.Source: §6.6 · pp. 37–38
• Maintenance applications also include log generation and summarization, bug triage, merge-conflict repair, tag recommendation, and traceability recovery.The review describes LANCE for injecting Java log statements and reports ChatGPT log summaries outperforming the state of the art; GPT-3 showed promise on semantic merge conflicts but did not replace domain-specific languages for some synthesis patterns.Source: §6.6 · pp. 38–40
• 6.5% to 35.6%: Transformer models outperformed existing SA4SE tools in macro/micro-averaged F1-scores across six datasets, with a runtime-efficiency trade-off.BERT-based feature extraction also outperformed TF-IDF for long-lived bug prediction. LLMs fixed 100% of synthetic and hand-created vulnerability scenarios, but real-life examples exposed functional-correctness challenges.Source: §6.6 · p. 39

6.7 How are LLMs used in software management?

• Software management is sparsely studied: the review reports three studies in this activity, including an evaluation of BERT for maintenance effort estimation.Across the review, the authors identify 85 SE tasks in six activities; software development had 229 papers mentioning more than 24 tasks.Source: §6.7 · p. 40

Paper search omission

• Combining manual searches of six SE venues, automated searches across seven publisher platforms, and forward and backward snowballing was intended to reduce paper omissions.The authors note that incomplete keyword coverage could still cause relevant papers to be missed.Source: §7 · p. 40

Study selection bias

• Automated screening may mislabel papers and manual review may introduce judgment bias; uncertain cases were retained, two experienced reviewers conducted a secondary review, and a replication package was provided.The selection process also used inclusion and exclusion criteria and quality assessment criteria.Source: §7 · pp. 40–41

Empirical knowledge bias

• Subjective judgments and experiential knowledge may bias analysis of 395 studies; the authors drew on Yang et al.’s survey and prior work [370] [472] [558] to shape RQs and analysis categories.They predefined approximate categories for each RQ and used prior work to distinguish LLM tuning techniques, including prompt engineering and PEFT, from traditional machine-learning techniques.Source: §7 · p. 41

8.1.1 Challenges in LLM Applicability.

• LLM4SE faces costly deployment, data scarcity and bias, benchmark contamination, privacy risks, and ambiguous code-generation inputs that can produce functionally incorrect code.Training and deploying large models demands substantial storage, compute, and energy; compressing CodeBERT to 3 MB reduced response latency by more than 70%. The review also calls for more context, domain knowledge, or model ensembles to handle ambiguity.Source: §8.1.1 · pp. 41–42

8.1.2 Challenges in LLM Generalizability.

• LLMs may fail to generalize across SE projects, languages, and domains, and CodeBERT performance declines after semantic-preserving variable-name substitutions [510].The review recommends careful fine-tuning, validation across datasets, and continuous feedback to reduce over-adaptation to training data.Source: §8.1.2 · p. 42

8.1.3 Challenges in LLM Evaluation.

• Standard metrics such as Accuracy, Recall, and F1-score can miss task-specific failures and qualitative traits; interpretability, trust, training-data ownership, and model security also remain concerns.Customized metrics in recent studies [141] [398] [495] [521] [540] illustrate the limits of widely used evaluation measures. Low-quality training data can produce insecure code [228] [511], while adversarial attacks and opaque training data further complicate trust.Source: §8.1.3 · pp. 42–43, §8.1.4 · p. 43

8.2.1 Optimization of LLM4SE.

• Code-specialized LLMs and ChatGPT are shaping LLM4SE, while task-specific training trades quick, low-overhead adaptation for deeper customization at greater computational cost.Collaborative LLMs, or combinations of LLMs with specialized machine-learning models, are proposed to improve precision and efficiency across SE tasks.Source: §8.2.1 · pp. 43–44

8.2.2 Expanding LLM’s NLP Capabilities in More SE Phases.

• Future work can expand LLM inputs to speech, diagrams, multimodal and graph-based data, and apply models beyond development and maintenance to requirements, design, and management.These input forms may enrich context and represent code structure and dependencies; the latter SE phases remain relatively underexplored compared with code completion and bug detection.Source: §8.2.2 · p. 44

8.2.3 Enhancing LLM’s Performance in Existing SE Tasks.

• For specialized SE domains, researchers should curate datasets with domain experts, fine-tune models, and establish adaptable evaluation frameworks with academic and industry input.Such frameworks could assess practical readiness, productivity, code quality, collaboration, bias, semantic misinterpretation, and context-specific limitations.Source: §8.2.3 · pp. 44–45

8.3 Roadmap

• The paper presents a roadmap for leveraging large language models in software engineering, while also recognizing a reciprocal relationship and emerging exploration of software engineering for large language models.Source: §8.3 · p. 45

9 CONCLUSION

• The review examines the LLMs used for SE tasks and characterizes their distinct features and applications.This addresses RQ1.Source: §9 · p. 47
• It investigates data collection, preprocessing, and usage, emphasizing the role of well-curated datasets in successfully applying LLMs to SE tasks.This addresses RQ2.Source: §9 · p. 47
• It reviews strategies for optimizing and assessing LLM performance on SE tasks.This addresses RQ3.Source: §9 · p. 47
• It surveys the SE tasks where LLMs have been applied and the practical contributions they have made.This addresses RQ4.Source: §9 · p. 47
• The authors summarize key LLM4SE challenges and provide a research roadmap outlining promising future directions.Source: §9 · p. 47

A DATA TYPES

• Combined datasets most often paired programming tasks with test suites or cases (17 studies), source code with comments (12), and programming tasks with solutions (8).These are the largest reported combined-dataset counts in the supplied passages.Source: Table 13 · p. 72, A DATA TYPES · p. 73
• Software repository-based datasets included code repositories (9 studies), Android apps (3), issues and commits (3), and pull requests (2).Source: Table 13 · p. 72, A DATA TYPES · p. 73
• Code-based datasets covered identifiers, labeled clone pairs, and packages, while graph-based datasets included GUI images; each type had 1 study.Source: A DATA TYPES · p. 73
• Combined datasets also paired code with descriptions, API-usage sequences, tests, bug reports, comments, solutions, summaries, annotations, error messages, Q&A pairs, logs, and vulnerability descriptions.Source: A DATA TYPES · p. 73

B INPUT FORMS

• Token-based inputs include text, code, and combined code-and-text, with 118 studies using code in tokens and 78 using code and text in tokens.The review also lists studies using text in tokens, but the supplied passage does not give a count for that form.Source: Table 14 · p. 73, B INPUT FORMS · p. 74
• Tree/graph-based inputs comprise code represented in tree structures in 2 studies and graph structures in 3 studies.Source: Table 14 · p. 73, B INPUT FORMS · p. 74
• Pixel-based input is listed in 1 study, while hybrid input forms are listed in 2 studies.Source: Table 14 · p. 73, B INPUT FORMS · p. 74

C PROMPT ENGINEERING

• Eight prompt-engineering techniques are mentioned: few-shot, zero-shot, CoT, APE, CoC, Auto-CoT, MoT, and SCoT prompting.Table 15 presents these techniques for software-engineering tasks.Source: C PROMPT ENGINEERING · p. 74, Table 15 · p. 74

D EVALUATION METRICS

• LLM-for-SE tasks are grouped as regression, classification, recommendation, and generation, with commonly used evaluation metrics summarized in Table 16.The supplied table captions identify the metrics table and its continuations, but do not expose the metric values or task-specific entries.Source: D EVALUATION METRICS · p. 75, Table 16 · pp. 75–77

E SE TASKS

• The six categories are requirements engineering, software design, software development, software quality assurance, software maintenance, and software management.The review organizes them according to the software development lifecycle.Source: E SE TASKS · p. 77
• Requirements engineering, software design, and software development form three of the review’s six SE-task categories.These categories are listed in the review’s software-development-lifecycle grouping.Source: E SE TASKS · p. 77
• Software quality assurance, software maintenance, and software management complete the six SE-task categories.Table 17 presents papers applying LLMs to the categorized tasks.Source: E SE TASKS · p. 77, Table 17 · pp. 77–79