A Survey on Large Language Models for Software Engineering

• The survey addresses fragmented evidence caused by diverse SE tasks, model architectures, parameters, bug types, and utilization paradigms.Source: §1 · pp. 1–2
• The study analyzes 1009 relevant SE studies and summarizes 62 representative code LLMs by architecture, pre-training objectives, downstream tasks, and open science.Source: §1 · p. 2
• LLM-based SE research spans 947 studies covering 112 code-related tasks across five phases: requirements and design, development, testing, maintenance, and management.Source: §1 · p. 2
• The survey also discusses evaluation, benchmarking, security, reliability, domain tuning, compression, distillation, and future challenges for applying LLMs in SE.Source: §1 · p. 2

Comparison with Existing Surveys

• Unlike earlier surveys focused on ML/DL in SE or narrow NL2Code tasks, this work covers the full SE workflow and corresponding LLM integration studies.Source: §1 · p. 2

2.1 Research Questions

• The survey analyzes LLMs, LLM-enabled SE tasks, and integration factors through three research questions spanning architectures, training, applications, evaluation, security, tuning, and compression.Source: §2.1 · pp. 2–3

2.2 Search Strategy

• The search combines SE- and LLM-related keywords derived from existing surveys and manually collected papers, then queries Google Scholar, ACM Digital Library, and IEEE Explorer through August 2024.Source: §2.2 · p. 3

2.3 Study Selection

• 32,560 papers were retrieved before filtering by publication year, length, duplication, and manual relevance assessment by two authors with third-author adjudication.Source: §2.2 · p. 3, §2.3 · p. 3
• Snowballing added three code-LLM papers and 36 LLM-based SE papers, producing the final corpus used by the survey.Source: §2.3 · pp. 3–4

2.4 Trend Observation

• 1009 papers were collected overall, with studies increasing rapidly from 2020 to 2024 as LLMs gained recognition and open-source code enabled training.Source: Figure 1 · p. 3, §2.4 · p. 4
• 382 of 1009 papers were not peer-reviewed, while the top five venues were ICSE, FSE, ACL, ASE, and ISSTA and 25 of the top 30 venues were conferences.Source: §2.4 · p. 4

3.1 RQ1.1: What LLMs have been released to support SE?

• The surveyed models generally derive from foundational NLP architectures and use code-aware objectives, while decoder-only architectures and model sizes continue to grow.Source: §3.1 · pp. 4–5

3.1.1 Encoder-only LLMs

• Encoder-only models use masked language modeling to produce fixed-size representations suited to contextual understanding tasks such as code search and vulnerability detection.Source: §3.1.1 · p. 5

3.1.2 Encoder-decoder LLMs

• Encoder-decoder models transform input sequences into output sequences and are particularly suited to tasks such as code summarization and program repair.Source: §3.1.2 · pp. 5–6
• T5 catalyzed code-focused encoder-decoder work, including PYMT5, T5-learning, PLABRT, CodeT5, CoditT5, AlphaCode, and JuPyT5.Source: §3.1.2 · p. 6

3.1.3 Decoder-only LLMs

• Decoder-only models generate text autoregressively from context, with code-focused examples including CodeGen, InCoder, StarCoder, CodeLlama, and related models.Source: §3.1.3 · pp. 6–7

3.1 RQ1.1: What LLMs have been released to support SE?

• Existing code LLMs mainly follow encoder-only, encoder-decoder, or decoder-only architectures, and no single architecture is best for every task.Source: §3.1 · p. 5, §3.1.3 · p. 7

3.2 RQ1.2: How are LLMs used in pre-training tasks?

• Pre-training tasks are organized into code sequence modeling, bidirectional code–natural-language generation, code structure and relationship understanding, and cross-modal representation learning.These categories span predicting code fragments, translating between programming and natural languages, modeling code structure, and learning multimodal representations.Source: §3.2 · p. 7, §3.2.2 · p. 8, §3.2.3 · p. 8

3.2.1 Code Sequence Modeling and Prediction

• CLM, MLM, and MSP train decoder-only, encoder-only, and encoder-decoder models by predicting next tokens, masked tokens, or masked spans.These objectives respectively support autoregressive completion, bidirectional contextual understanding, and sequence-to-sequence reconstruction.Source: §3.2.1 · p. 7

3.2.2 Bidirectional Understanding and Generation between Code and Natural Language

• Code–natural-language objectives use bidirectional generation and method-name prediction to align programming-language code with natural-language descriptions and intent.BDG optimizes NL→PL and PL→NL directions simultaneously, while MNG uses code, AST, and natural-language inputs to improve intent understanding.Source: §3.2.2 · p. 8

3.2.3 Code Structure and Relationship Understanding

• Structure-aware objectives model syntax, data flow, token–node alignment, and AST information to improve representations of code relationships and semantics.Examples include identifier tagging, edge prediction, node alignment, and code-AST prediction.Source: §3.2.3 · pp. 8–9

3.2.4 Cross-modal Representation Learning

• Cross-modal objectives learn code representations through contrastive alignment and comment generation between source code and other modalities.MCL distinguishes positive and negative representations, while CMG conditions comment generation on code and randomly swaps source and target inputs with 50% probability.Source: §3.2.4 · p. 9
• Overall, pre-training shifts from NLP-derived language modeling toward objectives that encode code variables, structural features, and code–natural-language relationships.The progression targets deeper understanding of code semantics and functional aspects rather than treating code only as token sequences.Source: §3.2.4 · p. 9

3.3 RQ1.3: How are LLMs used in downstream tasks?

• Code-code tasks include translation, program repair, cloze testing, mutant generation, and assert generation.These tasks transform or augment code while preserving functionality, refining errors, or supporting testing.Source: §3.3 · p. 9
• Text-code and code-text tasks generate or retrieve code from natural-language descriptions and produce natural-language summaries of code.Examples include code generation, code search, and code summarization.Source: §3.3 · p. 9
• Code-label tasks classify code for clone detection and defect detection, including whether snippets are similar or potentially vulnerable.These tasks apply classification to functional similarity and security-relevant defects.Source: §3.3 · p. 9
• Downstream tasks are classified by input–output type into code-code, code-text, text-code, and code-label tasks, or by task type into code understanding and code generation.The survey summarizes 15 representative downstream tasks evaluated in existing LLM papers.Source: §3.3 · p. 9
• Most downstream tasks use code or related natural-language comments, support automatic metrics and large-scale benchmarks, and can reduce developer effort through IDE integration.The survey reports preliminarily promising results and potential for broader SE applications.Source: §3.3 · p. 10

3.4 RQ1.4: How are LLMs open-sourced to support the open science community?

• LLM-based SE research creates reproducibility challenges through high computational costs, extensive data collection, and complex hyperparameter tuning.These challenges motivate examining whether models, datasets, and scripts are publicly available.Source: §3.4 · p. 10
• 54 of 62 investigated LLMs provide corresponding open-source repositories.The survey checks links to publicly available models or data in papers' main text or footnotes and records source-code, dataset, and trained-model availability.Source: §3.4 · p. 10
• Artifacts are commonly hosted on GitHub for code and Hugging Face for checkpoints, but some studies omit source code, datasets, or trained models, possibly for commercial reasons.The survey identifies incomplete artifact release as an ongoing issue.Source: §3.4 · p. 10
• High-quality artifacts are especially important for reproducing LLM studies because training requires expensive hardware, complex environments, controlled hyperparameters, random seeds, and quality datasets.The survey recommends open-source code and detailed reproduction instructions.Source: §3.4 · p. 10

4 RQ2: SE PERSPECTIVE

• The SE taxonomy covers requirements and design, development, testing, maintenance, and management, encompassing 947 LLM-based studies across 112 tasks.Examples of phase-specific tasks include fault localization and program repair in software maintenance.Source: §4 · p. 10

4.1 Software Requirements & Design

• LLMs support requirements and design work across specification generation, classification, quality assurance, repair, ambiguity detection, GUI layouts, and additional modeling tasks.The section covers seven named areas and also reports class diagram derivation, completeness detection, elicitation, prioritization, traceability, software modeling, and use case generation.Source: §4.1 · p. 10, §4.1.3 · p. 13, §4.1.4 · p. 13

4.1.1 Software Specifications Generation

• Few-shot prompting, code comprehension, and sequence-to-sequence learning are used to generate formal software specifications from comments or documentation.The surveyed approaches include empirical prompt studies, SpecGen, and SpecSyn, which translates natural-language input into formal specifications.Source: §4.1.1 · p. 11

4.1.2 Software Requirements Classification

• Requirement classification studies address functional and non-functional categories using transfer learning, pretrained embeddings, BERT, and explainability-driven fine-tuning.Han et al. extract concerns and generate explanations before fine-tuning BERT for classification.Source: §4.1.2 · pp. 12–13

4.1.3 Requirement Quality Assurance

• LLMs are studied for requirement quality assurance, including coverage review, design-to-requirement assessment, and user-story evaluation.The surveyed systems use Llama 2, GPT-3.5, GPT-4, BERT-style models, and ChatGPT across these quality tasks.Source: §4.1.3 · p. 13

4.1.4 Software Specifications Repair

• ChatGPT repairs some Alloy specification errors that other tools fail to correct, but it does not consistently exceed them in total repair count.The evaluated errors include logical inconsistencies, type errors, and misuse of programming constructs.Source: §4.1.4 · p. 13

4.1.5 Requirement Ambiguity Detection

• BERT-based and ChatGPT-based approaches detect or resolve requirement ambiguities, including contextual word meanings and anaphoric references.TABASCO uses BERT for contextual ambiguity, while other studies examine SpanBERT and ChatGPT for anaphora interpretation and resolution.Source: §4.1.5 · p. 13

4.1.6 GUI Layouts

• LLMs support GUI design by retrieving reusable interfaces and suggesting diverse layouts from textual instructions.Instigator uses LLMs to search and recommend GUI layouts, while BERT retrieves reusable GUIs from a large repository.Source: §4.1.6 · p. 13

4.2 Software Development

• LLM-based software development research spans code generation, search, translation, co-evolution, comment completion, summarization, completion, synthesis, editing, and related API and code tasks.The section organizes development applications into nine named subsections and lists additional tasks such as API inference, code representation, type inference, and unified development.Source: §4.2 · p. 13, §4.2.1 · p. 13, §4.2.2 · p. 14

4.2.1 Code Generation

• Code generation approaches use requirement clarification, intermediate artifacts, self-planning, execution feedback, multi-role collaboration, and empirical evaluation to improve generated programs.Studies also explore repository-level, retrieval-augmented, and agent-based generation, while empirical work examines robustness, efficiency, human judgments, prompts, benchmarks, and domain-specific generation.Source: §4.2.1 · pp. 13–14

4.2.2 Code Search

• Code search studies improve code–query representations through unimodal, bimodal, and multimodal contrastive learning, alongside evaluations of transfer learning and industrial adaptability.CodeRetriever learns function-level representations, CoCoSoDa aligns code snippets with natural-language queries, and other work evaluates scalability, robustness, and semantic sensitivity.Source: §4.2.2 · p. 14

4.2.3 Code Translation

• Code translation methods add test-case generation, iterative repair, and semantic-error detection, while empirical studies classify bugs across token, syntactic, library, and algorithm levels.UniTrans enhances translation accuracy through tests and repair, and TransMap detects semantic mistakes in translations produced by Codex and ChatGPT.Source: §4.2.3 · p. 14

4.2.4 Code Co-Evolution

• Code co-evolution learns edit sequences that synchronize changes between programming languages, exemplified by Codeditor for Java and C# across eight open-source projects.Source: §4.2.4 · p. 14

4.2.5 Code Comment Completion

• The T5 model outperforms an n-gram model in all tested Java code-comment completion scenarios, although the n-gram model remains competitive.Source: §4.2.5 · p. 14

4.2.6 Code Summarization

• Code summarization research combines code-summary pretraining, adapter or joint tuning, prompt engineering, static analysis, and empirical studies of explainability and metrics.Prompt-based methods add semantic code information or static-analysis results, while studies evaluate ChatGPT and compare human attention with LLM-generated summaries.Source: §4.2.6 · p. 15

4.2.7 Code Completion

• Code completion systems use semantic information, mutation-based error detection, repair, retrieval, static analysis, and reinforcement learning, with evaluations spanning token to block granularity.Source: §4.2.7 · p. 15

4.3.1 Fault Localization

• LLMs support fault localization, vulnerability detection, and decompilation by combining code and reports, program analysis, attention, static analysis, or multi-role refinement.Approaches include ChatGPT-based localization, line-level vulnerability detection, GPT with static analysis, and LLM-assisted variable-name recovery and decompiled-code correction.Source: §4.3.1 · p. 16, §4.3.3 · p. 16, §4.3.2 · p. 16

4.3.4 Unit Test Generation

• Unit-test generation uses fine-tuning, prompts, conversations, execution feedback, mutation testing, coverage information, and repair loops to improve validity, coverage, and bug detection.Representative systems include TestPilot, MuTAP, SymPrompt, CoverUp, TestART, TestRefineGen, ChatUniTest, CasModaTest, and CODAMOSA.Source: §4.3.4 · p. 17

4.3.5 Assertion Generation

• LLM-based assertion generation includes pretrained and fine-tuned models, few-shot prompting, test-to-code traceability, hardware benchmarks, formal postconditions, and multi-model empirical studies.The surveyed work targets both assertion and exceptional oracles, while also examining datasets, prompts, and benchmark quality.Source: §4.3.5 · pp. 17–18

4.3.6 Test Suite Minimization

• UniXcoder/Cosine is the best LTM configuration for combined effectiveness and efficiency, while LTM achieves higher fault detection and faster minimization than prior approaches.LTM represents test cases as embeddings and compares them with cosine similarity or Euclidean distance.Source: §4.3.6 · p. 18

4.3.7 Fuzzing

• LLMs are used for DL-library, compiler, protocol, general-purpose, and fuzz-driver generation, with structured prompting, coverage feedback, historical bug programs, and source-code analysis addressing traditional fuzzing limits.Fuzz4All improves code coverage across nine systems and six languages, while an empirical fuzz-driver study evaluates 189,628 generated drivers using 0.22 billion tokens.Source: §4.3.7 · pp. 18–19

4.3.8 Penetration Testing

• PentestGPT organizes penetration testing into reasoning, generation, and parsing modules, while related systems automate multiple strategies and support both planning and attack execution.These systems apply LLMs’ domain knowledge to complex penetration-testing workflows.Source: §4.3.8 · p. 19

4.3.9 Property-based Testing

• PBT-GPT generates random inputs and properties from API documentation, but its evaluation identifies failure modes and quality issues across validity, diversity, and strength.The study proposes prompting strategies, mitigation approaches, and an evaluation framework for generated property-based tests.Source: §4.3.9 · p. 19

4.3.10 Failure-Inducing Testing

• Differential Prompting uses program-intention inference, program generation, and differential testing, and significantly outperforms existing baselines on QuixBugs and Codeforces failure-inducing tests.Other testing applications cover mutation testing, GUI testing, NLP testing, and API, static-analysis, theorem-proving, and program-reduction tasks.Source: §4.3.10 · p. 19, §4.3.11 · p. 19, §4.3.13 · p. 20

4.4.1 Program Repair

• Program repair has progressed from a downstream code-refinement task to repair components, fine-tuned repairers, zero-shot cloze repair, hybrid traditional-LLM systems, and broad empirical evaluation.The surveyed transition includes CodeT5 and CodeBERT, CURE, fine-tuned BART/T5 systems, AlphaRepair, GAMMA, and studies of model size, fine-tuning, benchmarks, and repair costs.Source: §4.4.1 · pp. 20–21

4.4.2 Security Vulnerability Repair

• Simple fine-tuning enables LLMs to outperform state-of-the-art vulnerability repair techniques, and transfer learning from bug fixing can further enhance prediction accuracy.The empirical study covers more than 100 fine-tuned LLMs and examines preprocessing, training, inference, representation, and ChatGPT-based repair.Source: §4.4.2 · p. 22

4.4.3 Patch Correctness Assessment

• Developer-written test suites provide only partial behavioral specifications, so patches that pass available tests may overfit and fail on other potential test suites.The surveyed responses include feature extraction, fine-tuned and zero-shot patch assessment, semantic and syntactic reasoning, and LLM-generated fault-revealing tests.Source: §4.4.3 · p. 22

4.4.4 Commit Message Generation

• Maintenance studies use LLMs to generate commit messages, automate code review, detect duplicate reports, reproduce bugs, and update obsolete tests.Examples include CommitBERT, ExGroFi, CodeReviewer, Cupid, LIBRO, CrashTranslator, AdbGPT, CEPROT, SYNBCIATR, and TaRGet.Source: §4.4.4 · p. 23, §4.4.5 · p. 23, §4.4.6 · p. 23

4.4.9 Log Analysis

• LLM-based log analysis covers fine-tuning, few-shot, and zero-shot learning for parsing, resampling, structuring, anomaly detection, question answering, interpretation, and categorization.The surveyed maintenance scope also includes code-clone detection and tasks such as bug triage, code smells, incident management, issue labeling, and compiler optimization.Source: §4.4.9 · pp. 23–24, §4.4.10 · p. 24

4.5 Software Management

• LLMs are used in software management for effort estimation, tool configuration, developer behavior analysis, and software repository mining.Examples include BERT-based maintenance-effort estimation, GPT-4 tool configuration, emotion-cause detection, and automated repository-question answering.Source: §4.5 · p. 24
• Software management applications remain relatively unexplored, while software testing and development receive more extensive LLM research attention.The survey attributes the stronger emphasis in development and testing partly to their natural fit with sequence-to-sequence code generation.Source: §4.5 · p. 24

5.1.1 Empirical Study

• Empirical studies examine LLMs across multiple tasks, human participation and replication, and software-engineering education.Studies range from evaluations of eight models across seven CodeXGLUE tasks to GPT-4 replication of seven empirical papers and classroom investigations.Source: §5.1.1 · p. 25

5.1.2 Benchmarking

• Benchmarks are divided between established SE datasets and new datasets targeting data leakage or emerging scenarios such as multilingual, project-level, and decompilation tasks.EvalGPTFix uses post-2021 competitive-programming problems, while CrossCodeBench contains 216 tasks and more than 54M data instances.Source: §5.1.2 · pp. 25–26

5.2 Security and Reliability

• LLMs face adversarial, backdoor, and imitation attacks that can induce incorrect classifications, malicious retrieval, or copied capabilities across SE tasks.Examples include ALERT and CodeAttack for adversarial manipulation, BADCODE for stealthier backdoors, and imitation models trained from target-model responses.Source: §5.2 · pp. 26–27

5.3 Domain Tuning

• Domain tuning targets fine-tuning cost and catastrophic forgetting through parameter-efficient and continual-learning methods.CIRCLE achieves state-of-the-art performance on five benchmarks with one continually trained repair model, while REPEAT replays representative prior examples to reduce forgetting.Source: §5.3 · p. 27

5.4 Compressing and Distillation

• Compression and distillation address deployment barriers by reducing memory and inference demands, including compression of CodeBERT and GraphCodeBERT to 3 MB.Compressor combines genetic-algorithm model simplification with knowledge distillation for vulnerability prediction and clone detection.Source: §5.4 · p. 27

5 RQ3: INTEGRATION PERSPECTIVE

• LLM-based SE research addresses benchmarks, empirical evaluation, attacks, domain tuning, and deployment constraints as complementary integration challenges.The survey emphasizes that adaptation effectiveness must be considered alongside security, reliability, computational cost, and practical deployment.Source: §5 · p. 24, §5.4 · pp. 27–28

Trade-off between Effectiveness and Model Size

• Large models improve performance but impose major training and deployment costs, including 12 days to train CodeT5-base and 63 GB for Code Llama-34B.These resource demands can prevent deployment on resource-constrained or real-time devices.Source: §6 · p. 28
• Future work should optimize model size through pruning, quantization, and distillation while developing lightweight or distributed models.The goal is to reduce resource requirements without significantly compromising performance.Source: §6 · p. 28

Exploring Task-oriented Domain LLMs

• General-purpose LLMs can be oversized, contain task-irrelevant knowledge, and mismatch downstream objectives, motivating task-oriented domain models.The survey specifically proposes unit-testing LLMs trained with objectives centered on test generation and update.Source: §6 · p. 28

Clean Evaluation Datasets

• Data leakage threatens evaluation because public code may appear in pre-training data, while commercial models often disclose neither architecture nor training data.The survey notes leakage between Defects4J and CodeSearchNet and reports that ChatGPT can solve some LeetCode problems from their identifiers.Source: §6 · p. 28
• Clean evaluation can use manually written, recently released, or closed-source programs, alongside tests for memorization.These sources aim to reduce contamination and better assess genuine problem solving.Source: §6 · pp. 28–29

Application on More SE Tasks

• LLM applications are concentrated in development, testing, and maintenance, leaving requirements, design, management, and other complex or rare tasks comparatively sparse.The survey recommends integrating LLMs into existing workflows for complex tasks and designing domain-specific models for underrepresented tasks.Source: §6 · p. 29

Multi-task and Multi-dimensional Benchmarks

• Benchmarks should expand beyond common tasks and performance metrics to cover areas such as unit-test generation, robustness, and efficiency.The survey identifies limited task coverage and insufficient attention to time efficiency and robustness in existing evaluations.Source: §6 · p. 29

Beyond Text-based LLMs for Vision-based SE

• Multimodal LLMs could combine text and image understanding for GUI testing tasks such as test generation, recording, and replay.This direction targets source code, test scripts, test reports, and interface images together.Source: §6 · p. 29

7 CONCLUSION

• The survey covers 62 representative code LLMs and 947 studies spanning 112 code-related tasks across five SE phases.It also discusses architectures, pre-training objectives, downstream tasks, empirical evaluation, security, tuning, compression, and distillation.Source: §7 · p. 30

0.1 Encoder-only LLMs

• CodeBERT extends BERT with natural-language/programming-language data, replaced token detection, and six programming languages, improving on CuBERT’s code-only Python pre-training.Source: §0.1 · p. 53
• GraphCodeBERT adds data-flow graphs, edge prediction, node alignment, and graph-guided attention to model source-code structure beyond token sequences.Source: §0.1 · p. 53

0.2 Encoder-decoder LLMs

• Encoder-decoder models broaden code generation and understanding through objectives such as span denoising, identifier prediction, code-AST prediction, edit-aware generation, and reinforcement learning from unit-test signals.Source: §0.2 · p. 54
• CodeT5 significantly outperforms earlier models on most CodeXGLUE tasks, while CodeT5+ reports substantial performance across 20 code-related benchmarks under zero-shot, fine-tuning, and instruction-tuning settings.Source: §0.2 · pp. 54–55

0.3 Decoder-only LLMs

• PaLM-Coder achieved 88.4% pass@100 on HumanEval and an 82.1% compile rate on the DeepFix code repair task.Source: §0.3 · p. 57
• Open and multilingual decoder-only models extend accessibility and deployment, including CodeGeeX’s 23-language training, IDE integration, and weekly generation of 4.7 billion tokens.Source: §0.3 · pp. 55–57

APPENDIX B: DETAILED SUMMARIZATION OF EXISTING LLM-BASED SE STUDIES

• Figure 3 continues a taxonomy of LLM applications in different software-engineering domains.Source: Figure 3 · pp. 59–61