Towards an Understanding of Large Language Models in Software Engineering Tasks

1 Introduction

• Software engineering spans requirements, design, development, testing, and maintenance, but complex tasks depend on engineers’ expertise and raise costs and development challenges.The challenges include collaborative development, improving software quality, and shortening development cycles.Source: §1 · pp. 2–3
• Existing automation efforts include code generation, test generation, and vulnerability detection, but may not generalize and can produce vulnerable code or incomplete test coverage.The introduction notes that automated solutions may also introduce new issues or errors.Source: §1 · p. 3
• LLMs trained on code may learn syntax and semantics for code completion and summarization, yet a systematic review of their software engineering applications and evaluation is missing.The introduction also identifies high-quality code generation and high-coverage test generation as active research topics.Source: §1 · p. 3
• The review gathers literature from six academic databases, focuses on papers published after 2022, defines LLMs as transformer models with at least 0.8 billion parameters, and retains 123 papers.It uses card sorting to remove duplicate and irrelevant literature and aims to address two research questions.Source: §1 · p. 4

• The 123 selected papers are categorized according to the software engineering tasks they address.Source: RQ1: What are the current works… · p. 4
• The seven categories are Code Generation, Code Summarization, Code translation, Vulnerability Detection, Code Evaluation, Code Management, and Q&A Interaction.Source: RQ1: What are the current works… · p. 4
• The paper provides definitions and examples for each category to help researchers identify and address potential issues in applying LLMs to software engineering.Source: RQ1: What are the current works… · p. 4

RQ2: Can LLMs truly help better perform current software engineering tasks?

• LLMs excel at syntax-oriented tasks such as code summarization and repair, but perform less satisfactorily on semantic tasks such as code generation and vulnerability detection.The review also finds that performance continues to improve with each model iteration.Source: RQ2: Can LLMs truly help better… · p. 4
• The review manually selected 123 works from six databases and categorized software engineering applications of LLMs into seven task types.It provides examples and detailed explanations for each category to help researchers identify challenges in applying LLMs.Source: RQ2: Can LLMs truly help better… · p. 5
• The paper compiles LLM performance across software engineering tasks and analyzes differences among studies to inform optimization.The evaluated studies vary in their selected models and tasks, which the review organized during screening.Source: RQ2: Can LLMs truly help better… · pp. 4–5

2.1 Transformer

• Transformer attention enables parallel computation and captures long-range dependencies, supporting LLM training on large text corpora (Vaswani et al., 2017).The paper contrasts this with recurrent and convolutional neural networks.Source: §2.1 · p. 5

2.2 Model Architecture

• LLMs use encoder-decoder, causal-decoder, or prefix-decoder architectures, with causal decoders now the prevailing choice (Zhao et al., 2023b).Causal decoders use a diagonal mask so each token accesses only preceding tokens; GPT-3 demonstrated few-shot capability as model size and pretraining data increased (Brown et al., 2020).Source: §2.2 · pp. 6–7
• Encoder-decoder models encode input sequences before autoregressively generating outputs, suit sequence-to-sequence tasks, and are now uncommon among LLMs.Examples of encoder-decoder LLMs include Flan-T5 (Chung et al., 2022) and CodeT5+ (Wang et al., 2023f).Source: §2.2 · p. 6
• Prefix decoders attend bidirectionally over prefix tokens but unidirectionally over generated tokens, enabling generation conditioned on both (Tay et al., 2022; Zeng et al., 2022a).U-PaLM and GLM-130B are examples of prefix-decoder models.Source: §2.2 · p. 7

2.3 Emergent Abilities

• As parameters and training data increase, LLMs can develop emergent abilities, including in-context learning, instruction following, and step-by-step reasoning (Kaplan et al., 2020; Wei et al., 2022; Brown et al., 2020; Sanh et al., 2021; Ouyang et al., 2022; Wei et al., 2021; Shanahan, 2022).The paper describes these abilities as appearing when scaling passes a threshold.Source: §2.3 · p. 7

3 Methodology

• The review comprised literature search, literature screening, and data analysis.The methodology overview presents these as the three steps of the review.Source: §3 · p. 8, Figure 1 · p. 8

3.1 Literature Search

• Six databases were searched with six software-engineering and code LLM keywords, and the results required manual screening for duplicates and irrelevant papers.The databases were ACM Digital Library, IEEE Xplore, dblp, Science Direct, Google Scholar, and arXiv; the keywords included variants of “SE LLM,” “Software Engineering Large Language Model,” and “Code LLM.”Source: §3.1 · p. 8

3.2 Literature Selection

• Screening removed duplicate and irrelevant papers and excluded research conducted before 2022 because the definition of “Large” in LLM changes over time.Some papers may focus on LLMs or software engineering without mentioning “Large Language Model” in their abstracts, while earlier work may no longer meet the current LLM criteria.Source: §3.2 · pp. 8–9

Exclusion Criteria

• It also excludes studies not written in English and master’s or Ph.D. theses.Source: Exclusion Criteria · p. 9
• Pre-2022 studies are excluded: SE use of Transformer-based PLMs emerged around 2021, while LLM applications gained attention around 2022.Earlier SE work focused on PLMs and rarely covered code generation, reflecting the shift from encoder-decoder architectures to decoder-only LLMs.Source: Exclusion Criteria · pp. 9–10
• The review excludes studies outside the LLM–software-engineering intersection, including work focused solely on either area.The authors specifically focus on the intersection of LLM and software engineering.Source: Exclusion Criteria · pp. 9–10

Inclusion Topics

• The inclusion topics were LLMs in software engineering, their applications such as code generation, and empirical studies of LLMs on software engineering tasks.Source: Inclusion Topics · p. 10
• Six experienced researchers reviewed results from six databases, used closed card sorting to classify papers as relevant or irrelevant, and identified 123 papers for inclusion.Cards used paper titles and abstracts; researchers aligned their individual reviews through discussion.Source: Inclusion Topics · pp. 10–11

3.3 Data Analysis

• The analysis retained post-2022 work meeting Zhao et al.’s (2023b) LLM definition and answering either research question, then used open card sorting to identify findings.For the integration question, papers were grouped by task; tasks with only two related articles were classified separately, while others went under “Other Work.”Source: §3.3 · p. 11
• The task classification was checked against 40 LLM-related ICSE 2024 papers, grouped into seven classes; benchmarks and datasets were classified by research target, not as tasks.The authors state that the classes represent prominent LLM-and-SE research areas and encompass a significant portion of software engineering tasks.Source: §3.3 · p. 11
• 24 papers focused on Code Generation, 17 on Vulnerability Detection, 6 on Code Management, and 3 on Code translation; counts for the other tasks were similar.The task counts are presented in Fig. 2.Source: Figure 2 · p. 12, §3.3 · p. 12
• For the performance question, the researchers examined papers’ critical or explicit assessments of LLM performance, including instance studies and surveys.Source: §3.3 · p. 12

4.1 Code Generation

• Code-generation research evaluates LLM capabilities across languages and contexts while seeking to improve generated code quality, correctness, security, reliability, efficiency, and watermarking.Code completion, refactoring, and augmentation are included; generated code can contain errors and is not guaranteed to be usable.Source: §4.1 · pp. 13, 15, 16

4.2 Code Summarization

• Code-summarization studies assess models from varied perspectives and generally recognize their strong potential, while also applying them to documentation, test generation, robustness, and developer interaction.The task covers code understanding and summary generation at line, function, and module granularity.Source: §4.2 · pp. 16–17

4.3 Code Translation

• Code-translation research remains limited; studies examine language conversion and reverse engineering, while correctness, scarce high-quality code pairs, and evaluation ambiguity remain challenges.One empirical study covers C, C++, Go, Java, and Python.Source: §4.3 · p. 17

4.4 Vulnerability Detection& Repair

• Vulnerability-detection and repair research evaluates LLMs, improves correctness and performance, and applies models to vulnerability mining and related security tasks.Improvement approaches include formal verification, use of prior edits, and self-debugging; applications also include decompilation, hardware security analysis, and black-box testing.Source: §4.4 · p. 18

4.5 Code Evaluation

• Code-evaluation research applies LLMs to code assessment and increasingly to testing, especially test-case generation, testing frameworks, and testing assistants.The section’s scope also includes tasks such as code cloning, code-smell detection, and evaluating code quality.Source: §4.5 · pp. 18–19

4.6 Code Management

• Code-management work uses LLMs to reduce management complexity and pursue automation, including assistance with security incidents, repository context, version management, and personnel training.The motivation includes conflicts in collaborative development and the expertise required for complex merge and branch management.Source: §4.6 · pp. 19–20

4.7 Q&A Interaction

• Software-engineering Q&A research studies practitioner–LLM interaction, prompt engineering, and interaction security and efficiency, alongside prototypes and frameworks for developer support.Examples include cue-based chatbot design, developer interviews, and a human–LLM framework for visual programming.Source: §4.7 · pp. 20–21

4.8 Other Works

• Other work explores LLM performance in software-engineering subtasks, proposes solutions, develops tools or datasets, identifies issues, and investigates educational applications.Topics include visualization, information extraction, modeling, performance issues, and code-text datasets.Source: §4.8 · p. 21

Evaluation approach

• Because papers use inconsistent evaluation methods and criteria, the review converts reported conclusions into author-confidence categories and also presents the datasets and metrics used.The categories are High Confidence, Low Confidence, and Ambiguity; the review notes that confidence can coexist with reservations or a positive view of future potential.Source: §5 · pp. 21, 25, 26

ChatGPT evaluations

• ChatGPT performs well on log summarization, referential resolution, and code summarization, but struggles with vulnerability detection and test generation, with limited attention scope also reported.In vulnerability detection, it has difficulty distinguishing subtle code differences when versions have similar syntax; some authors remain positive about its potential in Text-to-SQL, testing questions, and test generation.Source: §5 · p. 21

Task-level evaluations

• ChatGPT handles code syntax, including Abstract Syntax Trees, better than code semantics, especially dynamic semantics; evaluations of several other specialized tasks find LLMs lagging state-of-the-art methods.These task evaluations cover reverse engineering, code explanation, code analysis, and vulnerability repair; other studies report positive findings for automated program repair, code generation, error fixing, and simple-game test generation.Source: §5 · pp. 24–25

Education, robustness, and security

• Studies of LLMs in education and programming courses report positive feedback, and one study finds GPT-4 significantly advanced over GPT-3; a small set of studies also examine robustness and security.The supplied discussion notes important findings on robustness and security but does not specify them.Source: §5 · p. 25

Task-level evaluations

• Authors report low confidence in code generation despite hopeful assessments, high confidence in program repair and Text-to-SQL, low confidence in vulnerability detection, and potential in test generation.Code summarization and explanation usually perform well but lack robustness; newer models have improved code generation, while test-generation studies reach contradictory conclusions.Source: §5 · p. 26

Overall findings

• LLMs generally do well on syntax-focused tasks such as code summarization and repair, struggle with semantic tasks such as code generation and vulnerability detection, and continue to improve across versions.They still do not fully match professional human programmers, but can assist developers by answering development questions and providing code examples.Source: §5 · p. 26

6.1 Other works on reviewing LLM

• This review systematically analyzes LLM applications, performance, and limitations across SE tasks, unlike broader LLM surveys that do not comprehensively examine SE.Zhao et al. focus on large-scale models and treat code synthesis as a language-generation capability; Gozalo-Brizuela and Garrido-Merchan classify generative AI but do not focus on LLM task performance.Source: §6.1 · p. 27
• Liu et al. (Liu et al., 2023e) review ChatGPT and GPT-4 in NLP, Fan et al. (Fan et al., 2023a) analyze over 5,000 LLM papers, and Wei et al. (Wei et al., 2023) survey CLMs and PLMs.These surveys address NLP applications and ethical concerns, LLM research across fields, or language-model development, rather than LLM applications in SE tasks.Source: §6.1 · p. 28
• Other reviews cover text generation, controllable generation, biomedical PLMs, and LLM-based recommendation, but focus on their respective domains or paradigms.Li et al. (Li et al., 2022a), Yang et al. (Yang et al., 2023b), Min et al. (Min et al., 2023), Wang et al. (Wang et al., 2023a), and Wu et al. (Wu et al., 2023) review these topics.Source: §6.1 · p. 28
• Related work also reviews LLM limitations, while SE-focused studies narrow their scope to code generation, AI-assisted programming, or deep learning.Meade et al. (Meade et al., 2022), Huang and Chang (Huang and Chang, 2023), Mialon et al. (Mialon et al., 2023), and Xu and McAuley (Xu and McAuley, 2023) examine bias, reasoning, hallucination mitigation, and model compression; Zan et al. (Zan et al., 2023), Wong et al. (Wong et al., 2023), and Watson et al. (2022) address narrower SE or programming topics.Source: §6.1 · p. 29

6.2 Pre-training model for Software Engineering Tasks

• Prior PLM work applied models to SE tasks including variable naming, bug classification, source-code generation, comment generation, and prompt learning.Examples include Chen et al. (Chen et al., 2022b), Lee et al. (Lee et al., 2022b), Zhang et al. (Zhang et al., 2022d), Li et al. (Li et al., 2022b), and Luo et al. (Luo et al., 2022).Source: §6.2 · p. 30
• PLM evaluations examined code comprehension and robustness, merge-conflict repair, code-task performance, bug triaging, programming-language grammar, interpretability, and automated code review.The cited studies span model probes and empirical comparisons; Tufano et al. (Tufano et al., 2022) reported positive results and superiority over previous deep-learning models for automated code review.Source: §6.2 · p. 30
• Because PLMs may lack strong code-generation capabilities, related work combined models with program analysis, specialized code pretraining, adversarial transformations, and compression.Examples include post-processing to improve generation reliability and SPT-Code, a sequence-to-sequence model designed for source code.Source: §6.2 · p. 30

7.1 Internal Validity

• Separate analyses by multiple experienced participants, followed by alignment to resolve disagreements, were used to reduce subjectivity in paper screening and content analysis.The participants, including non-co-authors, each had more than three years of software engineering research experience.Source: §7.1 · p. 31
• The authors’ early-stage task classification may omit aspects of software engineering, while differences in evaluation datasets and task perspectives can make results vary across studies.They describe the classification as capturing a significant portion of important tasks and organize evaluation results, datasets, and perspectives to help readers understand variation.Source: §7.1 · p. 31

7.2 External Validity

• Rapid changes in LLM research limit the findings to current work, and overlapping software engineering tasks can create classification ambiguity.The authors say their categorization remains applicable within the current research scope and define task boundaries, including placing code cloning and code smells under code assessment.Source: §7.2 · p. 31
• Because the accuracy of every reviewed study cannot be verified, RQ2 results may be biased; the authors report evaluation details and address contradictory assessments in their conclusion.They suggest tentative conclusions when studies consistently report poor or positive performance for a task, and describe their cross-task evaluation summary as fair.Source: §7.2 · pp. 31–32

• The review screens 123 works, categorizes them by software engineering task, and summarizes evaluations to expose LLM capabilities, limitations, and integration deficiencies.It aims to guide future research and optimization.Source: §8 · p. 32
• Many proposed methods for improving LLM performance on software engineering tasks lack detailed evaluation, which may indicate performance is insufficient or unstable.Source: §8 · p. 32
• Current evaluations mostly use general models such as ChatGPT, while detailed evaluations of code-centric models such as Codex remain lacking.Source: §8 · p. 32
• The authors identify whether task-specific fine-tuning should produce software-engineering models as an open question for future work.Source: §8 · p. 32

9 Conflict of Interests

• The authors declare no conflict of interest, including commercial or associative interests, and state that all authors approved the manuscript, which is original work not previously published or under consideration elsewhere.Source: §9 · pp. 32–33