PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends

Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, Jiachi Chen

arXiv:2311.10372v2cs.SE

TL;DR

Code LLMs have lacked a systematic review of their relationships and performance relative to general LLMs. This survey analyzes research and benchmark performance, finding that Code LLMs tend to outperform in code generation while GPT-4 remains competitive in other tasks.

  • Problem

    Code LLMs and their relationships to general models, as well as their performance across software-engineering tasks, have lacked systematic review.

    Source: §1 · pp. 2–3

  • Method

    The authors reviewed 134 works, categorized Code LLMs by developer-institution type and relationships, and compiled benchmark scores for 126 Code LLMs.

    Source: §7 · p. 38

  • Results

    Fine-tuned Code LLMs outperform their base models, and similarly sized Code LLMs tend to outperform general LLMs; leading Code LLMs outperform GPT-4 in code generation, while GPT-4 remains competitive on other tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The review can help Code LLM developers choose base models and help practitioners understand improvement directions for Code LLMs.

    Source: §1 · p. 3

    Outside code generation, results are less precise because comparative evaluations and relevant benchmarks are lacking.

    Source: §5.6 · p. 37

Abstract

from arXiv · show

General large language models (LLMs), represented by ChatGPT, have demonstrated significant potential in tasks such as code generation in software engineering. This has led to the development of specialized LLMs for software engineering, known as Code LLMs. A considerable portion of Code LLMs is derived from general LLMs through model fine-tuning. As a result, Code LLMs are often updated frequently and their performance can be influenced by the base LLMs. However, there is currently a lack of systematic investigation into Code LLMs and their performance. In this study, we conduct a comprehensive survey and analysis of the types of Code LLMs and their differences in performance compared to general LLMs. We aim to address three questions: (1) What LLMs are specifically designed for software engineering tasks, and what is the relationship between these Code LLMs? (2) Do Code LLMs really outperform general LLMs in software engineering tasks? (3) Which LLMs are more proficient in different software engineering tasks? To answer these questions, we first collect relevant literature and work from five major databases and open-source communities, resulting in 134 works for analysis. Next, we categorize the Code LLMs based on their publishers and examine their relationships with general LLMs and among themselves. Furthermore, we investigate the performance differences between general LLMs and Code LLMs in various software engineering tasks to demonstrate the impact of base models and Code LLMs. Finally, we comprehensively maintained the performance of LLMs across multiple mainstream benchmarks to identify the best-performing LLMs for each software engineering task. Our research not only assists developers of Code LLMs in choosing base models for the development of more advanced LLMs but also provides insights for practitioners to better understand key improvement directions for Code LLMs.

1 INTRODUCTION

The introduction frames a gap in systematic comparisons of Code LLMs with general LLMs and across software engineering tasks. It presents a review of their origins, relationships, and task-specific performance.

  • Code LLMs are designed for software engineering and may be produced through fine-tuning or traditional pretraining.Source: §1 · p. 2
  • Uncertainty from fine-tuning and training data, alongside strong general models such as GPT-4, makes their relative performance difficult to determine.Source: §1 · p. 2
  • The review classifies Code LLMs by developer institution and traces their chronology and development relationships, including iterations, fine-tuning, and improvements.Source: §1 · pp. 2–3
  • Code LLMs outperform their base models after fine-tuning, and comparable-size Code LLMs tend to outperform general LLMs, but GPT-4 remains competitive beyond code generation.The introduction names CodeFuse-CodeLlama-34B as outperforming GPT-4 in code generation.Source: §1 · p. 3
  • The paper presents itself as the first systematic review of Code LLMs and analyzes their performance against general LLMs across software engineering tasks.Source: §1 · p. 3

2 METHODOLOGY

The methodology searches four databases and communities, screens candidate works using predefined relevance categories, and analyzes the literature against three research questions.

2 METHODOLOGY

  • The review follows three stages: literature search, literature screening, and data analysis.Source: §2 · p. 3

2.1 Literature Search

  • Six search terms were used across dblp, Google Scholar, GitHub, and arXiv to find relevant publications, preprints, and open-source work.Source: §2.1 · pp. 3–4

2.2 Literature Selection

  • Closed card sorting classified papers as relevant or irrelevant; after independent review, discussion, and snowballing, 134 papers were selected.The authors excluded works outside the stated scope, including non-English studies, theses, duplicate papers, and work unrelated to LLMs or software engineering.Source: §2.2 · p. 5

2.3 Data Analysis

  • Open card sorting organized evidence for the research questions, including model origins and relationships, comparisons of Code and general LLMs, and task-specific performance.Source: §2.3 · pp. 5–6

3 LLMS FOR SOFTWARE ENGINEERING

This section surveys software-engineering LLMs across companies, universities, research teams, open-source communities, and individual developers, organizing models by affiliation and tracing their relationships and training or data lineages. It also describes model designs and task specializations alongside reported benchmark performance and evaluation gaps.

3.1 Company-led LLMs

  • A 1.3B Phi-1 [31] trained on 7B high-quality samples outperformed some larger models, while JuPyT5 [12] solved 77.5% of DSP problems in 100 sampling tests.The section also presents data quality as important to Phi-1’s performance.Source: §3.1 · p. 7
  • AlphaCode addressed competition-style programming with deep reasoning and performed around the Codeforces median; ERNIE-Code and Replit-Code lack systematic performance evidence in the reported work.AlphaCode’s authors noted that scaling upsampling and filtering to cluster candidate solutions can improve performance; ERNIE-Code lacks suitable multilingual benchmarks, and Replit-Code lacks technical and testing reports.Source: §3.1 · pp. 8–9
  • PanGu-Coder2 [81] achieved 62.20% pass@1 on HumanEval and reportedly outperformed previous Code LLMs on CoderEval and LeetCode.It builds on PanGu-Coder and uses the RRTF framework to align test and teacher feedback.Source: §3.1 · p. 9

3.2 University-led LLMs

  • PLBART [1] outperformed several baselines across code tasks; PolyCoder led contemporaneous models on C, while CodeGeeX [116] outperformed similarly scaled open models on generation and translation.PolyCoder was trained on 249GB of code in 12 languages; CodeGeeX used 850 billion tokens from 23 languages.Source: §3.2 · p. 11
  • With 6B parameters, CodeGeeX2 achieved nearly a 10% improvement over StarCoder-15B and supports Chinese and English inputs.It is based on ChatGLM2, supports sequences up to 8192 tokens, and can run after quantization with 6GB of GPU memory.Source: §3.2 · p. 11
  • MultiCoder [29] and MultiPL-T [10] target low-resource programming languages, while VeriGen [91] and Codeeditor [112] report gains on Verilog generation and code translation.MultiCoder’s PL-MoE improved performance across six languages; VeriGen exceeded GPT-3.5-turbo by 1.1%, and Codeeditor outperformed Codex and CodeT5 across metrics.Source: §3.2 · pp. 11–12

3.3 Research teams & Open source communities-led LLMs

  • Codex [17] solved 28.8% of HumanEval problems in a single sample, and FIM [7] addressed infilling by enabling generation conditioned on both a prefix and suffix.FIM modifies training data without changing model structure, and the approach was later adopted by models including InCoder and SantaCoder.Source: §3.3 · pp. 12–13
  • CodeT5 [100] uses identifier-aware masking, CodeRL [45] adds reinforcement-learning feedback, and CodeGen [67] links multi-step synthesis capability to model size and data scale.The section also describes CodeGen2 [66] as unifying four training components through a prefix-based language-modeling approach.Source: §3.3 · pp. 13–14
  • CodeGen2 results found no measurable Prefix-LM gain or infill-sampling free lunch, while within-file span corruption sufficed and mixed programming/natural-language distributions showed promise.CodeGen2 combined causal language modeling with span corruption and examined programming-language and natural-language data mixing.Source: §3.3 · p. 14
  • CodeT5+ [98] combines denoising, contrastive, text-code matching, and causal objectives; CodeT5Mix combines RoBERTa, GPT-2, and CodeGPT components for varied code tasks.CodeT5+ can initialize from frozen pretrained LLMs; CodeT5Mix supports retrieval-augmented code generation through code retrieval and generation.Source: §3.3 · p. 14
  • StarCoderBase, a 15.5B-parameter model trained on 1 trillion Stack tokens, outperformed existing open-source code LLMs on popular programming benchmarks; StarCoder adds 35B Python fine-tuning [50].StarCoder supports context lengths over 8,000 tokens; the BigCode community also released SantaCoder [3] and The Stack v1.1, a filtered 6.4TB dataset spanning 384 languages.Source: §3.3 · p. 15
  • OctoCoder and OctoGeeX achieved optimal performance on HUMANEVALPACK and outperformed GPT-4; the benchmark covers fixing, interpretation, and synthesis across six languages [64].The same work introduced COMMITPACK, a 4TB Git commit dataset spanning 350 programming languages.Source: §3.3 · p. 15
  • BLOOM lacked comprehensive software-engineering evaluation and limited code-generation tests found it weaker than Codex; GPT-CC reported no cross-LLM comparison or Code Clippy Data effectiveness analysis.Code Clippy Data is a 132GB dataset assembled from deduplicated GitHub code and code from the Pile.Source: §3.3 · p. 15
  • OpenCoderPlus reached 102.5% of ChatGPT’s Vicuna GPT-4 evaluation score and a 78.7% AlpacaEval win rate, while CodeFuse-CodeLlama-34B outperformed GPT-4 on HumanEval.OpenCoderPlus is fine-tuned on code with StarCoderPlus as its base model.Source: §3.3 · pp. 15–16
  • On HumanEval, CodeLlama-34B and CodeLlama-34B-Python scored 67.6% and 69.5% pass@1, while Phind-CodeLlama-34B-v2 scored 73.8% versus GPT-4’s 67%.The reported scores used an internal Phind dataset.Source: §3.3 · p. 16

3.4 Individuals & Anonymously-led LLMs

  • CodeAlpaca instruction-tunes LLaMA [94] on code-generation instructions, while Fig. 4 maps arrows from source models to models derived or developed from them.The described derivation can include improvements, fine-tuning, or borrowing techniques.Source: §3.4 · p. 16

4 DO CODE LLMS REALLY PERFORM BETTER THAN GENERAL LLMS IN SOFTWARE ENGINEERING TASKS?

The surveyed comparisons favor Code LLMs overall, yet do not establish a universal winner: outcomes vary by task and study settings, and benchmark fit and model scale also matter.

4 DO CODE LLMS REALLY PERFORM BETTER THAN GENERAL LLMS IN SOFTWARE ENGINEERING TASKS?

  • Model-size effects vary: CodeGen-Mono sizes differ little on MBPP and HumanEval, while Codex 2.5B and 12B differ more [110].Source: §4 · p. 19
  • Task winners vary: GPT-4 and GPT-3.5 outperform Code LLMs on ClassEval [25], whereas CodeT5 leads ChatGPT on CSN-Python summarization [87].Source: §4 · pp. 18, 21
  • 20 experiments favored Code LLMs, 11 favored general LLMs, and 7 were inconclusive across the 38 papers summarized in Table 3.The table aggregates comparisons across different software engineering tasks and studies.Source: §4 · p. 21, Table 3 · p. 22
  • Cross-study conclusions are difficult because model access, publication dates, baselines, benchmarks, tasks, and model versions differ [87] [43] [83] [25] [27] [105].Source: §4 · p. 21
  • At comparable parameter sizes, Code LLMs tend to outperform general LLMs [26], and current leading Code LLMs outperform GPT-4 in code generation [75].Source: §4 · p. 21
  • For the same model, software-engineering fine-tuning generally improves performance over the base model [50] [113].Source: §4 · p. 21
  • Encoder-decoder models perform worse than decoder-only models on HumanEval, which the cited work attributes to a mismatch with the benchmark setup.The authors argue that diverse benchmarks are needed because one benchmark may not capture Code LLM capabilities fully.Source: §4 · pp. 21–22

5 WHICH SOFTWARE ENGINEERING TASKS ARE DIFFERENT LLMS PARTICULARLY GOOD AT?

This section surveys LLM performance across software engineering tasks and benchmarks to identify task-specific model leaders. It also notes that evidence beyond code generation is sparse and often not comparable.

5.1 Code Generation

  • Unnatural-Code-LLaMA-34B leads HumanEval overall, while LATS (GPT-4 based) leads Pass@1; GPT-4 and Unnatural-Code-LLaMA-34B are identified as strong code-generation models.Unnatural-Code-LLaMA-34B leads Pass@10 and Pass@100; Code-LLaMA-Python-34B and -13B are the next best performers. The survey selects more recent reports where results for the same model vary.Source: §5.1 · p. 23, Table 4 · p. 24
  • Specialized code-generation evaluations yield different leaders: CodeGenAPI-6B for private libraries, ToolCoder for API calling, and GPT-3.5 for secure-code recommendations.CodeGenAPI-6B is based on CodeGen, but did not surpass CodeGen in Zhang et al.’s evaluation; the secure-code study’s models may not represent current state of the art, and results vary by programming language. LLMs also commonly exhibit API misuse in executable code aligned with user intent, without a clear best model.Source: §5.1 · pp. 25–26
  • Other code-generation evaluations favor CodeGen-2B for buggy-code completion, GPT-3.5-turbo for CodeApex, GPT-4 for ClassEval, and Codex for CoderEval.These comparisons are limited: CodeGen-2B was compared only with InCoder; CodeApex has few evaluations; and CoderEval compares only CodeGen, Codex, and PanGu-Coder. WizardCoder performs well on CodeApex and is the strongest Code LLM on ClassEval. Unnatural-Code-LLaMA-34B leads the available MBPP results, though many model scores are unavailable.Source: §5.1 · p. 27, §5.5 · p. 31

5.2 Test Case Generation

  • GPT-4 and GPT-3.5 (GPT-3.5-turbo) show stronger test-generation performance overall, with results varying by evaluation and model.GPT-3.5-turbo has slightly higher median statement and branch coverage than code-cushman-002, while StarCoder performs relatively poorly; another study reports Codex about three times better than CodeGen and ChatGPT improving further. One unit-test study provides no definitive winner.Source: §5.2 · pp. 28–29

5.3 Code Summarization

  • CodeT5 performs best among four evaluated models on CSN-Python, while CodeT5+ is reported to outperform GPT-3.5-turbo overall in code summarization.CodeT5Mix, an improved CodeT5 version, performs almost on par with CodeT5 without significant improvement; the evaluations cover a limited selection of models.Source: §5.3 · pp. 29–30

5.4 Code Translation

  • GPT-4 performs best overall in code translation; CodeGeeX-13B-FT and CodeGen-Multi-16B excel in different multilingual scenarios, while non-fine-tuned CodeGeeX-13B lags CodeGen-Multi-16B.On XLCoST, CodeGeeX narrowly outperforms CodeT5. Across seven datasets, models other than GPT-4 and StarCoder perform poorly; GO-related translation also fares poorly under stricter syntax constraints.Source: §5.4 · p. 30

5.5 Vulnerability Repair

  • Codex leads the reported vulnerability-repair comparisons, including outperforming TBar across the tested benchmarks; Unnatural-Code-LLaMA-34B leads the available MBPP results.The repair evidence is limited, with no comparison of state-of-the-art Code LLMs against GPT-4. In a zero-shot vulnerability-fixing study, code-davinci-002 performs relatively well, but current LLMs are not yet sufficient to deliver true value in program-repair frameworks.Source: §5.5 · pp. 31–33

5.6 Other Evaluation or New Benchmarks

  • WizardCoder-15B performs relatively well, especially on assertion generation, while guided LLMs can benefit from examples and downstream fine-tuning.Across new multilingual benchmarks, CodeGen-Mono-16B scores higher in each test. Multilingual models generally outperform equivalent-resource monolingual models when sufficiently large; few-shot prompts aid unfamiliar languages, fewer prompts reduce compilation or parsing errors, and multilingual models are more robust to prompt perturbations.Source: §5.6 · pp. 34–35
  • On BIOCODER, ChatGPT performs best across tasks, while larger InstructCodeT5+, CodeGen, and CodeGen2 fare worse than InCoder and SantaCoder.The authors attribute this to single-line rather than function-completion training and smaller context constraints, which affect model performance under different prompts.Source: §5.6 · p. 35, Table 26 · p. 36
  • Kou et al. [43] found CodeGen had the highest consistency with human programmers among five evaluated models, but did not test state-of-the-art models.Their comparison used keyword coverage and Cohen’s Kappa consistency.Source: §5.6 · p. 35
  • Beyond code generation, evaluations and widely recognized benchmarks are sparse, vulnerability repair receives limited attention, and comparisons among leading models are missing.The survey notes a lack of comparisons between state-of-the-art Code LLMs such as Unnatural-Code-LLaMA-34B and GPT-4 on code translation and summarization.Source: §5.6 · pp. 35, 37
  • Across tasks, Code-LLaMA models—especially Unnatural-Code-LLaMA-34B—lead code generation, ToolCoder API-related generation, GPT-4/GPT-3.5 test generation, CodeT5+ summarization, GPT-4 translation, and Codex vulnerability repair, based on limited results.The survey characterizes results outside code generation as less precise than code-generation results.Source: §5.6 · p. 37
  • Reported HumanEval performance varies across studies, especially for ChatGPT and GPT-3.5 models, and the reasons for these inconsistencies remain unclear.The survey says the variation is not simply due to citation errors.Source: §5.6 · p. 37

6 RELATED WORK

Prior work reviews Code LLMs, their capabilities, and software-engineering applications, while proposing models, tools, evaluation methods, and benchmarks. The surveyed work also identifies gaps in benchmark coverage and model capabilities.

  • Artetxe et al. [5] evaluated 27 code-based models released before December 2022 on HumanEval zero-shot and identified parameters, data quality, and expert tuning as success factors.Source: §6 · p. 37
  • Existing code-testing benchmarks often use English problem descriptions and Python solutions, leaving multiple natural and programming languages uncovered.Artetxe et al. [5] also identify challenges in comprehension, inference, explanation, adaptive learning, and multitasking.Source: §6 · p. 37
  • Zheng et al. [118] organized 123 works at the intersection of LLMs and software engineering by task, summarizing performance, limitations, and future directions.Source: §6 · p. 37
  • Enhancement efforts include CODETF for model and task access, XCode pretraining on about 1.5 million code snippets, and operators for grouping semantically equivalent variants.Lu et al. [55] discuss security, privacy, and licensing concerns in code models trained on uncleaned internet corpora and offer four recommendations.Source: §6 · pp. 37–38
  • Other proposed tools and approaches include CollabCoder for qualitative coding, Inductive Bias Learning, and COMEX for creating and harvesting code views.Source: §6 · p. 38
  • Evaluation studies examine code synthesis, completion, detection, and unit-test generation, while proposed code-generation benchmarks include CODETASKCL, CodeBLEU, CodeSearchNet, and Galeras.Source: §6 · p. 38

7 CONCLUSION

The paper surveys Code LLMs and their performance across software-engineering tasks. It organizes model relationships and benchmark results to inform model development and identify improvement directions.

  • The authors screened 134 works, categorized Code LLMs by their developers’ institutional types, and mapped relationships among Code LLMs and general LLMs.Source: §7 · p. 38
  • The paper analyzes and compiles general-LLM and Code-LLM performance across software-engineering tasks, reporting statistical results and observed phenomena.Source: §7 · p. 38
  • The authors maintained major-benchmark scores for 126 Code LLMs and analyzed performance across software-engineering tasks.They state that the overview can help developers choose base models and fine-tuning approaches and identify improvement directions.Source: §7 · p. 38
Loading 2311.10372v2…