PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Large Language Models for Code Generation

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, Sunghun Kim

arXiv:2406.00515v2cs.CLcs.AIcs.SE

TL;DR

Existing literature reviews have not provided a comprehensive, up-to-date synthesis focused specifically on LLMs for code generation and its advanced topics. This survey conducts a systematic review with a taxonomy, historical overview, and benchmark comparison, finding progressive capability improvements while identifying practical challenges and limitations.

  • Problem

    Existing reviews often span broad code-related tasks or mainly cover 2020–2022 text-to-code models, leaving recent code-generation advances insufficiently synthesized.

  • Method

    The survey systematically reviews LLMs for code generation, organizes developments with a taxonomy, traces their history, compares HumanEval, MBPP, and BigCodeBench, and examines practical challenges.

  • Results

    The survey documents progressive enhancements in code-generation capabilities and reports empirical comparisons across HumanEval, MBPP, and BigCodeBench, including strong results from recent models.

  • Takeaways & Limitations

    The survey provides a reference for researchers by connecting data, model advances, evaluation, applications, ethical and environmental issues, and the research-practicality gap.

  • Takeaways & Limitations

    LLM-based evaluation remains constrained by the evaluated LLMs’ position, verbosity, self-enhancement, and reasoning biases.

Abstract

from arXiv · show

Large Language Models (LLMs) have garnered remarkable advancements across diverse code-related tasks, known as Code LLMs, particularly in code generation that generates source code with LLM from natural language descriptions. This burgeoning field has captured significant interest from both academic researchers and industry professionals due to its practical significance in software development, e.g., GitHub Copilot. Despite the active exploration of LLMs for a variety of code tasks, either from the perspective of natural language processing (NLP) or software engineering (SE) or both, there is a noticeable absence of a comprehensive and up-to-date literature review dedicated to LLM for code generation. In this survey, we aim to bridge this gap by providing a systematic literature review that serves as a valuable reference for researchers investigating the cutting-edge progress in LLMs for code generation. We introduce a taxonomy to categorize and discuss the recent developments in LLMs for code generation, covering aspects such as data curation, latest advances, performance evaluation, ethical implications, environmental impact, and real-world applications. In addition, we present a historical overview of the evolution of LLMs for code generation and offer an empirical comparison using the HumanEval, MBPP, and BigCodeBench benchmarks across various levels of difficulty and types of programming tasks to highlight the progressive enhancements in LLM capabilities for code generation. We identify critical challenges and promising opportunities regarding the gap between academia and practical development. Furthermore, we have established a dedicated resource GitHub page (https://github.com/juyongjiang/CodeLLMSurvey) to continuously document and disseminate the most recent advances in the field.

1 INTRODUCTION

LLMs for code generation produce source code from natural-language descriptions, but existing reviews have often covered broader code tasks or stopped before recent advances. This survey addresses the gap through a systematic review, taxonomy, historical overview, empirical comparison, and discussion of challenges and opportunities.

  • Code generation produces source code from natural-language descriptions, optionally incorporating programming context such as function signatures or assertions.
  • Earlier code-generation techniques relied on rigid heuristic rules, expert systems, probabilistic grammars, or specialized language models, whereas Transformer-based LLMs became preferred for their proficiency and versatility.
  • Recent surveys often span many code-related tasks, leaving advanced code-generation topics such as data curation, instruction tuning, prompting, coding agents, and retrieval-augmented generation insufficiently reviewed.
  • The survey focuses primarily on code generation while also covering related applications including code completion, translation, repair, mutant generation, test generation, and code summarization.
  • The survey fills the review gap with a systematic literature review and taxonomy covering data curation, advances, evaluation, practical applications, challenges, and opportunities.
  • The survey is organized around preliminaries, review methodology, a taxonomy of the code-generation lifecycle, detailed developments, and the research-practicality gap.

2 BACKGROUND

Code LLMs use Transformer architectures and large-scale code-focused pre-training, with encoder-decoder and decoder-only designs supporting natural-language-to-code generation. The background also introduces attention, normalization, positional encoding, in-context examples, decoding strategies, and execution feedback.

  • LLM effectiveness is associated with model parameters, diversified large-scale datasets, and substantial training computation, although larger models can exhibit unpredictable emergent abilities.
  • Code LLMs are pre-trained or continually pre-trained on large unlabeled code corpora, typically with smaller portions of text and mathematics, and some incorporate synthetic data.
  • Transformer components: Transformer layers use multi-head self-attention to model semantic relationships across multiple latent representation spaces, followed by position-wise feed-forward processing.
  • Transformer components: Residual connections and layer normalization are used around Transformer modules, while positional embeddings supplement self-attention with token-position information.
  • Model architectures: The two principal code-generation architectures are encoder-decoder models, which encode inputs before decoding outputs, and decoder-only models, which use one stack for processing and generation.
  • Code-generation formulation: Code generation models map natural-language descriptions x to code solutions y through Pθ(y | x), optionally conditioning on k in-context exemplars.
  • Decoding and feedback: Generation may use greedy or beam search and sampling strategies, after which compiler or interpreter execution on unit tests supplies feedback for iterative refinement.

3 METHODOLOGY

The survey uses a systematic literature-review process to identify, filter, assess, and analyze research on LLMs for code generation. It combines manual venue review, automated database searches, eligibility criteria, quality assessment, and backward snowballing, yielding 235 papers.

  • Research Questions: The review addresses categorization, key insights, and challenges or opportunities in LLMs for code generation.Its research questions cover taxonomy and evaluation, data curation, advances, benchmarks, ethical and environmental implications, applications, and the academia–practice gap.
  • Search Strategy: The authors manually reviewed conference proceedings and journal articles to identify relevant studies and derive automated-search strings.The initial manual review yielded 42 papers and informed subsequent keyword queries.
  • Search Strategy: The automated search covered ACM Digital Library, IEEE Xplore, arXiv, and DBLP, retrieving 664 papers before filtering and deduplication.The search focused on titles containing keywords related to LLMs and code generation.
  • Eligibility Criteria: The review defines code generation as natural-language-to-code and excludes unrelated tasks such as code completion, translation, and text generation unless specifically in scope.This scope is applied through automatic content filtering and explicit inclusion and exclusion criteria.
  • Selection and Assessment: 235 papers remained after automatic filtering, eligibility screening, quality assessment, and backward snowballing.The sequence was 664 retrieved papers, 351 after automatic filtering, 247 after inclusion and exclusion criteria, and 235 after quality assessment and snowballing.

4 TAXONOMY

The taxonomy organizes LLMs for code generation by analyzing selected studies and structuring the field’s latest advances into a comprehensive reference. It supports discussion of problem definitions, challenges, model comparisons, and performance evaluation.

  • Taxonomy: The survey frames code LLM development as closely connected to broader LLM advances, including models adapted through continual pre-training or fine-tuning.Examples include Code Llama following LLaMA and code-focused models developed from general-purpose systems.
  • Taxonomy: The survey analyzes studies from widely used scientific databases to construct its taxonomy of LLMs for code generation.The taxonomy is intended to help researchers quickly familiarize themselves with the state of the art.
  • Taxonomy: The taxonomy’s recent-advances category emphasizes the core techniques used in current state-of-the-art code LLMs.The survey examines each category through definitions, challenges, prominent models, and performance evaluations.

5 LARGE LANGAUGE MODELS FOR CODE GENERATION

LLMs for code generation rely on curated code data, staged training, prompting, and evaluation across diverse benchmarks. The survey organizes these dimensions around data preparation, model development, instruction tuning, and benchmark design.

  • Overall Workflow: Code LLM development commonly includes code-data curation, pre-training, fine-tuning, feedback-based reinforcement learning, prompt engineering, inference, and evaluation.Figure 5 presents a general workflow connecting training stages, inference, evaluation, and associated databases.
  • Overall Workflow: The workflow separates pre-training into Stages 1 and 2 and post-training into Stages 3 and 4, with Stages 2 and 4 optional.StarCoder uses Stage 1, WizardCoder uses Stage 3, Code Llama uses Stages 2 and 3, and DeepSeek-Coder-V2 uses Stages 2, 3, and 4.
  • Data Curation and Processing: Raw code datasets require cleaning because they may contain redundancy, noise, and personal information that can create privacy-leakage concerns.Typical cleaning includes exact-match deduplication and code filtering based on line length and alphanumeric-character thresholds.
  • Data Curation and Processing: Code data are categorized into pre-training datasets, instruction-tuning datasets, and performance-evaluation benchmarks.The survey describes these three classes in subsequent sections as components of code LLM development.
  • Instruction Tuning: Instruction tuning is supervised fine-tuning on instruction-structured datasets intended to improve instruction following and generalization to unseen tasks.The survey discusses code-specific instruction-tuning datasets such as CodeAlpaca-20k and CommitPackFT.
  • Evaluation: Code-generation benchmarks span general-purpose, competitive-programming, data-science, multilingual, logical-reasoning, and repository-level contexts.These categories extend beyond the foundational HumanEval dataset to cover broader coding capabilities.

5.2 Data Synthesis

Synthetic data addresses the scarcity, cost, and coverage limitations of human-generated data, while enabling instruction datasets for code generation. However, its benefits remain constrained by diversity, factuality, fidelity, and bias risks.

  • High-quality datasets can substantially improve model performance, as LIMA matched or exceeded GPT-4 in 43% of evaluated cases using 1,000 curated examples.LIMA used a 65B-parameter LLaMA model without reinforcement learning or human preference modeling.
  • Synthetic data offers an alternative to labor-intensive human data collection affected by scarcity, privacy concerns, cost, and limited coverage of complex scenarios.The passage identifies synthetic data as a response to practical limitations in acquiring quality data.
  • Self-Instruct uses an off-the-shelf language model to generate instructions, inputs, and outputs, then removes invalid or redundant entries before fine-tuning.
  • Code Alpaca uses Self-Instruct to create 20k code instruction-following examples, while WizardCoder uses Evol-Instruct to expand the dataset to 78k evolved examples.
  • 57.3% pass@1 on HumanEval enabled WizardCoder 15B to surpass other open-source Code LLMs and leading closed models including Claude and Bard.
  • Synthetic data introduces risks involving insufficient diversity, uncertain factuality and fidelity, and amplification or creation of biases.

5.3 Pre-Training

Code-generation LLMs evolved from rigid heuristic and expert systems toward Transformer-based architectures trained on large unlabeled code corpora. Pre-training primarily uses causal language modeling or denoising autoencoding, with prompting enabling zero- and few-shot generation.

  • 5.3 Pre-Training: Transformer-based LLMs replaced early heuristic and expert-system approaches as the preferred code-generation method because of their proficiency and versatility.
  • 5.3.1 Model Architectures: Contemporary code LLMs generally use encoder-decoder architectures, such as CodeT5, or decoder-only architectures, such as Codex, StarCoder, Code Llama, and CodeGemma.
  • 5.3.1 Model Architectures: Pre-training on extensive unlabeled code corpora supplies programming knowledge about identifiers, code structure, and semantics while reducing dependence on scarce manual annotations.
  • 5.3.2 Pre-training Tasks: Causal language modeling trains decoder-only models to predict each token autoregressively from preceding tokens using causal attention masking.
  • 5.3.2 Pre-training Tasks: Encoder-decoder causal modeling conditions the generated target sequence on a source prefix and previously generated target tokens.
  • 5.3.2 Pre-training Tasks: During inference, pre-trained models can generate code zero-shot through prompt engineering, while few-shot or in-context learning can further enhance performance.
  • 5.3.2 Pre-training Tasks: Denoising autoencoding masks random spans in the input and trains encoder-decoder models to reconstruct them, requiring semantic relationships among token sequences.

5.4 Instruction Tuning

Instruction tuning supervisedly fine-tunes pre-trained LLMs on structured natural-language instructions, often using permissively licensed or synthetic code data. The section covers full-parameter and parameter-efficient strategies, including low-rank updates designed to reduce resource demands.

  • 5.4 Instruction Tuning: Instruction tuning is supervised fine-tuning on structured examples framed as natural-language instructions, with code descriptions guiding code generation.
  • 5.4 Instruction Tuning: Instruction data may come from permissively licensed source code or synthetic code data, and examples include generating primes or extracting URLs.
  • 5.4 Instruction Tuning: Code Alpaca’s instruction corpus contains varied tasks with distinct instructions, including prime-number generation and URL extraction.
  • 5.4.1 Full Parameter Fine-tuning: Full-parameter fine-tuning updates every pre-trained model parameter and is generally preferred when computational resources and training data are abundant.
  • 5.4.1 Full Parameter Fine-tuning: The StarCoder2-15B-instruct pipeline is described as fully self-aligned and trained with a fully permissive, transparent process, while other work combines open-source mathematics and synthetic code data.
  • 5.4.2 Parameter-Efficient Fine-tuning: Parameter-efficient fine-tuning updates only a minimal parameter subset or adds trainable parameters to reduce the computational and resource demands of fine-tuning.
  • 5.4.2 Parameter-Efficient Fine-tuning: LoRA represents the weight update as ΔW = B_trainable A_trainable with rank r much smaller than the matrix dimensions, while its scaling coefficient balances module importance.
  • 5.4.2 Parameter-Efficient Fine-tuning: CodeUp demonstrated parameter-efficient instruction tuning on Llama 2 with a single RTX 3090 GPU, while ASTRAIOS empirically examined such tuning for code comprehension and generation.

5.5 Reinforcement Learning with Feedback

Reinforcement learning with feedback aligns code-generation models using human, AI, or execution-based feedback, while newer methods address RL’s efficiency and scalability limits. Prompting-based self-refinement and repository-aware techniques extend feedback-driven improvement to broader coding contexts.

  • Feedback-based alignment: Human preference alignment uses RLHF, in which labelers rank model outputs and the model is further fine-tuned on those rankings.The passage identifies InstructGPT as a notable example and notes that collecting high-quality preference rankings is resource-intensive.
  • Feedback-based alignment: AI feedback methods use powerful language models such as ChatGPT and GPT-4 to simulate human annotators by generating preference data.
  • Execution feedback: Execution-based feedback lets compilers or interpreters evaluate generated code on unit tests, supporting reinforcement-learning approaches such as CodeRL.
  • Efficiency and scalability: 62.20% pass@1 was reported for PanGu-Coder 2 on HumanEval using RRTF, a non-PPO framework inspired by rank-response methods.
  • Prompting-based refinement: Prompting enables iterative code refinement without additional training by combining code explanations, execution results, and optional self-reflection.Self-Debugging is presented as an example of this pipeline.
  • Repository-level generation: Repository-level generation must handle cross-file dependencies, repository-specific conventions, context-length limits, and insufficient training data for proprietary or evolving projects.
  • Repository-level generation: Retrieval-generation systems such as RepoCoder, CoCoMIC, RepoHyper, RLPG, and Repoformer enrich repository context, while selective retrieval can bypass redundant retrieval.

5.8 Retrieval Augmented

Retrieval-augmented generation supplies external code, documentation, and other knowledge to improve code-generation context. Its effectiveness remains constrained by retrieval quality, integration, intent alignment, context size, and computational cost.

  • Concept and motivation: RAG retrieves relevant information from an external document corpus and combines it with the original query to improve response quality and accuracy.The survey presents RAG as a response to hallucination, obsolete knowledge, and opaque reasoning limitations.
  • Concept and motivation: Code LLMs can hallucinate or produce deprecated calls when instructions require information outside their training data or involve frequently updated libraries such as PyTorch.
  • Code-generation applications: Retrieval-augmented code methods include REDCODER for code generation, ReACC for code completion, and DocPrompting for retrieving relevant documentation.
  • Code-generation applications: RepoCoder uses iterative retrieval-generation and code analogies across repository files to improve repository-level code completion.
  • Limitations: Retrieval quality, query integration, intent alignment, context-window requirements, and computational demands remain central limitations of retrieval-augmented code generation.

5.9 Autonomous Coding Agents

Autonomous coding agents use language models to plan, remember, invoke tools, and execute software-engineering actions, including through multi-agent collaboration. Results show strong benchmark performance, but prompt design, context length, agent count, and toolsets still require refinement.

  • Agent architecture: Autonomous agents use an LLM as a controller to formulate and execute plans through tool or API calls while operating in shared environments.
  • Agent architecture: Agent architectures typically combine planning, memory, tools, and action to support problem solving and learning from past actions.
  • Multi-agent code generation: MetaGPT assigns code-related procedures to LLM roles such as Product Manager, Architect, and Engineer, integrating human workflow efficiencies into collaboration.
  • Software-engineering agents: SWE-agent resolved 12.5% of issues on SWE-bench while OpenDevin CodeAct 1.0 reportedly set a state-of-the-art result on SWE-Bench Lite.
  • Open challenges: Fully realized AI software engineers remain incomplete because prompt design, context length, agent count, and toolsets need optimization as task complexity increases.

5.10 Evaluation

The survey reviews metrics-based, human-centered, and LLM-based evaluation for code generation, highlighting their coverage, limitations, and empirical benchmark comparisons. It reports narrowing open- versus closed-source performance gaps while emphasizing that functional correctness alone is insufficient for holistic assessment.

  • Evaluation strategies: Code-generation evaluation comprises metrics-based, human-centered, and LLM-based approaches.The survey organizes these strategies into three principal categories and discusses their associated benchmarks.
  • Metrics-based evaluation: Token-matching metrics are rapid and inexpensive but often miss syntactic, functional, and semantic correctness.CodeBLEU addresses some limitations by incorporating abstract syntax trees and data-flow graphs.
  • Metrics-based evaluation: Pass@k estimates the probability that at least one of k generated code samples passes all unit tests.The unbiased estimator uses n sampled candidate solutions and c correct samples, with n≥k.
  • Evaluation limitations: Functional-correctness-focused methods do not holistically evaluate vulnerability, maintainability, readability, efficiency, style, or execution stability.Human evaluation can provide relatively robust results, but comprehensive assessment across these dimensions remains open.
  • LLM-based evaluation: LLM-based evaluation remains underexplored, although ICE-Score reportedly correlates better with functional correctness and human preferences without test oracles or references.The survey presents LLM-as-a-judge as an emerging evaluation direction.
  • Empirical comparison: The performance gap between open-source and closed-source models is gradually narrowing across HumanEval, MBPP, and BigCodeBench.Examples include 88.4% pass@1 for Qwen2.5-Coder-Instruct 7B on HumanEval versus 92.0% for Claude-3.5-Sonnet, and 59.7% for DeepSeek-Coder-V2-Instruct on BigCodeBench versus 61.1% for GPT-4o-0513.

5.11 Code LLMs Alignment

The survey frames Code LLM alignment around human intentions, values, and deployment risks. It proposes five objectives—GREST—and connects them to environmental cost, responsibility, efficiency, safety, and trustworthiness concerns.

  • Motivation: Code LLM alignment is especially important because non-expert users may execute generated code that causes serious system damage.Risks include malware infection, data loss, unauthorized access, privacy breaches, and financial or legal consequences.
  • GREST principles: The survey identifies Green, Responsibility, Efficiency, Safety, and Trustworthiness as five core alignment objectives, collectively called GREST.These principles are summarized in the survey’s alignment framework.
  • Green: Green alignment targets energy consumption, carbon emissions, and financial costs during Code LLM training, inference, and deployment.The survey notes that these stages are currently resource-intensive.
  • Efficiency: Efficiency methods include specialized hardware, model compression, parameter-efficient fine-tuning, and parallel strategies.The survey presents these techniques as ways to reduce computational requirements.
  • Safety: Safety requires testing and validation to mitigate vulnerabilities, errors, privacy breaches, and data-poisoning risks.The survey treats malicious training examples as one attack avenue.
  • Trustworthiness: Trustworthy Code LLMs should provide accurate, reliable, transparent, explainable, and consistently performing code generation.The survey links trustworthiness to user dependence and wider adoption.

5.12 Applications

The survey describes Code LLMs embedded in IDEs and development platforms as coding assistants for generation, explanation, translation, repair, and interaction. GitHub Copilot illustrates a workflow from natural-language problem description to generated code accepted by an online judge.

  • Application landscape: Code LLMs are integrated with IDEs and version-control systems to improve programming efficiency.The survey presents these integrations as widely used coding-assistant applications.
  • Other assistants: CodeGeeX supports code completion, comment generation, code translation, and developer interactions, including multilingual code generation.The survey also reports performance on HumanEval, HumanEval-X, and DS1000.
  • Other assistants: Codeium offers completion, explanation, translation, search, and chat across more than 70 programming languages.The survey characterizes it as an AI-accelerated coding toolkit.
  • Other assistants: Replit provides browser-based compilation and execution in more than 50 programming languages alongside collaboration and cloud services.It is described as a multifunctional online IDE and developer community platform.
  • GitHub Copilot: GitHub Copilot uses chat and inline interactions to plan solutions, write code, explain code, and refactor source code.The example uses GPT-4o, o1-preview, and o1-mini capabilities within Visual Studio Code.
  • GitHub Copilot: In the Copilot example, code generated for LeetCode’s Longest Palindromic Substring problem is accepted by an online judge.The workflow demonstrates generation followed by external execution-based validation.

6 CHALLENGES & OPPORTUNITIES

The survey identifies challenges spanning practical generalization, model architecture, data, evaluation, language coverage, adaptation, and safety. It also highlights opportunities to improve code LLMs through richer data, code-structured architectures, comprehensive benchmarks, continual learning, and stronger alignment and verification.

  • Practical Development: LLMs often struggle with unseen programming problems and repository- or software-level tasks beyond function-level code generation.These challenges are common in real-world software development scenarios.
  • Model Architectures: Transformer-based architectures may not optimally capture programming languages' rigid syntax and inherent structure.The survey calls for architectures tuned specifically to code structures.
  • Data Curation: High-quality, diverse datasets remain scarce, constraining generalization across unseen tasks, coding environments, and real-world software development.The survey points to repository mining, filtering, synthesis, and industry collaboration as data-acquisition opportunities.
  • Evaluation: Current benchmarks and metrics do not fully capture practical coding proficiency, including efficiency, style, readability, and maintainability.The survey advocates comprehensive evaluations that simulate real-world software development.
  • Coverage and Adaptation: LLMs need broader support for low-resource, low-level, and domain-specific languages, plus continuous learning for evolving programming knowledge.Retrieval-augmented code generation can mitigate outdated knowledge, but its performance depends on retrieval-context quality.
  • Safety and Alignment: Generated code may contain vulnerabilities or violate desired norms, motivating formal verification and alignment learning that reflects human ethical preferences.Transparent and explainable AI can also make decision-making more accessible to developers and support trust in generated code.

7 CONCLUSION

The survey systematically reviews LLMs for code generation across data, advances, evaluation, ethics, environmental impact, and applications. It also traces the field's evolution, compares established benchmarks, identifies academia–industry challenges, and provides a continuously updated resource website.

  • The survey covers data curation, recent advances, performance evaluation, ethical implications, environmental impact, and real-world applications.
  • It presents a historical overview and empirical comparisons using HumanEval, MBPP, and BigCodeBench.
  • The survey identifies challenges and opportunities related to the gap between academic research and practical development.
  • A dedicated resource website continuously documents and disseminates recent advances in LLMs for code generation.
Loading 2406.00515v2…