PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications

Nam Huynh, Beiyu Lin

arXiv:2503.01245v2cs.SEcs.LG

TL;DR

Automated code generation traditionally requires substantial coding skills and domain knowledge, while data preprocessing remains time-consuming. This survey organizes limitations, fine-tuning techniques, evaluation methods, and applications of LLMs for code generation.

  • Problem

    Data mining traditionally requires strong coding skills, domain knowledge, and substantial training, while data preprocessing can account for 50–80% of a data scientist’s workload.

  • Method

    The survey reviews LLM limitations and challenges, fine-tuning techniques, evaluation metrics and benchmarks, and applications across code-generation tasks.

  • Results

    The survey consolidates existing knowledge on LLM code generation across challenges, adaptation methods, evaluations, and applications.

  • Takeaways & Limitations

    LLMs can make coding tasks more accessible by automatically generating code and supporting tasks including code generation, completion, search, debugging, and translation.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated their remarkable capabilities in numerous fields. This survey focuses on how LLMs empower users, regardless of their technical background, to use human languages to automatically generate executable code. We begin with understanding LLMs' limitations and challenges in automated code generation. Subsequently, we review various fine-tuning techniques designed to enhance both the performance and adaptability of LLMs in code generation tasks. We then review the existing metrics and benchmarks for evaluations to assess model performance based on fine-tuning techniques. Finally, we explore the applications of LLMs (e.g. CodeLlama, GitHub Copilot, ToolGen) in code generation tasks to illustrate their roles and functionalities. This survey provides a comprehensive overview of LLMs for code generation, helps researchers in diverse fields better understand the current state-of-the-art technologies, and offers the potential of effectively leveraging LLMs for code generation tasks.

1 Introduction

LLMs make code generation more accessible by reducing the time and coding expertise required for data-preprocessing tasks. The survey organizes the field around challenges, fine-tuning, evaluation, and applications.

  • Motivation: Data preprocessing accounts for 50–80% of a data scientist’s workload, but LLM assistance can generate preprocessing code within minutes.The paper illustrates this use case with code for replacing missing values.
  • Motivation: LLMs can help users generate executable code through natural-language instructions, expanding access beyond technically trained programmers.The introduction frames this accessibility as a major potential of automatic code generation.
  • LLM capabilities: Training on code repositories, technical forums, and web coding data helps LLMs interpret comments, function names, and variable names.The survey cites OpenAI Codex as a model trained on billions of source-code examples.
  • Survey scope: The survey covers resource constraints, syntactic and semantic errors, biases, and security risks as key challenges in automatic code generation.It also reviews fine-tuning methods, evaluation metrics and benchmarks, and coding applications.
  • Illustrative examples: The introduction presents data-preprocessing instructions such as removing missing identifiers, imputing prices, and forward-filling quantities as a code-generation example.The example additionally asks the system to save the cleaned dataset and record generation time.

2 Overview: LLMs for Code Generations

LLMs combine transformer architectures with extensive training data to process language and support code generation. The section introduces these foundations and summarizes the reviewed studies.

  • LLM foundations: Transformer networks use self-attention and feed-forward layers to identify relationships within sequential data such as words in sentences.The paper describes stacked layers as a basis for deeper and more powerful models.
  • LLM foundations: Extensive and diverse training data enables LLMs to process tasks across domains including healthcare and education.The paper connects this capability to practical applications built on domain-relevant datasets.
  • Reviewed studies: Table 1 summarizes the studies reviewed in the survey.The supplied caption identifies the table but does not specify its row or column contents.

A. Fine-Tuning on Domain-Specific Datasets

The survey describes the data, model, and application developments supporting LLM-based code generation, including transformer models, prompting approaches, and rapidly advancing coding systems.

  • Fine-tuning and prompting: Prompting approaches reviewed in the survey include chain-of-thought prompting, security-focused prompting, and the ClarifyGPT framework.The supplied passages list these approaches without describing their comparative outcomes.
  • Model architecture: Transformers analyze sequential relationships through self-attention and feed-forward layers, enabling models to process prompts and generate responses.The supplied passages illustrate summarization as one prompt-to-output example.
  • Training foundations: Data preprocessing prepares open-source repositories and other datasets for LLM training by making them clean, standardized, and suitable for selected models.The paper presents preprocessing as an important stage before data enters the model.
  • Applications: LLM code-generation applications expanded from GitHub Copilot in 2021 to tools supporting completion, explanation, transformation, generation, and debugging.The survey also notes Bard’s 2023 coding support across more than 20 programming languages.
  • Recent coding models: OpenAI o1 achieved 92.4% on HumanEval and ranked in the 89th percentile on Codeforces, according to the cited evaluation summary.The passage attributes these results to comparisons with other top-performing models and describes o1’s internal reasoning and reinforcement learning.

3 Limits and Challenges of Using LLMs for Code Generation

LLM code generation faces constraints across resources, correctness, bias, and security. The survey reviews evidence that these problems vary with model size, task complexity, language, and evaluation practice.

  • Resource Constraints: Training and deploying code-generation LLMs requires substantial computational capacity and memory, especially for large models.CodeLlama models range from 7B to 70B parameters; the 70B model was trained on 1 trillion tokens.
  • Resource Constraints: Quantization reduces memory usage but can worsen output quality as numerical precision decreases.OmniQuant achieved perplexity 5.97 at 4-bit precision and 6.65 at 3-bit precision; GPTQ reached 7.36 at 3-bit versus 5.86 at 8-bit.
  • Resource Constraints: Under tight resource limits, smaller CodeLlama models achieved 5–15% performance gains over the 70B model and reached 60% on HumanEval in one-quarter the time.The comparison used HumanEval and MBPP under a small-budget regime capped at 32 normalized FLOP units and 64 wall-time units.
  • Syntactic and Semantic Errors: Semantic errors generally dominate syntactic errors, increasing with benchmark complexity and often reflecting misunderstood requirements, logic failures, or format problems.On APPS+, DeepSeekCoder, Llama-3, Phi-3, and GPT-3.5 each exceeded 50% semantic-error proportions; syntactic errors were below 10% overall.
  • Syntactic and Semantic Errors: Code translation is especially vulnerable to source–target misalignment, with nearly 80% of reported issues arising from such discrepancies.Syntactic and semantic misalignments accounted for 30.5% of translation errors, while unmet target-language requirements accounted for 24.3%.
  • Syntactic and Semantic Errors: Generated programs can contain runtime and maintainability problems that require manual correction.Illegal Index errors comprised 46.4% of 97 Java runtime errors, and 47% of 1,930 snippets exhibited maintainability issues.
  • Biases: LLM code generation exhibits linguistic, programming-language, and social biases that can trade off against generation quality.Chinese instructions reduced Python Pass@1 by 17.2% for base models and 14.3% for instruction-tuned models; Codex achieved Pass@1 47.03% alongside CBS 82.64%.
  • Security Risks: Security risks arise from vulnerable training data, model design, prompting, and inadequate evaluation metrics.Copilot generated insecure code in about 40% of cases, while BLEU and Exact Match were described as insufficient indicators of security and functionality.

4 Fine-Tuning Techniques for Enhancing LLM Performance in Code Generation

Fine-tuning techniques improve LLM code generation through specialized training, data pruning, feedback, reasoning, and prompting strategies. The surveyed approaches target performance, adaptability, requirement understanding, convergence, and code quality.

  • Fine-tuning adapts pretrained LLMs to specialized applications while preserving their remaining knowledge and can improve performance.The survey presents fine-tuning as a strategy for addressing limitations in code generation.
  • LLaMoCo combines a comprehensive instruction set with contrastive-learning warm-up before instruction tuning to improve fine-tuning convergence.
  • Training-data pruning uses clustering and diversity or density metrics to reduce redundancy while maintaining code accuracy and functionality.On HumanEval(+), pruning a small portion of training data improved performance by up to 2.7%.
  • ClarifyGPT checks code consistency, asks targeted questions about ambiguous requirements, and then generates solution code.On MBPP-sanitized and MBPP-ET, it raised GPT-4 performance from 68.02% to 75.75% and ChatGPT performance from 58.55% to 67.22%.
  • Feedback-based methods refine generated code through execution feedback, crowd-sourced human feedback, or Bayesian aggregation of feedback from multiple sources.RLEF uses code execution and PPO, while cRLHF combines diverse feedback into a shared belief without complicated reward modeling.
  • Prompting methods improve code generation by adding solution plans, retrieved examples, constructed prompts, or security-focused prompting strategies.AceCoder improved Pass@1 over prior prompting techniques by up to 56.4% in MBPP, 70.7% in MBJP, and 88.4% in MBJSP.

5 Evaluation Metrics and Benchmarks for Assessing LLM-Generated Code

The survey reviews metrics and benchmarks for evaluating the correctness, similarity, reliability, and practical usefulness of LLM-generated code. It contrasts functional and granular metrics with benchmarks spanning functions, classes, repositories, and complex real-world tasks.

  • Evaluation research asks whether LLM comparisons are fair, statistically meaningful, and transferable across performance-evaluation settings.
  • Metrics quantify properties such as correctness, semantic similarity, and hallucination, while benchmarks combine evaluation datasets, test cases, and metrics.
  • CodeBLEU combines weighted n-gram matching with syntactic AST alignment and semantic matching to evaluate generated code.
  • Metrics: Pass@k measures the probability that at least one of the top k generated samples passes unit tests, with Pass@1, Pass@10, and Pass@100 reflecting different sample counts.The accompanying calculation uses E for expected problems, n for total samples, and k for the top samples considered.
  • Metrics: Pass-ratio@n averages test-case pass rates across n generated solutions and can provide more granular information than pass@k.In one example, pass-ratio@5 scored 61% when no solution passed every test, whereas pass@k scored 0%.
  • Benchmarks: HumanEval evaluates function-level functional correctness, whereas ClassEval targets class-level generation and reports substantially worse performance than standalone method-level benchmarks.ClassEval contains 100 Python classes and 412 methods; GPT-4 and GPT-3.5 performed best, with different generation strategies preferred by different models.
  • Benchmarks: SWE-bench evaluates repository-level software engineering using 2,294 GitHub tasks across 12 Python repositories, while BigCodeBench evaluates 1,140 practical function-level tasks with library and multi-call requirements.Claude 2 solved 1.96% of SWE-bench tasks with the BM25 retriever; BigCodeBench uses Pass@1 with greedy decoding and 99% branch coverage.

6 LLMs’ Applications in Code Generation and Development

LLMs support a broad range of code-generation and development tasks, including completion, code search, advanced programming, debugging, and translation. Applications such as GitHub Copilot, Code Llama, ToolGen, RepoRift, CodeBERT, AlphaCode, GPT-4, and Flourine illustrate different capabilities, architectures, and workflows.

  • 6 LLMs’ Applications in Code Generation and Development: LLMs assist software developers with code completion, translation, and other coding tasks, but model selection remains important because models have different strengths.The survey links suitable model selection with productivity, streamlined processes, reduced errors, and effective workflow integration.
  • 6.1 Code Generation and Completion: GitHub Copilot generates context-sensitive code suggestions using recurrent neural networks and transformers trained to model code structure and developer coding habits.Its development combines GitHub’s software resources with OpenAI’s AI development.
  • 6.1 Code Generation and Completion: Code Llama offers base, Python-specialized, and instruction-tuned models in 7B, 13B, 34B, and 70B parameter sizes for code completion and natural-language instruction following.Each model is trained on 500B code tokens, while the 70B model uses 1T tokens; selected smaller models support fill-in-the-middle generation.
  • 6.1 Code Generation and Completion: ToolGen combines offline trigger insertion, model fine-tuning, and online tool-integrated generation to address repository-level dependency errors such as undefined variables and missing members.It manipulates autocompletion tools to fill repository-level dependencies during code generation.
  • 6.2 Code Search and Advanced Code Generation: CodeBERT uses a Transformer-based bimodal pretraining strategy to understand and generate natural-language and programming-language code, outperforming RoBERTa on CodeSearchNet and documentation generation.On documentation generation across six programming languages, CodeBERT achieved a 1.3 BLEU score gain and state-of-the-art performance.
  • 6.2 Code Search and Advanced Code Generation: AlphaCode targets competitive programming through curated problem data, efficient transformer models, automatic generation and filtering, and large-scale sampling.The survey identifies a high-quality dataset, efficient transformers, and large-scale sampling as critical components of its performance.
  • 6.3 Code Translation and Code Debugging: GPT-4 outperformed Gemini Ultra and Claude 2 in reported programming-code generation comparisons and exceeded a 90% success rate on tasks solved by more than 20% of human participants.Prompt strategies also allowed GPT-4 to salvage over 60% of easy and medium tasks after an initial failure, while supporting code translation.

7 Conclusion

The survey synthesizes the landscape of LLM-based automatic code generation by reviewing its challenges, fine-tuning techniques, evaluations, and applications. It highlights resource constraints, errors, biases, and security risks while presenting approaches such as prompt engineering, reinforcement learning, and domain-specific tuning.

  • 7 Conclusion: The survey covers resource constraints, syntactic and semantic errors, biases, and security risks as key challenges in automatic code generation.These challenges are presented as factors requiring mitigation.
  • 7 Conclusion: It reviews prompt engineering, reinforcement learning, and domain-specific dataset tuning as fine-tuning approaches for addressing challenges in LLM code generation.The survey frames these techniques as approaches for improving code-generation systems.
Loading 2503.01245v2…