A Survey on Large Language Models for Code Generation

• The survey adopts NL2Code as its consistent definition of code generation and distinguishes it from other code tasks, while also covering code completion in Sections 5.7 and 5.8.Source: §1 · p. 2, Table 1 · p. 3
• HumanEval Pass@1 rose from 3.6% for PaLM 8B [54] to 95.1% for LDB [325] as Transformer-based LLMs supplanted rigid, hard-to-scale earlier approaches.Instruction-following capabilities let novice programmers generate code by articulating requirements, broadening coding accessibility.Source: §1 · p. 2
• A relevant NL2Code review [16] focused on models released from 2020 to 2022, leaving newer models and advanced code-generation topics insufficiently reviewed.Source: §1 · pp. 2–3
• The survey offers a systematic reference and lifecycle-aligned taxonomy spanning data curation, advanced topics, evaluation, and applications, and identifies challenges and opportunities to bridge research and practice.Source: §1 · p. 3

2.1 Large Language Models

• LLMs benefit from scaling, but larger models can exhibit emergent abilities absent in smaller models; neither superiority nor emergence is universal.Examples include instruction following, in-context learning, and step-by-step reasoning.Source: §2.1 · p. 4, §2.1.1 · p. 4
• Code LLMs share Transformer architectures but train mainly on code; encoder-only models suit comprehension, decoder-only generation, and encoder-decoder models support both without necessarily outperforming either.Some models also incorporate synthetic training data.Source: §2.1 · p. 4

2.1.1 Multi-Head Self-Attention Modules.

• Multi-head self-attention relates tokens across latent representation spaces; decoder masking restricts attention to prior and current tokens, while cross-layer attention uses encoder keys and values.Causal attention is another name for masked self-attention.Source: §2.1.1 · pp. 4, 6

2.1.2 Position-wise Feed-Forward Networks.

• A position-wise feed-forward network follows self-attention, applying two linear transformations with an intervening ReLU independently and identically at each sequence position.Its parameters are shared across positions and individually initialized across layers.Source: §2.1.2 · p. 6

2.1.3 Residual Connection and Normalization.

• Transformer layers use residual connections and Layer Normalization to address vanishing or exploding gradients; normalization may follow residual addition (Post-Norm) or precede each sub-layer (Pre-Norm).The passage associates Pre-Norm with GPT-2 and Post-Norm with the vanilla Transformer.Source: §2.1.3 · p. 7

2.1.4 Positional Encoding.

• Transformers add sinusoidal absolute positional embeddings to token embeddings, with learnable, relative, RoPE, and ALiBi alternatives; encoder-decoder maps domains, while decoder-only supports sequence continuation.Encoder-decoder models encode the input into representations used by the decoder to produce output.Source: §2.1.4 · pp. 7–8, §2.1.5 · p. 8

2.2 Code Generation

• Code generation maps natural-language descriptions, optionally with programming context and exemplars, to code; decoding may be deterministic (greedy or beam) or sampling-based (temperature, top-k, or top-p).With no exemplars, the process is zero-shot generation.Source: §2.2 · p. 8
• Generated code is executed against unit tests, and failures can provide feedback used iteratively to refine the solution.The execution may use a compiler or interpreter.Source: §2.2 · pp. 8–9

3.1 Research Questions

• The review’s three research questions address a taxonomy of advances, broad insights into the field, and challenges and opportunities for bridging research and practice.RQ2 covers data curation, advances, evaluation, ethical and environmental implications, applications, historical development, and benchmark comparisons; RQ3 notes that HumanEval may not reflect practical development scenarios [68] [123] [162].Source: §3.1 · pp. 9–10

3.2 Search Process

• The search process began with manual review of top-tier LLM and software-engineering venues, using relevant studies to derive search strings for automated database searches.The keyword list is presented in Table 3.Source: §3.2.1 · p. 10, Table 3 · p. 10

3.2.2 Search Databases.

• Automated searches of four databases targeted titles containing LLM and code-generation keywords, then content filtering excluded out-of-scope work and scripts deduplicated results.The survey defines code generation as natural-language-to-code [16] [17] and excludes code completion and translation unless relevant to Sections 5.7 and 5.8; database retrieval yielded 294 arXiv, 73 ACM, 36 IEEE, and 261 DBLP papers.Source: §3.2.2 · pp. 10–11

3.3 Inclusion and Exclusion Criteria

• Inclusion and exclusion criteria screened papers for relevance to LLM code generation, and two authors independently assessed eligibility, with a third resolving disagreements.Eligible work included datasets or benchmarks, LLM techniques or evaluations, ethical or environmental studies, and applications; exclusions covered non-English or non-peer-reviewed sources (except arXiv), duplicates, non-source-code tasks, non-LLM methods, encoder-only models, and work mentioning LLMs only incidentally.Source: §3.3 · pp. 11–12

3.4 Quality Assessment

• Ten quality-assessment criteria evaluated relevance, clarity, validity, and significance, retaining published studies scoring at least 11.2/14 and arXiv studies at least 9.6/12.A study scoring -1 on all first three criteria was excluded before detailed content assessment; the remaining seven criteria used scores from -1 to 2.Source: §3.4 · p. 12

3.5 Snowballing Search

• The authors used backward snowballing only and found no additional studies, which they attributed as a possibility to the task-specific literature’s earlier publication history.They considered the extensive manual and automated searches likely to have covered the relevant literature comprehensively.Source: §3.5 · p. 12

3.6 Data Collection and Analysis

• After automated retrieval of 664 papers, automatic filtering left 351, inclusion/exclusion review left 247, quality assessment left 235, and snowballing kept the collection at 235 papers on LLMs for code generation.Source: §3.6 · p. 13

4 TAXONOMY

• Code-generation models increasingly arise through adapting general-purpose LLMs or developing models specifically for code, including StarCoder [147], OctoCoder [187], and CodeGen [193].Examples of adaptation include LLaMA [252] followed by Code Llama [227], and DeepSeek LLM [26] extended into DeepSeek Coder [88].Source: §4 · p. 14
• The taxonomy categorizes and evaluates the latest advances in code LLMs, based on an analysis of selected papers from widely used scientific databases.It is depicted in Figure 6 and intended to help researchers familiarize themselves with the field.Source: §4 · p. 14
• Subsequent category analyses will define each problem, discuss its challenges, and compare prominent models and their performance.The recent-advances category emphasizes the core techniques used in current state-of-the-art code LLMs.Source: §4 · p. 14

5 LARGE LANGAUGE MODELS FOR CODE GENERATION

• The Code LLM workflow spans data curation and synthesis, pre-training and post-training, feedback-based reinforcement learning, prompting, inference, and evaluation.Figure 5 distinguishes pre-training from post-training stages, notes that some stages are optional, and shows that a pre-trained model can be used through prompting.Source: §5 · pp. 14–15, Figure 5 · p. 15

5.1 Data Curation & Processing

• Raw code corpora require rigorous cleaning to remove duplicates, noisy or auto-generated files, and personal information that could expose contributors’ names or email addresses.Repository stars and Stack Overflow votes can also help identify higher-quality code and answers.Source: §5.1 · p. 17

5.1.1 Pre-training.

• Unlabeled pre-training data helps models learn code structure, identifier semantics, and sequence logic, using corpora such as CodeSearchNet [110] and The Stack [132].The surveyed corpora range from CodeSearchNet’s 2 million comment–code pairs to The Stack’s more than 6TB of source code.Source: §5.1.1 · pp. 17–18

5.1.2 Instruction Tuning.

• Instruction tuning uses supervised fine-tuning on instruction-structured data to improve instruction following and generalization to unseen tasks, including code-generation tasks.Code-specific datasets include synthetic instruction-following data, filtered commit messages, and datasets with more complex code instructions.Source: §5.1.2 · pp. 18–19

5.1.3 Benchmarks.

• Code-generation benchmarks span general-purpose, competitive-programming, data-science, multilingual, logical-reasoning, and repository-level evaluation contexts.The survey groups logical reasoning with math-related benchmarks that test code-based solutions to complex mathematical problems.Source: §5.1.3 · pp. 19–20
• Benchmark designs extend beyond standalone problems: HumanEval+ [162] expands test cases, BigCodeBench [333] tests complex cross-library tasks, and SWE-bench [123] evaluates GitHub issue resolution.HumanEval+ increases the original test-case scale by 80 times; BigCodeBench covers 1,140 tasks involving 723 function calls from 139 libraries.Source: §5.1.3 · pp. 21–22
• Other benchmarks target data-science code, multilingual generation, and mathematical reasoning, including DS-1000 [136], MultiPL-E [39], and GSM-HARD [79].GSM-HARD increases GSM8K’s difficulty by replacing its numbers with larger, less common values.Source: §5.1.3 · pp. 21–22
• Repository-level benchmarks assess code completion and repository context, while SWE-bench [123] tests whether models can solve GitHub issues.RepoBench [167] includes retrieval, completion, and pipeline tasks; SWE-bench contains 2,294 issue–pull request pairs from 12 Python repositories.Source: §5.1.3 · p. 22

5.2 Data Synthesis

• LIMA performed on par with or better than GPT-4 in 43% of evaluated cases, Bard in 58%, and DaVinci003 in 65% using 1,000 curated examples.QuRating-selected data for a 1.3B-parameter model reduced perplexity and strengthened in-context learning compared with baseline models [280].Source: §5.2 · p. 24
• Alpaca’s 52k instruction examples yielded performance comparable to text-davinci-003, while Phi models trained predominantly on synthetic high-quality data outperformed larger counterparts across benchmarks [84] [150] [4].Self-Instruct generates instructions, inputs, and outputs, then removes invalid or redundant entries before fine-tuning.Source: §5.2 · pp. 24–25
• MagicoderS-CL-7B achieved 66.5% pass@1 on HumanEval Plus, versus ChatGPT’s 65.9%, using OSS-Instruct data [278].The model was fine-tuned on 75k synthetic instructions generated with gpt-3.5-turbo-1106.Source: §5.2 · p. 25
• WizardCoder 15B achieved 57.3% pass@1 on HumanEval, surpassing other open-source Code LLMs and outperforming Claude and Bard [173].Its 78k evolved code-instruction examples were built from Code Alpaca’s initial 20k examples.Source: §5.2 · p. 25
• Phi-1 achieved 50.6% pass@1 on HumanEval and 55.5% on MBPP, setting a state-of-the-art among small language models for Python coding [84].Its training used 6 billion tokens of web-sourced textbook data and 1 billion tokens of synthetic textbooks and exercises.Source: §5.2 · p. 25
• Code Alpaca uses Self-Instruct, WizardCoder Evol-Instruct, and Magicoder OSS-Instruct; StarCoder2-15B-instruct generates from filtered Stack v1 seed functions and self-validates responses [173] [278].StarCoder2-15B-instruct uses its own base model as the data generator, avoiding reliance on large proprietary models such as GPT-3.5-turbo.Source: Figure 8 · p. 24, §5.2 · p. 25
• Synthetic instruction data addresses human-data scarcity, privacy concerns, and production costs [165], but can lack diversity, factuality, or fidelity and amplify or introduce biases [280] [89].The section describes GPT-3.5-turbo and GPT-4 as enabling rich synthetic datasets without human annotation [5] [165].Source: §5.2 · pp. 24–25

5.3 Pre-Training

• Transformers became LLMs’ de facto backbone for parallel computation and representational capacity [298] [319]; MoE [35] [233] and DUS [130] scale models to billions or trillions of parameters, with emergent abilities [97] [127].Source: §5.3.1 · pp. 25–26

5.3.1 Model Architectures.

• Code-generation LLMs use encoder-decoder architectures such as CodeT5, CodeT5+ [269], and CodeRL [139], or decoder-only models such as Codex [48], StarCoder [147], Code Llama [227], and CodeGemma.Source: §5.3.1 · p. 26
• Because annotated scratch training is costly and data-limited, especially for low-resource languages, code LLMs shift to unlabelled code pre-training with CLM, DAE, and auxiliary objectives [38] [46] [48].Auxiliary tasks include masked identifier prediction, identifier tagging, bimodal dual generation, text-code matching, and text-code contrastive learning.Source: §5.3.2 · p. 26

5.3.2 Pre-training Tasks.

• CLM predicts each token autoregressively from preceding tokens, using causal attention masks; encoder-decoder CLM splits a sequence at a pivot token into encoder context and decoder targets.Pre-trained models can generate code zero-shot through prompt engineering [33], while few-shot in-context learning has also been explored [145] [206].Source: §5.3.2 · pp. 27–28
• DAE corrupts encoder input with randomly masked spans, then trains an autoregressive decoder to reconstruct them with sentinel tokens, supporting semantically and syntactically accurate code generation [7].Compared with CLM, DAE requires deeper capture of semantic relationships among token sequences [217].Source: §5.3.2 · p. 28

5.4 Instruction Tuning

• Instruction tuning supervises pretrained LLMs on varied natural-language instructions, treating the process as multi-task prompted training that improves generalization to unseen tasks [114] [274] [313].Source: §5.4 · p. 29
• Code instruction datasets typically come from permissively licensed source code or synthetic code data, while full-parameter fine-tuning can demand substantial compute, motivating PEFT [110] [132] [170] [173] [278] [67] [153].Source: §5.4 · p. 29

5.4.1 Full Parameter Fine-tuning.

• Full-parameter fine-tuning updates every model parameter and is often favored with ample compute and training data; 20k Code Alpaca samples produced instruction-following InstructCodeT5+ with improved code-generation capabilities.Source: §5.4.1 · p. 30
• Evol-Instruct expanded 20K Code Alpaca samples to 78K for WizardCoder, while Pangu-Coder 2 used 68k samples and RRTF to further enhance performance [173] [234].Source: §5.4.1 · p. 30
• OctoPack curated 4 terabytes of Git commits across 350 programming languages as CommitPack, fine-tuning StarCoder and CodeGeeX2 into OctoCoder and OctoGeeX [187] [147].Source: §5.4.1 · p. 30
• OSS-INSTRUCT uses open-source snippets to reduce bias in synthetic data; StarCoder2-15B-instruct is described as fully self-aligned, while CodeGemma 7B was tuned with math datasets and synthetic code [278] [95] [58].Source: §5.4.1 · p. 31

5.4.2 Parameter-Efficient Fine-tuning.

• PEFT updates only a subset of parameters or adds trainable ones; LoRA freezes original weights and trains two low-rank matrices whose product is added to a pretrained weight matrix [67, 153, 103].Source: §5.4.2 · p. 31
• Code-generation PEFT remains limited: CodeUp pioneered single-RTX-3090 parameter-efficient instruction tuning on Llama 2, and ASTRAIOS empirically examined PEFT for code comprehension and generation [253] [334].Source: §5.4.2 · p. 31

5.5 Reinforcement Learning with Feedback

• RLHF aligns instruction-tuned models using human-ranked outputs, but preference rankings are resource-intensive; RLAIF uses off-the-shelf LLMs to generate preference data instead.The survey notes RLHF’s role in developing conversational models such as ChatGPT and Bard.Source: §5.5 · pp. 31–32
• Compiler and interpreter feedback supports code-generation reinforcement learning: CodeRL predicts token-level correctness, CompCoder targets compilability, PPOCoder combines execution and structural rewards, and RLTF uses fine-grained adaptive feedback.PPOCoder also applies a KL-divergence penalty to keep the learned policy close to its pretrained reference model.Source: §5.5 · p. 32
• 62.20% pass@1 on HumanEval was reported for PanGu-Coder 2’s RRTF, while non-PPO methods address reinforcement learning’s inefficiency, instability, resource demands, and tuning complexity.RRTF is inspired by RRHF; the survey identifies coding style and readability as potential future feedback features.Source: §5.5 · p. 33

5.6 Prompting Engineering

• Prompting engineering avoids additional training and includes reasoning techniques such as CoT and self-improvement methods that refine code through explanations, execution, or debugging feedback.Self-Debugging can use code explanations alone when unit tests are unavailable; SelfEvolve first generates domain-specific knowledge and trial code.Source: §5.6 · p. 33
• 94.4% pass@1 on HumanEval was achieved by GPT-4 using LATS, which builds trajectories with Monte Carlo Tree Search and learns from external feedback.LATS uses LLMs as agents, value functions, and optimizers.Source: §5.6 · pp. 33–34

5.7 Repository Level & Long Context

• Repository-level generation is difficult because dependencies span files, repositories have distinct structures and styles, context can exceed model limits, and training data may omit proprietary or developing projects.These limitations contribute to LLMs’ weaker repository-level performance compared with function-level generation.Source: §5.7 · pp. 34–35
• Repository methods retrieve and combine relevant context, selectively skip redundant retrieval, or plan multi-step edits; CodeS decomposes natural-language repository creation into sketches and function filling.Repoformer self-assesses retrieval utility to avoid potentially noisy contexts; CodePlan plans edits using repository-wide context and preceding modifications.Source: §5.7 · p. 35
• Repository-level benchmarks include RepoEval, Stack-Repo, Repobench, EvoCodeBench, SWE-bench, CrossCodeEval, and SketchEval, while coding agents show potential for complex problems amid persistent editing and debugging burdens.The survey points to AutoCodeRover, SWE-Agent, and OpenDevin as examples of LLM-powered coding agents.Source: §5.7 · p. 35

• RAG addresses hallucination, stale knowledge, and opaque reasoning by retrieving external information and combining it with the query, avoiding retraining for updates [80].Its usual components are a vector database, retriever, re-ranker, and generator; instruction-tuning or feedback can bring forgetting and training-compute costs, while RAG has advanced tasks [47] [143].Source: §5.8 · p. 36
• Code LLMs may hallucinate when prompts require unseen data or current packages; stale APIs can cause errors and safety risks, motivating RACG.Public code libraries such as PyTorch change frequently, making deprecated calling methods a challenge.Source: §5.8 · pp. 36–37
• Retrieval-augmented methods include HGNN for code summarization [166], REDCODER for code generation, ReACC for code completion [171], and DocPrompting.ReACC combines lexical copying with semantic referencing and achieves state-of-the-art performance on CodeXGLUE [172]; DocPrompting retrieves documentation relevant to a natural-language query.Source: §5.8 · p. 37
• RepoCoder uses iterative retrieval and cross-file code analogies for repository-level completion, while [242] combines multiple knowledge sources with active retrieval.The multi-source “knowledge soup” includes web searches, documentation, execution feedback, and evolved code snippets; active retrieval iteratively refines the query.Source: §5.8 · p. 37
• Retrieval-augmented code generation remains limited by retrieved-information quality, integration with the query, intent mismatch from over-reliance, and increased context-window compute demands.Source: §5.8 · p. 37

5.9 Autonomous Coding Agents

• An autonomous agent uses an LLM as its controller to plan and execute tasks through tools or APIs, with memory, planning, and tool-use components supporting interaction in a shared environment.The survey names AutoGPT, AgentGPT, BabyAGI, and AutoGen [283] as examples of the expanding application of LLM-based autonomous agents.Source: §5.9 · p. 37, Figure 14 · p. 38
• 96.3% pass@1 on HumanEval was achieved by AgentCoder [104]; SWE-agent resolved 12.5% of SWE-bench issues, and OpenDevin CodeAct 1.0 was reported state-of-the-art on SWE-Bench Lite [123].Devin [61] and OpenDevin [199] also drew interest, while L2MAC was introduced for generating long, consistent outputs using a von Neumann architecture.Source: §5.9 · pp. 37, 39
• AgentCoder [104] divides code generation, unit-test design, and execution with feedback among three specialized agents; MetaGPT [100] assigns code-related procedures to role-based LLM collaborators.CodeAct [265] uses executable Python as a unified action space, while AutoCodeRover [316] autonomously resolves GitHub issues.Source: §5.9 · pp. 38–39, Figure 15 · p. 38
• Prompt design, context length, agent count, and toolsets still require optimization as software-engineering tasks become more complex.The survey says fully realized AI software engineers using LLM-powered agents remain a work in progress [284].Source: §5.9 · p. 39

5.10 Evaluation

• Evaluation strategies for code-generation LLMs fall into metrics-based, human-centered, and LLM-based categories.The survey directs readers to Section 5.1.3 and Table 6 for detailed benchmarks.Source: §5.10 · p. 39

5.10.1 Metrics.

• Token-matching metrics are fast but miss code syntax, function, and semantics; execution metrics such as pass@k assess whether at least one of k samples passes all unit tests.CodeBLEU adds AST syntax and DFG semantics to BLEU but does not resolve execution errors. Execution metrics depend on unit-test quality and executable code; when tests or ground-truth labels are unavailable, token-matching metrics or unsupervised perplexity can be used. Current metrics lack holistic coverage of vulnerability, maintainability, readability, complexity, efficiency, style, and execution stability, especially for repository-scale code with cross-file dependencies.Source: §5.10.1 · pp. 39, 41, §5.10.2 · p. 41

5.10.4 Empirical Comparison.

• Open–closed gaps narrow: HumanEval pass@1 reaches 90% and 88.4% for DeepSeek-Coder-V2-Instruct and Qwen2.5-Coder-Instruct, versus Claude-3.5-Sonnet’s 92.0%.On MBPP, Qwen2.5-Coder-Instruct reaches 83.5% pass@1 versus GPT-3.5-Turbo’s 52.2% and Claude-3-Opus’s 86.4%; on BigCodeBench, DeepSeek-Coder-V2-Instruct scores 59.7%, just below GPT-4o-0513’s 61.1%. The comparison uses pass@1 and draws on original-paper results or recognized leaderboards because of computational-resource limits.Source: Table 9 · p. 40, Figure 17 · p. 43, Figure 18 · p. 44

5.10.2 Human Evaluation.

• Human evaluation can provide robust, adaptable assessments and validate metrics such as CodeBLEU, but evaluator bias, cost, and poor reproducibility limit it.Code-generation alignment methods cited in the survey typically use compiler or interpreter execution feedback rather than human evaluators.Source: §5.10.2 · pp. 41–42

5.10.3 LLM-as-a-Judge.

• LLM-as-a-Judge can scale evaluation and explain its ratings; ICE-Score reports stronger correlations with functional correctness and human preferences without test oracles or references.LLM judges remain vulnerable to position, verbosity, and self-enhancement biases and limited reasoning; prompt engineering and fine-tuning may partly address these shortcomings.Source: §5.10.3 · pp. 42–43

5.10.4 Empirical Comparison.

• Instruction-tuned models outperform their pretrained counterparts on HumanEval and MBPP, with average gains including 26.04% for Qwen2.5-Coder-Instruct and 35.20% for StarCoder2-Instruct.The survey reports gains of 11.26% for CodeGemma-Instruct, 23.71% for DeepSeek-Coder-Instruct, and 13.80% for Code Llama-Instruct; it notes that instruction-tuning data quality matters. It also finds HumanEval nearly saturated, while MBPP and the more practical, challenging BigCodeBench still demand more capable models and broader evaluation.Source: §5.10.4 · pp. 44–45, Table 10 · p. 45

GREST

• GREST defines five Code LLM alignment principles: Green, Responsibility, Efficiency, Safety, and Trustworthiness.The survey presents these principles as broad objectives encompassing multiple concepts and properties.Source: §5.11 · p. 46, Table 11 · p. 47

Green

• 160 times smaller models from distilling CodeBERT [76] and GraphCodeBERT [86] reduced energy consumption by up to 184 times and carbon footprint by up to 157 times.Green alignment targets energy use, carbon emissions, and financial costs across training and inference; quantization of CodeGen [193] and Incoder [77] reduced storage by 67.3% to 70.8%, carbon footprint by 28.8% to 55.0%, and pricing costs by 28.9% to 55.0%.Source: §5.11 · pp. 46–47

Responsibility

• Responsibility addresses bias, fairness, transparency, legal compliance, accountability, misuse prevention, and clear communication about model capabilities and limitations.Suggested mitigations include bias detection and mitigation, evaluation, ethical guidelines, and diverse stakeholder involvement.Source: §5.11 · pp. 47–48

Efficiency

• Efficiency focuses on improving code-generation performance and speed while reducing computational resources through compression, optimized algorithms, parallelism, and parameter-efficient fine-tuning.The survey cites GPT-3 training as requiring approximately 1,024 NVIDIA V100 GPUs and approximately 34 days.Source: §5.11 · pp. 48–49

Safety

• Safety requires testing for vulnerabilities and privacy breaches, with defenses including private-data removal, prompt filtering, adversarial training, and reinforcement learning.Risks discussed include data poisoning, black-box prompting that elicits vulnerable code, memorized personal information, and safety-alignment circumvention through non-natural-language interactions.Source: §5.11 · p. 49

Trustworthiness

• Trustworthiness aims for accurate, reliable, transparent code generation and user understanding through explanations and consistent performance.Ji et al. [120] use causal graphs linking prompts and generated code, while Palacio et al. [202] present ASTxplainer, which visualizes token predictions aligned with AST nodes.Source: §5.11 · p. 49

GitHub Copilot

• Copilot suggests code from context and public GitHub code, and can also help programmers learn through implementation examples.The paper describes it as an AI pair programmer powered by OpenAI’s Codex.Source: §5.12 · pp. 49, 51
• Copilot generated an accepted solution to LeetCode’s “Longest Palindromic Substring” problem, demonstrating its coding workflow.The example uses Copilot’s chat and workspace features in VS Code; the generated code was submitted to an online judge.Source: Figure 19 · p. 51, §5.12 · pp. 52–53

CodeGeeX

• CodeGeeX supports completion, comment generation, translation, and developer interaction, with multilingual code generation.Its underlying model was trained on extensive code data and is reported to perform strongly on HumanEval, HumanEval-X, and DS1000.Source: §5.12 · p. 52

CodeWhisperer

• CodeWhisperer provides on-the-fly, personalized recommendations tailored to coding patterns and comments, from short comments to complex functions.Source: §5.12 · p. 52

Codeium

• Codeium combines completion, explanation, translation, search, and chat, and supports more than 70 programming languages.Source: §5.12 · p. 52

CodeArts Snap

• CodeArts Snap generates function-level code from Chinese or English descriptions and can generate tests, analyze code, and repair it.Source: §5.12 · p. 52

Tabnine

• Tabnine offers team-tailored AI automation with stated privacy, security, and compliance standards, and supports more than one million developers.The paper presents it as a tool for improving coding efficiency, code quality, and developer satisfaction.Source: §5.12 · p. 52

Replit

• Replit combines an online collaborative IDE and cloud services with browser-based compilation and execution in more than 50 programming languages.Source: §5.12 · p. 52

Enhancing complex code generation at repository and software scale

• AlphaCode [151] achieved an average ranking in the top 54.3% in competitions, while Claude 2 [123] solved 1.96% of real-world GitHub issues; weak reasoning, dependencies, and context limits hinder repository-scale work [105] [22].The survey calls for models with problem-solving capabilities beyond function-level code generation.Source: §6 · p. 53

Innovating model architectures tuned to code structures

• Code-tuned architectures could better capture programming structure through AST-based tree networks or compiler intermediate representations that support generalization across languages.The survey argues that conventional Transformer architectures may not be optimally designed for code’s rigid syntax and structure.Source: §6 · p. 53

Curating high-quality code data for pre-training and fine-tuning of LLMs

• Scarce high-quality, diverse code datasets limit generalization; repository mining, filtering, synthesis, industry collaboration, and open-source sharing could expand training data.The proposed data sources aim to cover more tasks, styles, languages, coding environments, and real-world development scenarios.Source: §6 · pp. 53–54

Developing comprehensive benchmarks and metrics for coding proficiency evaluation in LLMs

• Benchmarks should model practical development and varied tasks, while metrics assess efficiency, style, readability, and maintainability alongside correctness.The survey proposes debugging, refactoring, and optimization tasks, with community-driven platforms enabling ongoing comparisons.Source: §6 · p. 54

Support for low-resource, low-level, and domain-specific programming languages

• Low-resource, low-level, and domain-specific languages remain underrepresented; transfer learning, meta-learning, expert partnerships, and multilingual generation could broaden coverage.These approaches could extend knowledge from high-resource languages and support specialized fields and systems programming.Source: §6 · p. 54

Continuous learning for LLMs to keep pace with evolving coding knowledge

• Because retrieval-augmented generation depends on retrieved-context quality, continuous learning could use repository monitoring, incremental updates, and active learning with developers.Incremental systems should incorporate new knowledge without forgetting previously acquired knowledge.Source: §6 · p. 54

Ensuring code safety and aligning LLM outputs with human coding preferences

• Generated code may contain vulnerabilities or violate human preferences; formal verification, alignment learning, and explainability could improve safety and trust.Alignment frameworks are proposed to reflect human ethical preferences and societal values.Source: §6 · pp. 54–55

7 CONCLUSION

• The survey systematically reviews data curation, recent advances, performance evaluation, ethical implications, environmental impact, and real-world applications of LLMs for code generation.It also provides a historical overview of the field’s evolution in recent years.Source: §7 · p. 55
• The survey empirically compares LLMs for code generation on HumanEval, MBPP, and BigCodeBench to highlight progressive capability enhancements.The authors describe BigCodeBench as more practical and challenging.Source: §7 · p. 55
• The survey identifies challenges and promising opportunities around the gap between academia and practical development for future investigation.Source: §7 · p. 55
• The authors established a dedicated resource website to continuously document and disseminate recent advances in LLMs for code generation.Source: §7 · p. 55
• The authors envision LLMs ultimately writing safe, helpful, accurate, trustworthy, and controllable code like professional programmers and solving coding problems humans cannot currently solve.Source: §7 · p. 55