Source-linked AI summary
Large Language Models Meet NL2Code: A Survey
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, Jian-Guang Lou
TL;DR
NL2Code asks how effectively language models can generate code from natural-language requirements, an important challenge as LLMs have advanced. This paper surveys 27 NL2Code LLMs, benchmarks, and metrics, compares models on HumanEval, and analyzes their success factors. It identifies Large Size, Premium Data, and Expert Tuning while discussing the remaining ability gap between LLMs and humans.
Problem
NL2Code seeks to generate code from natural-language descriptions, but the field needs a comprehensive account of its rapidly expanding models, benchmarks, metrics, and remaining human–model ability gap.
Method
The paper surveys 27 NL2Code LLMs, reviews benchmarks and metrics, compares models on HumanEval, and analyzes their design choices and success factors.
Results
The analysis concludes that Large Size, Premium Data, and Expert Tuning are key factors associated with LLM success in NL2Code.
Takeaways & Limitations
The survey provides an overview of NL2Code progress, identifies success factors, and highlights challenges and opportunities concerning the ability gap between LLMs and humans.
Takeaways & Limitations
The survey may omit aspects or newer works because NL2Code is evolving rapidly, and some models are not publicly available for comparison.
Abstract
from arXiv · showhide
The task of generating code from a natural language description, or NL2Code, is considered a pressing and significant challenge in code intelligence. Thanks to the rapid development of pre-training techniques, surging large language models are being proposed for code, sparking the advances in NL2Code. To facilitate further research and applications in this field, in this paper, we present a comprehensive survey of 27 existing large language models for NL2Code, and also review benchmarks and metrics. We provide an intuitive comparison of all existing models on the HumanEval benchmark. Through in-depth observation and analysis, we provide some insights and conclude that the key factors contributing to the success of large language models for NL2Code are "Large Size, Premium Data, Expert Tuning". In addition, we discuss challenges and opportunities regarding the gap between models and humans. We also create a website https://nl2code.github.io to track the latest progress through crowd-sourcing. To the best of our knowledge, this is the first survey of large language models for NL2Code, and we believe it will contribute to the ongoing development of the field.
1 Introduction
NL2Code asks whether software can be generated from natural-language requirements, a longstanding challenge with potential effects across society. This survey reviews the rapidly expanding LLM-based field and identifies factors associated with success.
- Motivation: NL2Code aims to generate code from natural-language descriptions, raising whether novices can create software without programming experience.The question has implications for software engineering, programming languages, artificial intelligence, education, the economy, and the labor market.
- Background: Early NL2Code approaches based on rules, expert systems, domain-specific languages, n-grams, and Hidden Markov models were inflexible, sparse, or unable to model long-term dependencies.These limitations motivated later approaches using large Transformer-based language models trained on code corpora.
- Survey Scope: The survey investigates 27 advanced NL2Code LLMs, reviews benchmarks and metrics, and compares models on HumanEval.It analyzes why models succeed and discusses the ability gap between LLMs and humans.
- Conclusions: The authors attribute LLM success in NL2Code to Large Size, Premium Data, and Expert Tuning, and provide a crowdsourced website tracking field progress.The website is available at https://nl2code.github.io.
2 Large Language Models for NL2Code
NL2Code models generate demanded code from natural-language problem descriptions, typically using Transformer architectures trained on large unlabeled code corpora. The surveyed models span diverse architectures and sizes, with later systems generally becoming larger and decoder-only designs favored at larger scales.
- Task Definition: NL2Code automatically generates demanded code from a natural-language problem description, across varying programming languages and problem domains.Figure 1 illustrates the task with a problem description, predicted solution, and test cases.
- Modeling Approach: Most NL2Code LLMs use Transformer architectures and causal language modeling on large-scale unlabeled code-related corpora.During inference, they can solve problems zero-shot without fine-tuning, while some studies use few-shot or in-context approaches.
- Model Landscape: The survey covers 27 representative LLMs that vary in architecture, size, and accessibility, presenting them chronologically with their largest model sizes.Model sizes consistently grow as the field advances, and decoder-only architectures are favored among larger pretrained models.
- Model Evolution: Early models such as GPT-C, PyMT5, and PLBART had relatively few parameters and weak zero-shot code-generation capabilities.GPT-Neo and GPT-J had billion-level scales but limited NL2Code power because their training corpora contained little code.
- Model Evolution: Codex, AlphaCode, PaLM-Coder, CodeGen-Mono, and PanGu-Coder are described as powerful systems, although many leading models are not readily accessible.InCoder and SantaCoder also achieve decent left-to-right results despite using fill-in-the-middle training.
3 What makes LLMs successful?
The survey attributes successful NL2Code large language models to large size, premium data, and expert tuning. Its comparisons and analyses show that scale generally improves performance, data quality matters, and tuning choices shape outcomes.
- Overview: HumanEval performance varies widely among models with similar sizes, while Codex leads across several size ranges and some smaller models perform comparably.The comparison excludes models that cannot pass any benchmark problem.
- 3.1 Large Model Size: Larger models generally achieve higher pass@1 and lower syntax-error rates, but large scale does not guarantee semantic correctness.CodeGen-Mono with 16 billion parameters has a 6% syntax-error rate yet only 29% pass@1, indicating remaining semantic errors.
- 3.2 Premium Data: Training-corpus scale and quality are important because models with limited code in their training data have not demonstrated exceptional code-generation capabilities.The survey reviews preprocessing practices across Codex, AlphaCode, CodeGen, InCoder, and PyCodeGPT, including removal of likely auto-generated or unfinished code.
- 3.3 Expert Tuning: Expert tuning includes learning rate, batch size, window size, warmup, gradient accumulation, and sampling temperature; learning rates decrease as model size increases.The survey also reports that a small model with a large window size sometimes outperforms a large model with a small window size.
- Benchmarks: The survey summarizes 17 NL2Code benchmarks whose characteristics differ in size, language, complexity, and scenario.The table records instance counts, natural-language and programming-language fields, test-case counts, and description or solution lengths.
4 Benchmarks and Metrics
The survey reviews NL2Code benchmarks and automatic metrics, highlighting variation in benchmark characteristics and unresolved evaluation challenges.
- 17 NL2Code benchmarks differ in size, language, complexity, and scenario.
- HumanEval and MBPP contain 164 and 974 instances, respectively, partly because hand-written benchmarks help prevent training-data leakage.
- Recent multilingual benchmarks expand coverage across programming languages and natural languages beyond predominantly English-description, Python-solution datasets.
- Execution-based metrics evaluate executable code using test cases, whereas BLEU, ROUGE, and CodeBLEU target non-executable code but cannot precisely assess correctness.
- Metric design remains challenging for properties including vulnerability, maintainability, clarity, execution complexity, and stability.
5 Challenges and Opportunities
The survey identifies gaps between LLMs and humans in understanding, judgment, explanation, adaptive learning, and multitasking, and outlines directions for addressing them.
- Understanding Ability: LLMs are sensitive to context and may degrade unexpectedly or struggle with complex problems containing many conditions and requirements.
- Understanding Ability: Breaking complex problems into multiple steps is proposed as one potential direction for improving LLM understanding.
- Judgement Ability: Because causal language modeling makes models always return solutions, they may lack humans’ ability to recognize unsolvable programming problems.
- Judgement Ability: Reinforcement learning from user feedback and self-validation are being explored to improve model judgment, although high-quality feedback is costly and challenging to collect.
- Adaptive Learning Ability: Unlike humans, LLMs require substantial effort and resources for retraining or fine-tuning to adapt to new knowledge and changing APIs.
- Multi-tasking Ability: LLMs can support many code and code-like tasks, but seamless task switching and rapid mastery of multiple programming languages remain limitations compared with humans.
6 Conclusion
The paper surveys LLMs for NL2Code, reviews evaluation resources, examines the model–human ability gap, and provides a mechanism for tracking field developments.
- The survey covers 27 LLMs for NL2Code and analyzes factors underlying their success.
- It reviews benchmarks and metrics and presents challenges and opportunities concerning the gap between models and humans.
- A website tracks the latest findings in the field.
Limitations
The survey’s coverage and comparisons are bounded by rapid field evolution, heterogeneous or inaccessible models, and criteria based on official publications.
- Rapidly evolving research means the survey may overlook some aspects or omit newly published works.
- Differences in model size, architecture, corpus, preprocessing, tokenization, hyperparameters, and training platforms make completely fair comparisons impractical.
- Some models are not publicly available, limiting the fairness and reproducibility of direct comparisons.
- Evaluating LLMs requires substantial computational resources, motivating the public release of generated files.
- Model sizes are surveyed from official papers, so later larger versions are excluded when the original paper reported a smaller maximum size.
B An Online Website
The paper introduces a real-time, crowd-sourced website for tracking NL2Code research and helping users find relevant papers. The surrounding benchmark material identifies MBPP as an evaluation resource with reproduced and sourced results.
- An Online Website: The website collects recent NL2Code research and accepts community updates through GitHub pull requests.
- An Online Website: Fuzzy search and custom tag categories help researchers quickly locate relevant papers.
- An Online Website: The MBPP results table reports reproduced results alongside results taken from Chen et al. (2023).CodeGPT, GPT-CC, and PLBART are omitted because their reported numbers are zero.
C Experimental Setup
The experiments evaluate generated programs with pass@k on HumanEval and MBPP. This metric samples candidate solutions and counts a problem as solved when at least one selected solution passes its tests.
- Experimental Setup: The experiments use pass@k as the evaluation metric on the HumanEval and MBPP benchmarks.
- Experimental Setup: For each problem, n candidate solutions are sampled and k are randomly selected; any passing solution marks the problem solved.
- Experimental Setup: pass@k is the proportion of benchmark problems solved by the sampled code solutions.
- Experimental Setup: The primary metric is chosen because executing test cases provides precise code-accuracy evaluation.Other discussed metrics either originate from pass@k or are considered less precise.
C.2 Implementation Details
The implementation details standardize sampling, generation limits, temperature settings, and post-processing across benchmark experiments. The authors also reproduce selected model results and release code and generated outputs publicly.
- Implementation Details: HumanEval reproduction uses 200 samples, 200 newly generated tokens, top_p 0.95, and the best temperature from 0.1 to 1.0.
- Implementation Details: MBPP experiments use 100 samples, 200 newly generated tokens, top_p 0.95, and temperature 0.8.
- Implementation Details: Both benchmarks terminate sampling when specified code sequences, including new class, function, comment, conditional, or print lines, appear.
- Implementation Details: The authors make their experimental code and generated HumanEval and MBPP results publicly available for reproducibility.CodeT5 770M denotes the causal-language-modeling version used in the experiments.
D Context Window vs. Performance
The section examines how context-window size relates to NL2Code performance alongside model scale. On APPS, a longer context window can make a much smaller GPT-NeoX model comparable to a much larger one.
- Context Window vs. Performance: Prior work claims that context-window size plays a vital role in enhancing LLM performance for NL2Code.
- Context Window vs. Performance: 165M GPT-NeoX with an 8,000-token context window is comparable to 20B GPT-NeoX with a 2,000-token window on APPS.The comparison varies model sizes of 165M and 20B and context windows of 2K, 4K, and 8K.
- Context Window vs. Performance: The APPS observation suggests that context window should be considered when training NL2Code models.