L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models

1 Introduction

• Language-to-code generation serves both AI applications and as a testbed for language understanding, reasoning, grounded understanding, and tool use.Examples include dialogue systems, coding assistants, database interfaces, and robotic control.Source: §1 · p. 1
• 7 L2C tasks across semantic parsing, math reasoning, and Python programming are standardized for controlled comparisons among 54 models from 13 organizations.The evaluation spans models from 1 billion to an estimated 170B+ parameters and varied training-data mixtures.Source: §1 · p. 2
• The evaluation studies model-size and training-data scaling alongside instruction tuning and zero- or few-shot prompting.These are identified as factors potentially affecting language-to-code performance.Source: §1 · p. 2
• Robustness and calibration measurements, together with common error cases, are analyzed across models with different capabilities.Source: §1 · p. 2
• L2CEval releases model texts and logits for every dataset to facilitate future studies.Source: §1 · p. 2

2.1 Language-to-Code Generation

• An L2C model maps natural-language intent and optional programming context to a program, which an executor runs to compare its result with the gold execution result.Execution accuracy is used as a proxy for whether the user’s original intent is satisfied.Source: §2.1 · p. 2

2.2 Few-shot Prompting with LLMs

• Execution-based evaluation can produce false positives, thereby overestimating performance.Source: §2.2 · p. 2
• A prompt combines a task-specific instruction, few-shot exemplars, and the test problem’s intent and programming context.The model predicts a program from this prompt; the promptify function concatenates its inputs into a string.Source: §2.2 · p. 3

• Together, the three domains assess natural-language understanding, reasoning through a solution, and conversion of that solution into executable code.Source: §3 · p. 3

Semantic parsing

• Semantic parsing translates natural-language utterances into executable structured programs, and this evaluation uses Spider and WikiTQ with database schemas or table headers as additional input.The selected task is text-to-SQL.Source: §3 · p. 3

Math reasoning

• Math reasoning tests abstraction of mathematical relations and multistep numerical and logical reasoning using GSM8k and SVAMP word problems answered with Python programs.GSM8k contains about 8K problems and SVAMP contains 1K examples.Source: §3 · p. 4

Python programming

• Python programming evaluates whether models generate functional code that implements a developer’s natural-language intent, using MBPP, HumanEval, and DS-1000.Source: §3 · p. 4

4.1 Model Selection

• The evaluation includes 54 models varying in size, training-data mixture, context length, architecture, and training methods.The selection spans models from 1B to 170B+ parameters, prioritizes code-specific models, and includes general and instruction-tuned models.Source: §4 · p. 4, §4.1 · p. 5

4 Models

• Table 3 presents the top three models in different size ranges, using head-to-head task comparisons to calculate mean win rate across tasks.Source: Table 3 · p. 5

4.2 Model Access

• Open-source models are run locally, while proprietary OpenAI models are accessed through an API; analysis primarily focuses on open-source models because proprietary technical details are unclear.Source: §4.2 · p. 5

4.3 Evaluation Details

• All models use greedy decoding, standardized prompts not tailored to particular models, and execution accuracy across tasks.Source: §4.3 · p. 5

5 Results and Analysis

• Although prior work reported degenerated outputs with greedy decoding, the authors did not observe this during human inspection.The paper uses greedy decoding to generate one program per example.Source: §5 · p. 5

5.1 Scaling

• Model-size scaling differs by task: curves are smoother on MBPP, while GSM8k’s reasoning-heavy curve appears more emergent.Mean win rate is defined as the fraction of models a model outperforms, averaged across five tasks.Source: §5.1 · p. 6

5.2 Data Mixture

• Code-specific models typically outperform other models across size groups, including on reasoning and grounded-language tasks; code-token training also appears more compute-efficient for L2C.The compute-efficiency pattern has exceptions, including CodeGen-multi models initialized from general language models.Source: Figure 2 · p. 6, §5.2 · pp. 6–7

5.3 Instruction-tuning

• Instruction-tuned models achieve much higher execution rates, especially zero-shot, and improve performance in most settings; selected GSM8k zero-shot models all score zero after failing answer formatting.For GSM8k, the outputs fail to provide the answer by ending the Python solution with `answer = x`.Source: Table 4 · p. 7, §5.3 · p. 7

5.4 Sensitivity to Prompt

• More few-shot exemplars generally raise execution accuracy, but not consistently; on MBPP, moving from 3 to 8 exemplars lowered codex-cushman by 4.0%.The authors hypothesize that prompt programs can bias models toward similar programs and away from the specification.Source: §5.4 · p. 7, Figure 3 · p. 8

5.5 Error Modes

• On GSM8k, StarCoder and code-cushman make more intermediate-step and deformed-program errors than stronger models; weaker MBPP models often miss crucial implementation steps.Extraneous code lines are rare in the analyzed code-generation outputs.Source: Figure 4 · p. 8, §5.5 · p. 8

5.4 Sensitivity to Prompt

• Performance variance across different exemplars was not large enough to change model rankings or the study’s conclusions.The table reports means and standard deviations over three runs with randomly chosen exemplars.Source: Table 5 · p. 8, §5.4 · p. 8

5.6 Model Calibration

• Model calibration generally correlates with performance, but the best-performing models are not the best-calibrated.The authors note that confidence can indicate generation quality and support confidence-based reranking or voting.Source: Figure 5 · p. 9, §5.6 · p. 9

6 Limitations

• Greedy decoding evaluates one program per example, whereas pass@k and n@k allow multiple tries but cost k times more compute and are left to future work.The authors note that multi-sample evaluation is closer to practical Python use, where test cases can filter incorrect samples.Source: §6 · p. 9
• Execution-based accuracy can count spurious programs as correct; human evaluation found a non-trivial portion for Spider, but not for other datasets.Execution may also be difficult in practice with complex dependencies and potentially harmful programs.Source: §6 · p. 9
• Model comparisons can be confounded by size, pretraining data, architecture, and objective, limiting the validity of conclusions across models.The study mitigates this by fixing variables where possible, including comparisons within the same model series, but urges caution in cross-series interpretation.Source: §6 · p. 10
• For proprietary models, opaque details and mismatches between published descriptions and API engines leave very few scientific conclusions supportable.The authors generally rely on open-source models for findings to avoid speculative details, while still evaluating proprietary APIs and human-evaluating three strong code-generation models.Source: §6 · p. 10

7 Related Work

• Code-generation benchmarks draw on GitHub and StackOverflow data, with execution-based evaluation joining lexical evaluation in newer benchmarks.The passage cites ODEX, DS-1000, HumanEval, and MBPP as examples of execution-based evaluation and notes growing attention to cross-language generalization.Source: §7 · p. 10
• Other code-related directions include code understanding, code search, code translation, and program repair, which the authors leave for future systematic evaluation.Examples include CodeXGLUE for defect detection, clone detection, and code search, and BigCloneBench for predicting functional similarity between code pairs.Source: §7 · p. 10

8 Conclusions

• L2CEval evaluates LLM natural-language-to-code generation across model scale, training data, few-shot exemplars, and instruction tuning, with analyses of calibration and common error modes.The authors hope the study provides useful insights for downstream code applications and future model development.Source: §8 · p. 10

A.1 Example Model Inputs and Outputs

• Tab. 6 illustrates how the study unifies model inputs and outputs across language-to-code tasks.Source: §A.1 · p. 16

A.2 Task-specific Setups

• Spider prompts include database schemas, while WikiTQ adds a non-empty example value for each column to clarify noisy web-table semantics.Spider execution accuracy compares predicted and gold SQL execution results.Source: §A.2 · p. 16
• GSM8k and SVAMP use idiomatic programs whose executed answer variable supplies the response; MBPP requires passing all three assertions, including two hidden tests.Source: §A.2 · p. 16
• HumanEval prompts use function headers and docstrings for completion, while DS-1000 prompts request Python lines completing natural-language-described functionality.Source: §A.2 · p. 16

A.3 Details for Selected Models

• OpenAI model names follow API engine names; the appendix identifies codex-cushman-001 as 12B and text-davinci-002 as based on code-davinci-002.Source: §A.3 · p. 16
• CodeGen spans 2B–16B and progresses from Pile-trained CodeGen-nl to BigQuery-trained CodeGen-multi and BigPython-trained CodeGen-mono.Source: §A.3 · p. 16
• The appendix describes Pythia sizes and checkpoints, Dolly’s instruction tuning, BigCode models’ Stack training, and Alpaca’s instruction-tuning data.Pythia ranges from 1.4B to 12B with 154 intermediate checkpoints; SantaCoder and StarCoder are 1.1B and 15.5B, respectively, and Alpaca uses 52K instruction-following examples distilled from text-davinci-003.Source: §A.3 · p. 17

B.1 Full Few-shot Results

• Full few-shot results compare models by mean per-dataset win rates, using 2 shots for Spider and WikiTQ, 8 for GSM8k, and 3 for MBPP.The shot counts aim to match prior work while accommodating models with smaller context lengths.Source: §B.1 · p. 17

• Spider’s main model differences lie in execution errors caused by failures to understand the database schema, not malformed SQL syntax.Source: §B.2 · p. 17
• About 7% of Spider programs judged correct are spurious, while almost none are observed for GSM8k and MBPP.The paper suggests Spider’s generic execution results, such as numbers or cell values, make chance success easier.Source: §B.2 · p. 17
• Across Spider, GSM8k, and MBPP, models often produce correct programs that differ from the gold program, suggesting potential benefit from self-bootstrapping methods such as (Ni et al., 2022).Source: §B.2 · p. 17

B.4 Scaling Curves for Each Task

• Larger models are generally better across tasks, with smoother scaling for programming-demanding Spider and MBPP and more emergent trends for WikiTQ and GSM8k.Source: Figure 6 · p. 18, §B.4 · p. 18

C Full Prompts

• The appendix provides few-shot prompt examples for Spider, WikiTQ, GSM8k, SVAMP, MBPP, HumanEval, and DS-1000.Source: C Full Prompts · p. 18

B.3 Full Calibration Evaluation Results

• SCAA is more correlated with execution accuracy than ECE, which can rate a highly accurate model as poorly calibrated.ECE is calculated independently of model performance, whereas SCAA is positively impacted by performance.Source: §B.3 · pp. 17–18, Table 9 · p. 21