Source-linked AI summary
Measuring Coding Challenge Competence With APPS
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, Jacob Steinhardt
TL;DR
Whether large-scale language models can reliably write code remains open, and rigorous evaluation is difficult despite programming’s broad importance. The paper introduces APPS, a 10,000-problem benchmark using natural-language specifications and test cases, and finds low but improving performance with exponentially decreasing syntax errors.
Problem
Whether large-scale language models can reliably write code remains an open question, while rigorous code-generation evaluation is needed for this broadly important task.
Method
APPS evaluates Python generation from natural-language specifications across 10,000 problems using over 130,000 test cases and curated programming tasks.
Results
Models show nonzero accuracy, syntax errors decrease exponentially as models improve, and the strongest evaluated model passes almost 20% of introductory test cases given five attempts.
Takeaways & Limitations
APPS provides a rigorous benchmark for tracking progress on code generation, which remains challenging but is becoming tractable for large-scale language models.
Takeaways & Limitations
Some problems have only two test cases, creating a potential false-positive risk, especially when the output space is small.
Abstract
from arXiv · showhide
While programming is one of the most broadly applicable skills in modern society, modern machine learning models still cannot code solutions to basic problems. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to accurately assess code generation performance rigorously. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximately 20% of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an important measure for tracking advancements.
1 Introduction
APPS addresses the open question of whether language models can reliably write code by evaluating Python generation from natural-language specifications. It combines diverse problems with test-case-based assessment and finds low but improving performance, including exponentially decreasing syntax errors.
- Benchmark motivation: APPS evaluates whether models can generate Python code that satisfies arbitrary natural-language specifications, rather than translating code or pseudocode.The setting is intended to resemble human software-developer assessment.
- Benchmark scope: The benchmark contains 10,000 problems spanning introductory, interview-level, and coding-competition difficulties.Its coverage is designed to assess both task understanding and algorithm design.
- Evaluation: APPS uses over 130,000 test cases to assess program correctness instead of relying on metrics such as BLEU.The tests probe functionality across the input space and provide a gold-standard quality metric.
- Findings: Models exhibit nonzero accuracy, while syntax errors decrease exponentially as models improve and accuracy declines with problem difficulty.Accuracy also improves with fine-tuning and larger model size.
- Findings: Almost 20% of test cases are passed on introductory problems by the strongest evaluated model given five attempts.These results characterize code generation as challenging but tractable for large-scale language models.
- Significance: The benchmark is intended to track code-generation advances rigorously as programming remains broadly used across society.The authors also note the potential economic and social implications of automatic code generation.
2 Related Work
Prior program-synthesis and code-understanding datasets typically constrain the specification or programming setting. APPS extends evaluation to long natural-language specifications and general-purpose programming languages while addressing the need for resilient benchmarks.
- Program Synthesis: Deductive synthesis uses formal logic specifications and complex optimization, but converting behavior into a formal language can make the approach rigid.Inductive synthesis instead uses example input-output behavior but may incompletely specify behavior.
- Program Synthesis: Earlier natural-language synthesis work generated short programs in ad-hoc languages or translated natural-language queries into short SQL commands.These settings constrain program length, language, or task form.
- Code Understanding Datasets: Other code datasets address limited settings such as translation between programming languages or pseudocode-to-code generation.These approaches demonstrate success without directly evaluating arbitrary natural-language specifications and general-purpose code.
- Evaluating Large-Scale Language Models: As language models improve across text-based benchmarks, finding resilient benchmarks where they significantly underperform humans becomes more difficult.APPS is presented as an attempt to fill this evaluation gap and separate model performance from expert-human performance.
3 The APPS Dataset
APPS is a 10,000-problem benchmark that evaluates Python code generation from natural-language specifications across difficulty levels and problem formats. It uses standardized execution and test-case evaluation to assess solution correctness, while documenting potential false positives in sparse-test problems.
- Dataset Construction: APPS contains 10,000 coding problems spanning introductory, interview, and competition difficulty levels.The dataset is split evenly into 5,000 training and 5,000 test problems.
- Dataset Construction: The dataset is manually curated from open-access programming sites and normalized with custom parsers for formatting such as LaTeX, lists, and sections.Sources include Codewars, AtCoder, Kattis, and Codeforces.
- Problem Formats: Problems use call-based or standard-input formats, requiring models either to return a function result or print answers to STDOUT.Call-based problems generally provide starter code, whereas standard-input problems generally do not.
- Test Case Quality: Sparse test suites can produce false positives, especially for problems with only two tests and small output spaces.Among GPT-Neo 2.7B solutions manually checked, 8 of 12 two-test problems were false positives, compared with 1 of 6 three-test problems; estimated rates were below 1% in both groups.
4 Experiments
The experiments evaluate language models on APPS using automatic test-case metrics and examine how model scale, fine-tuning, sampling, syntax errors, and evaluation choices affect code-generation performance.
- Test Case Evaluation: GPT-Neo achieves approximately 4% strict accuracy and passes approximately 15% of test cases on introductory problems.These values are reported for GPT-Neo in Table 2.
- Evaluation: APPS evaluates generated programs with test cases using test case average and strict accuracy, while strict accuracy requires passing every test case.Test case average captures partial progress, whereas strict accuracy can obscure improvements when solutions miss corner cases.
- Model Performance Analysis: Average test-case performance increases with larger fine-tuned models, indicating improving code-generation performance as model size grows.Figure 4 reports this trend across larger fine-tuned models.
- Model Performance Analysis: Sampling five candidate programs improves performance: GPT-Neo’s introductory strict accuracy exceeds 5% when the best of five beams is selected.The experiment uses beam search with width 5, giving each model five attempts per problem.
- Syntax Errors: Syntax errors decrease exponentially with fine-tuning and model size, falling from approximately 59% for GPT-3 to approximately 3% for GPT-Neo on introductory problems.Syntax errors are identified by whether generated Python can be loaded as a module.
- Evaluation and Model Choice: BLEU is unsuitable as a substitute for test-case evaluation because it can increase with problem difficulty and be anticorrelated with gold-standard accuracy.The experiments also report that larger or fine-tuned models improve performance, while solving APPS efficiently may require architectural or algorithmic improvements.
5 Conclusion
APPS is a rigorous benchmark for generating Python code from natural-language specifications across 10,000 problems and varied difficulty levels. Model performance remains low, but syntax errors decrease exponentially as models improve.
- APPS benchmarks Python code generation from natural-language specifications using hundreds of thousands of test cases and ground-truth solutions.The benchmark spans different difficulty levels and is intended to track program-synthesis advances.
- 10,000 Python programming problems form the benchmark, covering a broad testbed for evaluating language models.
- Overall model performance was low, although syntax errors decreased exponentially as models improved.
- Recent models such as GPT-Neo solved over 5% of introductory problems.
- APPS provides a proxy for tracking code-generation capability as automatic code generation becomes more socially significant.
A Auxiliary Dataset Information
The dataset uses publicly available coding-challenge material and addresses licensing, copyright, privacy, and responsibility. The authors state that APPS data is licensed under CC BY-SA 3.0 and accompanying code under MIT.
- APPS scrapes public question text, ground-truth solutions, and test cases from coding-challenge websites, avoiding paywalled material.
- The source websites span AtCoder, CodeChef, Codeforces, Codewars, HackerRank, Kattis, and LeetCode.
- The authors discuss compliance with international copyright laws, including the DMCA and WIPO Copyright Treaty.
- Identifying information in comments on publicly posted solutions is removed to preserve privacy.
- APPS data is licensed under CC BY-SA 3.0, while the code is open sourced under the MIT license.
B Datasheets
The dataset was created to track progress in generating arbitrary Python code from complex natural-language specifications, a setting the authors describe as lacking a rigorous benchmark. It contains coding-challenge problems with solutions and test cases.
- APPS was created to track progress on generating arbitrary Python code from complex natural-language specifications.
- The dataset targets a challenging code-generation setting that lacked a rigorous benchmark before APPS.
- The datasheet reports no associated grant and no additional comments.
B.2 Composition
APPS consists of coding-challenge problems represented by text, solutions, and test cases, with difficulty-balanced training and test splits. The data was scraped from public websites, manually checked, and filtered for quality and suitability.
- Composition: 10,000 problems, 232,421 ground-truth solutions, and 131,777 test cases comprise APPS.
- Composition: APPS contains a subset of possible test cases, and duplicate or near-duplicate problems are removed.
- Composition: Each instance includes text and numerical data, with test cases providing a ground-truth signal for functional correctness.
- Composition: The dataset provides training and test splits optimized to increase test cases in the test split while maintaining fixed problem counts per difficulty.
- Composition: The dataset contains no confidential or sensitive data, but the presence of potentially offensive content is unknown.
- Collection Process: Problems were scraped from coding-challenge websites using off-the-shelf and custom scrapers, then manually checked against the source text.
- Collection Process: Some scraped problems were excluded because they required images, lacked ground-truth solutions or test cases, or duplicated other problems.
- Collection Process: Data was collected from late 2020 to early 2021 and refined for six months by student authors.
B.5 Uses
The dataset is publicly distributed and available through GitHub, with plans for updates and extensions. Its documentation reports no prohibited tasks or retention limits, while describing legal-compliance practices and licensing constraints.
- The documentation reports no tasks for which the dataset should not be used and no applicable retention limits.
- The dataset is publicly distributed and currently available through GitHub.
- The experimental framework is distributed under an MIT license, while scraped sources may impose copyright-related restrictions addressed through Fair Use and GDPR compliance.
- The creators plan to add test cases appearing in problem statements and allow extensions with additional problems following the existing format.
C Additional Dataset Information
The supplementary dataset information situates APPS among earlier code-generation datasets and describes how comparisons count programs and exercises. GitHub pretraining data were filtered to reduce overlap with coding-challenge problems.
- Expanded Dataset Comparisons: APPS is compared with Hearthstone, Django, and NAPS, alongside datasets based on card games, code comments, and pseudocode-to-code translation.
- Expanded Dataset Comparisons: In dataset comparisons, “Number of Programs” counts human-written programs or functions, whereas “Number of Exercises” counts tasks the network must solve.
- Excluded Keywords: The GitHub pretraining corpus excludes keywords associated with coding-challenge platforms and programming exercises to prevent overlap with APPS-like questions.
D Additional Results
Additional results show that model performance depends on sampling, problem type, and syntax reliability. GPT-3 solved only three of 5,000 test problems, while fine-tuned models exhibited far fewer syntax errors.
- Top-5 evaluation generates five candidate solutions and selects the best-performing one.
- The reported GPT-3 evaluation used separate few-shot prompts for standard-input and call-based questions, with two example questions and solutions per prompt.
- GPT-3 solved 3 problems out of 5,000, including two introductory problems and one interview problem requiring potentially nontrivial reasoning.
- GPT-3’s improper if-else formatting caused a syntax error, whereas GPT-2 0.1B had fewer syntax errors and GPT-Neo 2.7B had almost none.
- The examples include coding problems involving directed graphs, tree edge coloring, and the even-odd game.
Model Output
The model-output examples illustrate both malformed and structurally composed code. They include excessive imports, a mirror-reflection problem, and a call-based format with starter code.
- One generated output contains many import statements, including libraries such as NumPy and SciPy.
- The examples include a mirror-reflection problem with input p = 2 and q = 1, whose output is receptor 2.
- Another example uses the call-based format with starter code, including a mirrorReflection method signature.