Source-linked AI summary
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau Yih, Daniel Fried, Sida Wang, Tao Yu
TL;DR
Data science code generation lacks a benchmark combining realistic everyday problems, naturalistic contexts, and reliable execution-based evaluation. DS-1000 addresses this gap with 1,000 StackOverflow-derived problems across seven libraries, multi-criteria tests and surface-form checks, and perturbations against memorization. Codex-002 achieves 43.3% accuracy, while the benchmark’s evaluation accepts only 1.8% incorrect predicted programs and leaves substantial room for improvement.
Problem
Existing benchmarks do not jointly target everyday data science usage, naturalistic contexts, and reliable execution-based evaluation.
Method
DS-1000 builds a 1,000-problem benchmark across seven Python libraries using realistic StackOverflow problems, multi-criteria evaluation, and perturbations against memorization.
Results
43.3% accuracy is achieved by Codex-002, while model performance ranges from 7.4% to 43.3%; only 1.8% of accepted Codex-002 predictions are incorrect.
Takeaways & Limitations
DS-1000 can track progress and support fair comparisons in data science code generation while providing construction methods for tasks with difficult-to-evaluate ground truth.
Takeaways & Limitations
Insufficient test coverage may occur despite more test cases on average, because some benchmark problems have only two or three test cases.
Abstract
from arXiv · showhide
We introduce DS-1000, a code generation benchmark with a thousand data science problems spanning seven Python libraries, such as NumPy and Pandas. Compared to prior works, DS-1000 incorporates three core features. First, our problems reflect diverse, realistic, and practical use cases since we collected them from StackOverflow. Second, our automatic evaluation is highly specific (reliable) -- across all Codex-002-predicted solutions that our evaluation accept, only 1.8% of them are incorrect; we achieve this with multi-criteria metrics, checking both functional correctness by running test cases and surface-form constraints by restricting API usages or keywords. Finally, we proactively defend against memorization by slightly modifying our problems to be different from the original StackOverflow source; consequently, models cannot answer them correctly by memorizing the solutions from pre-training. The current best public system (Codex-002) achieves 43.3% accuracy, leaving ample room for improvement. We release our benchmark at https://ds1000-code-gen.github.io.
1. Introduction
DS-1000 addresses the lack of a realistic, naturalistic, and reliably evaluated benchmark for data science code generation. It introduces 1,000 problems across seven Python libraries, with defenses against memorization and substantial variation in model performance.
- Existing benchmarks lack everyday data science applications, naturalistic contexts, and reliable execution-based evaluation.
- DS-1000 contains 1,000 problems covering seven widely used Python data science libraries.The libraries include NumPy, Pandas, TensorFlow, PyTorch, SciPy, Scikit-learn, and Matplotlib.
- StackOverflow-sourced problems capture diverse real-world requests, including efficiency improvements, bug fixes, API questions, and specified input-output functionality.
- The benchmark uses multi-criteria execution-based evaluation that checks functional correctness and surface-form constraints such as avoiding inefficient for-loops.
- Problem perturbations defend against memorization by changing problems while preserving or modifying their reference solutions.
- 43.3% accuracy is achieved by Codex-002, while evaluated model performance ranges from 7.4% to 43.3%, leaving substantial room for improvement.
2. Benchmark Construction
DS-1000 is constructed through a five-stage pipeline that selects and rewrites StackOverflow problems, builds executable contexts and reviewed multi-criteria evaluations, and perturbs problems against memorization. Its construction combines realistic task sourcing with execution and surface-form checks, while perturbation experiments demonstrate memorization risks in web-derived benchmarks.
- Benchmark pipeline: The five-stage construction pipeline selects StackOverflow problems, rewrites them, implements evaluation, perturbs them, and quality-checks the metric.
- Problem sourcing: 451 unique StackOverflow problems were selected using popularity filters and annotator ratings for clarity, usefulness, difficulty, and evaluability.
- Executable evaluation: The evaluation fixes library versions, creates executable contexts, runs test cases against predicted programs, and applies surface-form constraints.
- Executable evaluation: Surface-form checks target specific APIs or keywords rather than requiring uniform similarity to a reference solution, complementing functional correctness.
- Memorization defense: 72.5% pass@1 accuracy on unmodified numpy-100 falls to 40.6% after perturbation, including 23.6% on semantic perturbations.
- Memorization defense: DS-1000 applies surface and semantic perturbations and adds difficult rewrites to proactively reduce memorization and increase challenge difficulty.
3. Dataset Statistics
DS-1000 contains 1,000 problems from 451 unique StackOverflow problems, spanning seven libraries and emphasizing realistic intents, contexts, and reliable evaluation. Its multi-criteria metric combines execution tests with surface-form constraints, with low measured disagreement against manual review.
- 1,000 problems originate from 451 unique StackOverflow problems, including 152 surface perturbations, 235 semantic perturbations, and 162 difficult rewrites.
- The benchmark checks both execution semantics and surface-form constraints, with 1.6 manually annotated corner test cases per problem on average and constraints on 19.4% of problems.
- DS-1000 covers seven Python libraries and represents more diverse, naturalistic intents and contexts than other listed benchmarks.The benchmark comparison distinguishes general Python benchmarks from data science code-generation benchmarks and highlights StackOverflow-based realism.
- 5.7% of problems contained at least one incorrect accepted prediction, and 5.7% contained at least one correct prediction rejected by the metric.Problem-level measures require judging all 40 sampled predictions for each problem.
4. Benchmarking State-of-the-Art Models
The benchmark evaluates multiple pretrained code models using standardized prompts and finds substantial variation across models, libraries, formats, and perturbation conditions. Codex-002 performs best at 43.3% average accuracy, while insertion prompts improve its accuracy over completion prompts.
- 4.3. Main Results: Codex-002 Insertion achieves 43.3% accuracy on DS-1000, indicating substantial room for improvement.Other models, including CodeGen-6B and InCoder-6B, perform below 5% on some libraries.
- 4.2. Models: The experiments compare Codex, InCoder, and CodeGen families, including Codex-002, Codex-001, Codex-Cushman, and 6B-parameter InCoder and CodeGen models.
- 4.3. Main Results: Model accuracy varies across libraries, so performance on one library may not generalize directly to others.
- 4.1. Prompt Format: Codex-002 averages 4.1% higher accuracy with Insertion format than with Completion format.Insertion supplies both left and right code context, whereas Completion transfers right-context information into the left context for left-to-right models.
- 4.4. Results by Perturbation: Codex-002 performance drops by 3.4% under surface perturbations and 9.0% under semantic perturbations.The authors report these decreases as less severe than those observed on numpy-100.
- 4.4. Results by Perturbation: Codex-002 performs much worse on rewritten problems intentionally made more difficult for human programmers.These rewritten problems are intended as a challenge for future models.
5. Related Work
Prior benchmarks increasingly target complex code generation, but reliable evaluation remains concentrated on competition-style tasks rather than naturalistic, real-world data science problems. DS-1000 addresses this gap with manually implemented execution-based metrics for naturally occurring problems.
- Natural-language-to-code research spans database queries, logical forms, and increasingly complex executable programs.
- 43.3% overall pass@1 accuracy is reported for Codex-002 on 1,000 DS-1000 problems, indicating substantial room for improvement.Table 5 averages across all libraries and uses 40 generated samples per problem; its upper and lower sections correspond to Completion and Insertion formats.
- Most benchmarks with reliable test cases focus on competition, interview, or programming challenges because those settings provide many existing test cases.
- Real-world program evaluations often rely on unreliable surface-form metrics, creating a streetlight effect toward easily evaluated but less practical problems.
- DS-1000 combines naturalistic data science problems with manually implemented reliable execution-based metrics to evaluate practical code generation.
6. Conclusion
DS-1000 is a data-analysis code-generation benchmark designed around realistic problems, reliable automatic metrics, and proactive defenses against memorization. The authors intend it to support progress tracking, fair model comparisons, and benchmark construction in other difficult-to-evaluate domains.
- DS-1000 benchmarks code generation for data analysis using realistic problems, reliable automatic metrics, and defenses against memorization.
- The benchmark is intended to track research progress and facilitate fair comparisons between models.
- Its construction methods may inspire benchmarks in areas where tasks are complicated and ground truth is challenging to evaluate.
A.1. Problem Selection
DS-1000 sources popular, high-quality StackOverflow data-science problems and filters them for suitability under its multi-criteria evaluation. The benchmark is built with specified software versions.
- The initial StackOverflow pool contains thousands of problems selected by removing duplicates and requiring at least one vote, 1,000 views, and an accepted answer.The passage lists 15,881 NumPy, 26,248 Pandas, 1,965 PyTorch, 8,258 TensorFlow, 4,141 SciPy, and 4,499 Scikit-learn problems after filtering.
- Annotators exclude problems when multi-criteria evaluation is not applicable, including hardware and software-error cases.
- DS-1000 is constructed against controlled software versions documented in Table 7.
A.2. Example Problems
DS-1000 examples span varied libraries and problem types, requiring evaluation designs tailored to numerical variation, sparse matrices, plotting, cross-library interactions, and perturbations. The examples also illustrate how reference solutions and prompts are adapted for testability and robustness.
- Example problems: The examples cover NumPy, Pandas, TensorFlow, PyTorch, SciPy, Scikit-learn, and Matplotlib problems.
- Evaluation designs: For random NumPy outputs, evaluation uses a Kolmogorov-Smirnov test rather than exact output equivalence.
- Evaluation designs: The SciPy sparse-matrix example checks type, elements, and nonzero count while rejecting conversions such as .toarray() and .todense().
- Evaluation designs: Examples require fixing reference-solution bugs, designing TensorFlow tests, hiding PyTorch inputs, and testing interactions between Scikit-learn and Pandas.
- Evaluation designs: Matplotlib evaluation compares generated and reference images and checks axis conditions such as grid-line existence and color.
- Perturbations: Perturbations include surface changes, semantic rewrites, altered keywords, indices, orderings, output shapes, and prompt-format transformations.
B. Details of Experiments on numpy-100
The numpy-100 experiments transform short NumPy exercises into contextual code-generation problems, then apply surface and semantic perturbations before testing model performance.
- numpy-100 contains 100 NumPy exercises collected from mailing lists, StackOverflow, and NumPy documentation.
- Each problem originally uses a short one-sentence description without code context, followed by a reference solution.
- The authors add executable code contexts and use an Insertion prompt for each problem.
- Surface perturbations paraphrase descriptions or modify code contexts while preserving the reference solution.
- Semantic perturbations change problem requirements and reference-solution semantics without changing difficulty.
- Each problem and perturbation receives one test case and automatic evaluation before Codex-002 testing on sampled problems.
C. Error Analysis
The error analysis illustrates failures involving composite requirements, specialized library operations, context complexity, and resource constraints across DS-1000-style problems.
- Codex-002 fails on a composite NumPy requirement by generating one operation that removes all duplicates instead of removing adjacent non-zero duplicates.The reference solution uses a binary selection array and two operations.
- Examples span specialized operations such as removing sparse-matrix diagonal entries and converting between dense and sparse matrices.
- Other examples require matching library-specific behavior, including Pandas constraints, TensorFlow tensor comparison, and PyTorch tensor construction.
- The examples also test cross-library transformations, such as applying Scikit-learn preprocessing to a Pandas dataframe and multiplying sparse matrices with NumPy-like broadcasting.
- Perturbed problems alter wording, inputs, keywords, indices, output order, result types, or complexity while requiring the model to preserve the intended task.
- Some problems include richer contexts such as completion prompts, failed attempts, error messages, and GPU-memory limitations.