Source-linked AI summary
LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, Xiaolong Xu
TL;DR
LeetCodeDataset targets the shortage of reasoning-focused coding benchmarks and self-contained training testbeds. It curates Python LeetCode problems with metadata, verified tests, and temporal splits for evaluation and training. Reasoning models perform better than non-reasoning models, and 2.6K model-generated SFT samples can match much larger datasets.
Problem
LLM research lacks coding benchmarks that accurately assess reasoning abilities and self-contained testbeds for training competition-level coding.
Method
LeetCodeDataset curates over 90% of Python LeetCode problems with rich metadata, 100+ inputs, verified responses, and a pre/post-July-2024 temporal split.
Results
Reasoning models outperform non-reasoning models, while 2.6K model-generated SFT samples achieve performance comparable to models trained on 110K code examples.
Takeaways & Limitations
The dataset provides contamination-free evaluation and efficient training for code-generation models, with test cases also supporting RL workflows.
Takeaways & Limitations
The dataset lacks extremely complex input patterns and has imbalanced test-case distributions, leaving residual false-positive risks.
Abstract
from arXiv · showhide
We introduce LeetCodeDataset, a high-quality benchmark for evaluating and training code-generation models, addressing two key challenges in LLM research: the lack of reasoning-focused coding benchmarks and self-contained training testbeds. By curating LeetCode Python problems with rich metadata, broad coverage, 100+ test cases per problem, and temporal splits (pre/post July 2024), our dataset enables contamination-free evaluation and efficient supervised fine-tuning (SFT). Experiments show reasoning models significantly outperform non-reasoning counterparts, while SFT with only 2.6K model-generated solutions achieves performance comparable to 110K-sample counterparts. The dataset and evaluation framework are available on Hugging Face and Github.
1 Introduction
LeetCodeDataset addresses missing reasoning-focused coding benchmarks and self-contained training testbeds by curating richly annotated Python problems with verified tests and temporal splits. Evaluations show reasoning models outperform non-reasoning models, while small-scale SFT can match much larger training sets.
- LLM research lacks coding benchmarks that accurately assess reasoning abilities.
- Existing competition datasets lack combinations of live updates, detailed algorithmic tags, and convenient support for rigorous training workflows.
- LeetCodeDataset covers over 90% of Python problems with rich metadata, 100+ varied test cases per problem, an evaluation toolkit, and a strict pre/post-July-2024 split.
- Reasoning models outperform non-reasoning models on competitive programming tasks, with Claude 3.7 Sonnet leading its model category.
- 2.6K SFT samples produce performance comparable to models trained on 110K code examples.
2 LeetCodeDataset
LeetCodeDataset combines broad Python coverage, metadata, verified solutions, generated test cases, and release-date splits into a benchmark and training resource. Its construction supports reliable evaluation and SFT, while test cases can also serve as RL verifiers.
- Data Collection: Each problem collects identifiers, difficulty, description, starter code, and topic tags through LeetCode’s GraphQL API.
- Data Collection: Canonical solutions from open-source repositories were verified on LeetCode, establishing ground-truth solutions with a 100% acceptance rate.
- Data Collection: Repeated one-shot prompting generates an average of over 100 inputs per problem, including complex cases intended to reduce false positives.
- Data Collection: 2,869 Python problems were processed successfully, covering over 90% of the platform’s available Python problems.
- Dataset Overview: The dataset uses release dates to separate earlier training problems from post-July-2024 evaluation problems.
- Data Collection: Verified model-generated query-response pairs support SFT, while test cases can function as verifiers for RL training.
3 Holistic Evaluation
The evaluation compares six models on 256 post-July 1, 2024 LeetCode problems using temporal accuracy analysis, difficulty-level pass rates, and topic-tag performance. Reasoning models lead overall and show more consistent performance across topic tags, while non-reasoning models vary substantially by task.
- Evaluation setup: 256 newly released post-July 1, 2024 problems form the evaluation set for six proprietary and open-source models under identical generation parameters.The experiments use temperature=0.2 and top_p=0.95 for fair comparisons.
- Overall results: 65.23% pass@1 for DeepSeek-R1 and 56.25% for QwQ-Plus identify the top-performing reasoning models.These results are reported as evidence of an advantage for long-CoT reasoning models on complex competition-level coding problems.
- Overall results: Claude-3.7-Sonnet performs best among the evaluated models in its category, while GPT-4o and DeepSeek-V3 achieve the same overall score.GPT-4o performs slightly better on easy problems, whereas DeepSeek-V3 performs slightly better on hard problems.
- Topic-tag analysis: DeepSeek-R1 maintains pass rates mostly between 60% and 70% across topic tags, whereas GPT-4o ranges from 7.7% in Binary Search to 63.2% in Simulation.The comparison indicates substantially greater topic-to-topic consistency for DeepSeek-R1.
- Topic-tag analysis: Reasoning and non-reasoning models differ substantially on Dynamic Programming, Binary Search, and Tree-related tasks.The paper identifies these domains as areas where additional reasoning capabilities are needed.
4 Efficient Training
The paper evaluates LeetCodeDataset as a supervised fine-tuning resource against larger coding datasets under matched training conditions. Model-generated responses and a 2.6K-sample training set deliver strong benchmark performance, but the small-scale model underperforms on hard benchmarks.
- Training setup: Qwen2.5-Coder-7B is trained for three epochs with a 1e-5 learning rate, 0.1 warmup ratio, cosine scheduling, and batch size 32.These hyperparameters are kept consistent across the training experiments.
- Experimental design: The study compares LeetCodeDataset with five coding datasets containing 9.5K to 111.1K samples across four code-generation benchmarks.The evaluation includes HumanEval, MBPP, LiveCodeBench, and the LeetCodeDataset test set.
- Response-source comparison: 79.9% versus 55.5% on HumanEval and 77.5% versus 53.4% on MBPP favor model-generated over human-written LeetCode responses.Both response types were verified as correct.
- Data efficiency: 2.6K model-generated LeetCode samples achieve 79.9% on HumanEval and 77.5% on MBPP, surpassing models trained on 9.5K–111.1K-row datasets.The result is reported as high data efficiency for domain-specific code generation.
- Limitations: The SFT model trained on 2.6K samples underperforms on hard benchmarks despite those problems being in-distribution for LeetCodeDataset.The paper suggests that small-scale SFT primarily develops basic programming skills.
5 Related Work
Prior work includes foundational Python benchmarks, competitive-programming datasets, and synthetic code fine-tuning resources. These resources differ in language coverage, problem difficulty, update strategy, and the availability of training data or reasoning traces.
- Code-generation benchmarks: HumanEval and MBPP are widely used foundational Python benchmarks, while EvalPlus provides a more rigorous variant and Multiple-E extends them to 18 additional languages.The paper places these datasets within broader code-generation benchmark development.
- Competitive-programming benchmarks: APPS, CodeContests, and TACO target competitive programming, but existing datasets may not provide live updates or tools suited to reinforcement-learning workflows.The discussion contrasts these resources with the needs addressed by LeetCodeDataset.
- Code fine-tuning datasets: CodeAlpaca and Magicoder synthesize code fine-tuning data using few-shot prompting, teacher models, or open-source code snippets.These datasets represent synthetic-data approaches for code-specific supervised fine-tuning.
- Code fine-tuning datasets: OpenThoughts provides 114K synthetic examples across multiple domains, while Open-R1 CodeForces-CoTs includes 10K CodeForces problems with up to five DeepSeek-R1 reasoning traces.The paper contrasts broad synthetic coverage with competition-focused reasoning data.
6 Limitations
LeetCodeDataset has three limitations: residual false-positive risks, an unaddressed complexity-analysis requirement, and incomplete coverage of problems with multiple solution entry points.
- False positives remain possible because the dataset lacks extremely complex input patterns and has imbalanced test-case distributions.Incorrect solutions may pass tests despite logic errors.
- Determining time and space complexity requires LeetCode-style test cases tailored to each algorithm’s behavior.Addressing this gap requires manual problem-specific validation beyond the current scope.
- The dataset excludes certain problem types, particularly those with multiple solution entry points.
7 Conclusion
LeetCodeDataset is a rigorously curated resource for code-generation research, combining broad Python problem coverage, rich metadata, diverse testing, and temporal evaluation. Its experiments show strong data efficiency, with 2.6K curated samples matching models trained on 110K examples.
- The temporal split supports contamination-free benchmarking and longitudinal studies.
- The dataset covers algorithms and data structures while supporting fine-grained skill analysis and integrated model comparison.
- 2.6K curated LeetCodeDataset samples matched the performance of models trained on 110K examples from previous benchmarks.
A.1 Prompts
The appendix describes prompts for generating problem-consistent inputs, including specialized complex-input prompting and data-structure handling for robust test construction.
- A.1 Prompts: Input generation samples three tree, three linked-list, and four other problems as one-shot examples for domain-specific prompting.The ten examples are selected with constraints matching each problem type’s structure.
- A.1 Prompts: The Input-Generation-Prompt asks an expert Python programmer to generate inputs consistent with the problem specification and starter code.
- A.1 Prompts: Figure 4 presents the prompt structure used for input generation.
- A.1 Prompts: Complex-Input-Generation-Prompt extends input generation beyond the basic prompt.
- A.1 Prompts: The complex-input prompt uses a question, starter code, and sample inputs to generate additional consistent inputs.
- A.1 Prompts: Figure 5 presents the prompt structure for complex input generation.
- A.1.1 Handle Data Structures: Essential imports are prepended to completions, with separate serialization and deserialization utilities for binary trees and linked lists.