PaperlayerView exact revision on arXiv

Source-linked AI summary

LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs

Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, Xiaolong Xu

arXiv:2504.14655v1cs.LGcs.CLcs.SE

TL;DR

Research lacks coding benchmarks that accurately assess LLM reasoning and self-contained training testbeds. LeetCodeDataset curates Python problems with metadata, test cases, and temporal splits for evaluation and training. The authors report that reasoning models outperform non-reasoning models, while 2.6K model-generated SFT samples reach 79.9% on HumanEval, surpassing models trained on 9.5K–111.1K rows.

  • Problem

    Research lacks coding benchmarks that accurately assess LLM reasoning; LiveCodeBench covers few problems and lacks detailed algorithm and data-structure tags.

    Source: §1 · p. 1

  • Method

    LeetCodeDataset curates over 90% of Python LeetCode problems with metadata and 100+ test cases, using release-date temporal splits for SFT and evaluation.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Reasoning models outperform non-reasoning models in competitive programming; 2.6K model-generated samples reach 79.9% on HumanEval, surpassing models trained on 9.5K–111.1K rows.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The dataset is a self-contained testbed for code-generation model evaluation and training, and its test cases can also serve as RL verifiers.

    Source: §2.1 · p. 4

    The dataset lacks extremely complex input patterns and has an imbalanced test-case distribution, leaving residual false-positive risk.

    Source: §6 · p. 9

Abstract

from arXiv · show

We introduce LeetCodeDataset, a high-quality benchmark for evaluating and training code-generation models, addressing two key challenges in LLM research: the lack of reasoning-focused coding benchmarks and self-contained training testbeds. By curating LeetCode Python problems with rich metadata, broad coverage, 100+ test cases per problem, and temporal splits (pre/post July 2024), our dataset enables contamination-free evaluation and efficient supervised fine-tuning (SFT). Experiments show reasoning models significantly outperform non-reasoning counterparts, while SFT with only 2.6K model-generated solutions achieves performance comparable to 110K-sample counterparts. The dataset and evaluation framework are available on Hugging Face and Github.

1 Introduction

LeetCodeDataset addresses gaps in reasoning-focused coding evaluation and self-contained training resources with a broad, richly annotated dataset and temporal split. Evaluations show reasoning-model advantages, while SFT with 2.6K samples matches performance of counterparts trained on 110K code examples.

1 Introduction

  • LiveCodeBench (Jain et al., 2024) uses live updates to limit contamination but covers few problems per platform and lacks detailed algorithm and data-structure tags.Source: §1 · p. 1
  • Existing competition datasets lack live updates and easy RL-training tools, while Open-R1 CodeForces-CoTs does not filter for solution correctness.APPS (Hendrycks et al., 2021), CodeContests (Li et al., 2022), and TACO (Li et al., 2023) provide competition problems split into training and test sets.Source: §1 · pp. 1–2
  • LeetCodeDataset covers over 90% of platform Python problems, with rich metadata and 100+ test cases each, plus an evaluation toolkit and a pre/post-July 1, 2024 temporal split.Problems released after July 1, 2024, form the test set; earlier problems form the training set.Source: §1 · p. 2
  • With only 2.6K SFT samples, the resulting model performs comparably to counterparts trained on 110K code examples; reasoning models outperform non-reasoning models, with Claude 3.7 Sonnet best in its category.The evaluation covered popular proprietary and open-source models.Source: §1 · p. 2

2 LeetCodeDataset

LeetCodeDataset combines verified LeetCode Python problems, extensive generated tests, and date-based training/evaluation splits to support benchmarking and model training. Its overview characterizes problem difficulty, release dates, and topic tags as dimensions for evaluating coverage and selecting recent benchmark problems.

2.1 Data Collection

  • The dataset records each problem’s identifier, difficulty, description, starter code, and topic tags, while a sandbox computes canonical-solution outputs for generated inputs.Special handling supports data structures such as binary trees and linked lists.Source: §2.1 · pp. 2–3
  • Data collection focuses on single-function starter code, because specialized validation for multi-function design or simulation problems requires unavailable, difficult-to-develop judgment code.The entry point is generally identified automatically by text pattern matching.Source: §2.1 · p. 3
  • 2,869 problems—over 90% of LeetCode’s Python problems—have generated outputs, with verified canonical solutions and an average of over 100 inputs per problem.Canonical solutions were verified on LeetCode with a 100% acceptance rate; repeated prompting produced many complex inputs to reduce false positives.Source: §2.1 · pp. 2–3
  • The dataset supports evaluation and SFT through a temporal split: post-cutoff problems form evaluation, while earlier problems provide training data and verified model-generated responses.Qwen2.5-Coder-32B-Instruct (Hui et al., 2024) generated responses through high-temperature sampling and test verification; ground-truth snippets served as hints for persistently failing problems. The dataset can also support RL using test cases as verifiers.Source: §2.1 · pp. 3–4

2.2 Dataset Overview

  • The overview organizes problems by Easy, Medium, and Hard difficulty and by multiple algorithm and data-structure tags, such as Array and Binary Search.Difficulty ranges from foundational syntax and data structures to complex algorithmic combinations or specialized optimizations; tags can help learners target skills and may provide insights for LLMs.Source: §2.2 · pp. 4–5
  • Approximately 350 new problems have been added annually in recent years; release dates are estimated from contest dates and question IDs, and the authors recommend problems from the past 6–12 months for benchmarking.The authors argue this recency window balances bias and variance in contamination-free evaluation.Source: §2.2 · p. 4

3 Holistic Evaluation

On 256 newly released problems, reasoning models achieved the strongest overall results, while performance across topic tags varied by model. The evaluation also compared monthly pass-rate changes and difficulty-level results to examine possible contamination.

3 Holistic Evaluation

  • The evaluation tested six models on 256 problems released after July 1, 2024, using identical generation settings of temperature=0.2 and top_p=0.95.The set included two proprietary systems and four open-source models.Source: §3 · p. 5
  • Monthly pass-rate changes by problem release month were analyzed to detect possible contamination, with post-release accuracy declines treated as evidence of overfitting to pre-release training data.Figure 3 shows monthly pass rates; Table 2 summarizes pass rates across difficulty levels.Source: §3 · p. 5, Figure 3 · p. 6, Table 2 · p. 6
  • GPT-4o and DeepSeek-V3 achieved the same overall score; GPT-4o performed slightly better on easy problems, whereas DeepSeek-V3 performed slightly better on hard problems.Claude-3.7-Sonnet, without extended thinking, achieved superior performance within its model category.Source: §3 · pp. 5–6
  • On the LeetCodeDataset evaluation, DeepSeek-R1 achieved a 65.23% pass@1 rate and QwQ-Plus 56.25%, with the authors highlighting long-CoT reasoning models’ substantial advantage on complex competition-level coding problems.Source: §3 · p. 5, Table 2 · p. 6
  • DeepSeek-R1 (DeepSeek-AI et al., 2025a) stayed mostly between 60% and 70% across topic tags, while GPT-4o ranged from 7.7% on Binary Search to 63.2% on Simulation.Table 3 presents model pass rates across topic tags.Source: §3 · p. 7, Table 3 · p. 7
  • Reasoning and non-reasoning models differed substantially on Dynamic Programming, Binary Search, and Tree-related tasks, indicating a need for additional reasoning capabilities in these domains.Source: §3 · p. 7

4 Efficient Training

LeetCodeDataset supports efficient supervised fine-tuning: 2.6K model-generated samples outperformed models trained on datasets of 9.5K–111.1K samples on HumanEval and MBPP. The results also show that model-generated responses outperform human-written responses, while the small-scale model underperforms on hard benchmarks.

4.1 Experiment Setup

  • SFT used Qwen2.5-Coder-7B for 3 epochs with a 1e-5 initial learning rate, 0.1 warmup ratio, cosine scheduling, and batch size 32.The same hyperparameters were used across datasets.Source: §4.1 · p. 7

4.2 Results

  • 79.9% vs. 55.5% on HumanEval and 77.5% vs. 53.4% on MBPP favored model-generated over human-written responses, although both response types were verified as correct.Both models used responses from the pre-2024-07 LeetCodeDataset.Source: §4.2 · p. 8
  • The 2.6K-trained model underperformed on hard benchmarks despite being evaluated in-distribution on post-2024-07 LeetCodeDataset problems.The passage suggests small-scale SFT primarily develops basic programming skills.Source: §4.2 · p. 8
  • 79.9% on HumanEval and 77.5% on MBPP with 2.6K model-generated samples surpassed models trained on datasets of 9.5K–111.1K samples.The evaluation covered HumanEval, MBPP, LiveCodeBench, and the LeetCodeDataset evaluation set.Source: Table 4 · p. 8, §4.2 · p. 8

5 Related Work

Related work covers foundational and competitive-programming code-generation benchmarks, alongside synthetic and benchmark-provided datasets for code-model fine-tuning.

Code Generation Benchmarks

Fine-tuning Dataset of Code

  • Code SFT resources include synthetic instruction data from CodeAlpaca and Magicoder (Wei et al., 2024), benchmark training splits, reasoning traces in pen-R1 CodeForces-CoTs, and OpenThoughts (Team, 2025a).OpenThoughts contains 114K examples spanning math, science, code, and puzzles; pen-R1 CodeForces-CoTs covers 10K CodeForces problems with up to five DeepSeek R1-generated traces each.Source: §5 · p. 9

6 Limitations

The dataset has three limitations: residual false-positive risks, no problem-specific complexity validation, and gaps in problem-type coverage.

False Positive Risks

  • Diverse tests reduce incorrect solutions passing, but missing extreme input patterns and imbalanced test distributions leave false-positive risks.Solutions may pass tests despite logic errors.Source: §6 · p. 9

Complexity Analysis Gap

  • Time and space complexity evaluation requires algorithm-specific LeetCode-style tests and manual problem-specific validation, which is outside the current scope.Source: §6 · p. 9

Coverage Gaps

  • Coverage omits certain problem types, particularly those with multiple solution entry points.Source: §6 · p. 9

7 Conclusion

LeetCodeDataset combines curated Python LeetCode problems, rich metadata, extensive tests, and a temporal split to support contamination-free evaluation and efficient training. The authors report that 2.6K curated samples can match performance from 110K examples in previous benchmarks.

7 Conclusion

  • 2.6K curated LeetCodeDataset samples match the performance of models trained on 110K examples from previous benchmarks.The authors present this as evidence of strong data efficiency.Source: §7 · p. 9
  • The dataset aggregates 2,869 Python LeetCode problems with difficulty, tags, and release-date metadata, each augmented with 100+ diverse test cases.Its temporal split uses post-July 2024 problems as the test set, supporting clean benchmarking and longitudinal studies.Source: §7 · p. 9
  • Broad coverage of algorithms and data structures supports overall evaluation and fine-grained skill analysis, while an integrated toolkit streamlines model assessment and comparison.The authors position the dataset as a resource for developing, training, and evaluating code-generation models.Source: §7 · p. 9

A Appendix · A.1 Prompts

The appendix describes type-matched one-shot examples for generating problem inputs, plus imports and helper functions for evaluating code that uses linked lists and binary trees.

A.1 Prompts

  • 10 sampled problems—three tree, three linked-list, and four other types—serve as one-shot examples for input generation.Their input specifications are extracted from problem descriptions.Source: §A.1 · p. 12
  • The input-generation prompt selects examples from the same structural domain as the target problem type.Tree problems use tree examples, linked-list problems use linked-list cases, and other problems follow the same principle.Source: §A.1 · p. 12
  • The appendix includes prompt structures for input generation and complex input generation.Figure 4 covers input generation; Figure 5 covers complex input generation.Source: Figure 4 · p. 12, Figure 5 · p. 12

A.1.1 Handle Data Structures

  • Code completions receive essential imports, such as `from typing import List`, to support robust evaluation.The imports are prepended to all code completions.Source: §A.1.1 · p. 13
  • Binary trees and linked lists require additional utility functions for serialization and deserialization.The appendix also shows additional imports related to linked lists and binary trees.Source: §A.1.1 · p. 13, Figure 6 · p. 13, Figure 7 · p. 14
  • Figures 6 and 7 show additional imports for linked lists and binary trees, respectively.Source: Figure 6 · p. 13, Figure 7 · p. 14

Related papers

Loading 2504.14655v1…