Source-linked AI summary
TACO: Topics in Algorithmic COde generation dataset
Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, Ge Li
TL;DR
Code-generation benchmarks often fail to reflect challenging real-world programming and lack sufficiently fine-grained algorithmic coverage. TACO constructs a large, verified dataset of competition-level problems with detailed labels and evaluates models on a difficult test set. The benchmark exposes substantial difficulty for current models while supporting skill-specific training and evaluation.
Problem
Existing code-generation benchmarks provide limited coverage of challenging real-world problems and insufficiently detailed algorithmic metadata.
Method
TACO combines competition-level programming tasks, verified Python 3 solutions, fine-grained algorithmic labels, and benchmark evaluation procedures.
Results
GPT-4 scores 31.5 pass@1 on TACO’s easy test level, while other coding models generally score below 10 across all five difficulty levels.
Takeaways & Limitations
TACO provides a challenging benchmark and fine-grained training resource for assessing and improving code-generation capabilities by programming skill.
Takeaways & Limitations
Because some TACO problems were reclassified from APPS and CodeContest, those datasets are unsuitable as evaluation benchmarks after TACO-based training.
Abstract
from arXiv · showhide
We introduce TACO, an open-source, large-scale code generation dataset, with a focus on the optics of algorithms, designed to provide a more challenging training dataset and evaluation benchmark in the field of code generation models. TACO includes competition-level programming questions that are more challenging, to enhance or evaluate problem understanding and reasoning abilities in real-world programming scenarios. There are 25433 and 1000 coding problems in training and test set, as well as up to 1.55 million diverse solution answers. Moreover, each TACO problem includes several fine-grained labels such as task topics, algorithms, programming skills, and difficulty levels, providing a more precise reference for the training and evaluation of code generation models. The dataset and evaluation scripts are available on Hugging Face Hub (https://huggingface.co/datasets/BAAI/TACO) and Github (https://github.com/FlagOpen/TACO).
1 Introduction
Existing code-generation benchmarks often underrepresent real-world programming difficulty and provide limited evidence of model capability. TACO addresses this gap with challenging competition-level tasks and fine-grained algorithmic annotations.
- 7% success rate was achieved by GPT-4 on LeetCode hard tasks, indicating weak performance on challenging programming problems.
- Mainstream benchmarks often use basic function-completion tasks, while their high scores reduce the reference value of evaluation results.HumanEval’s SOTA model reaches 94.4 pass@1 and MBPP’s reaches 81.1 Acc.
- Existing benchmarks can also have test-set quality problems, including missing manual answers and insufficient deduplication.
- TACO models programming synthesis as multiple instance-level tasks because programming problems require heterogeneous strategies, algorithms, and data structures.
- TACO provides 26,443 programming tasks spanning mathematics, data structures, and graph theory, with detailed algorithmic annotations.
2 Overview and Characteristics
TACO combines broad programming-task coverage with algorithmic metadata and verified Python solutions. Its examples retain detailed problem, solution, constraint, and labeling information for code-generation training and evaluation.
- TACO includes metadata such as task skills, algorithms, time and space constraints, timestamps, sources, and expected complexity.The example record includes labels for number theory, mathematics, geometry, and complete search.
- The dataset’s problem records pair natural-language questions with solutions, labels, examples, and structured input-output information.
- TACO was designed to incorporate algorithmic competencies into code-generation tasks through a broader and more finely annotated dataset.
- TACO contains 26,443 problems and 1,539,152 verified Python 3 solutions, averaging 58.21 correct solutions per problem.The training set has 25,443 problems and the test set has 1,000.
- The dataset consolidates 968 algorithmic labels into 36 distinct categories for programming-challenge tasks.
3 Dataset Construction
TACO was assembled from competition platforms and existing code datasets, then processed through parsing, validation, augmentation, cleaning, and deduplication. The dataset preserves diverse sources while consolidating algorithmic metadata into 36 coarse-grained labels.
- Data collection phase: TACO combined web-scraped programming-contest problems with existing datasets, including APPS, CodeContest, and Description2code.Sources included CodeChef, CodeForces, HackerRank, and GeeksforGeeks, with data merging and deduplication across multiple sources.
- Data collection phase: Custom HTML parsers extracted problem information and algorithmic tags from heterogeneous platform pages, with accuracy checked through extensive manual sampling.OCR APIs were used occasionally to convert SVG information into LaTeX format.
- Data processing phase: Each harvested solution was tested against predefined unit tests, and solutions failing correctness, resource, or coding-standard criteria were removed.
- Data processing phase: The processing pipeline removed comments with AST parsing, near-duplicates with MinHash and Jaccard similarity, and incompatible Python 2 solutions through Python 3 conversion.The deduplication threshold was a Jaccard similarity of 0.85; failed conversions were excluded.
- Data processing phase: Test-case supplementation used verified solutions to generate and repeatedly check input-output pairs until each problem had at least 200 unit tests.
- Skill algorithm categorization: Fine-grained metadata was consolidated into 36 coarse-grained algorithmic labels while retaining original, algorithm, and skill labels for later reclassification.
4 Comparisons with Code Datasets
TACO extends existing code datasets with algorithmic annotations, broader programming-challenge coverage, and richer evaluation metadata. It also addresses weaknesses in prior benchmarks, including missing Python solutions and limited validation of modern code models.
- A notable proportion of APPS and CodeContest test problems lack human-submitted Python solutions, limiting validation of model inference.
- APPS mainly benchmarks GPT-2 and GPT-Neo, while TACO includes benchmarks produced through fine-tuning contemporary code models, including StarCoder.
- TACO adds systematic algorithmic annotations to programming problems, alongside original labels, skill categories, and time and space constraints.
- TACO supports both Standard Input and Call-Based formats, whereas APPS primarily uses Standard Input.
5 Code Generation Evaluation
TACO evaluates code generation across programming skills and difficulty levels using repeated sampling and pass@k metrics. Results show that the benchmark remains difficult for both GPT-4 and smaller code models, while skill-specific fine-tuning can improve selected abilities.
- TACO evaluation covers programming skills and difficulty levels, with pass@1, pass@10, and pass@100 computed from 200 generated attempts per problem.
- Evaluation setup: GPT-4 uses one generation with top p=0.95 and temperature=0.7, whereas general code models generate 200 seeded attempts followed by execution on all test cases.
- Results: 31.5 pass@1 is GPT-4’s score on easy TACO problems, while other coding models generally score below 10 pass@1 across all five difficulty levels.
- Skill-specific fine-tuning: StarCoder-1B skill-specific LoRA fine-tuning outperforms full-training-set fine-tuning for Data Structures, Greedy Algorithms, and Sorting.
- Skill analysis: GPT-4’s pass@1 scores range from 5 to 10 across programming skills, supporting the dataset’s skill categorization to some extent.
6 Limitations
TACO’s overlap with APPS and CodeContest creates a benchmark boundary after TACO-based training. Those datasets should not be reused to evaluate models trained on TACO.
- Because some TACO problems were collected from APPS and CodeContest and reclassified, those datasets are unsuitable evaluation benchmarks after TACO training.
7 Other Application Scenarios
TACO’s algorithmic skill labels support applications beyond code generation, including code understanding, education, algorithm recommendation, and LLM-based code assistance. The supplied passages describe these uses as prospective applications of the dataset’s annotated challenges and metadata.
- Code understanding: Algorithmic competency labels can connect programming challenges with source-code semantics to support more nuanced code understanding.
- Education and learning: Annotated challenges and skill labels can support programming textbooks, curricula, practice repositories, and assignment assessment.
- Algorithm recommendation: Challenge-skill associations can support algorithm recommendation and more precise code-completion suggestions.
- LLM applications: TACO can support LLM applications including task-specific code generation, code annotation, and documentation synthesis.
A More Implementation Details
TACO’s construction required acquiring heterogeneous problem data, labels, solutions, and test cases from multiple programming platforms. The authors addressed platform-specific formats, access barriers, missing labels, image-based content, and incomplete metadata through customized parsing, targeted crawling, and manual review.
- The main acquisition challenge was collecting high-quality problem descriptions, algorithmic labels, Python 3 solutions, and test cases from diverse contest platforms.
- CodeChef: CodeChef required manual format reviews and specialized HTML parsers because identical content appeared in multiple HTML syntaxes; crawling covered all problems through April 2023.
- CodeForces: CodeForces crawling used time splits: older problems contributed URLs, tags, and difficulty levels for matching existing data and enriching algorithmic labels.
- HackerRank: HackerRank data required a platform-specific parser and two-stage processing to convert SVG images containing text or formulas into extractable content.
- GeeksforGeeks: GeeksforGeeks’ anti-crawling mechanisms and irregular metadata required regular-expression templates, manual supplementation, and review to preserve problem information.
- Other platforms: Targeted crawling of CodeWars, Kattis, and LeetCode supplied missing algorithmic tags and URLs, while picture_num recorded image-based problem information such as graphs and trees.