Source-linked AI summary
The Stack: 3 TB of permissively licensed source code
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, Harm de Vries
TL;DR
Code-LLM research lacks openly inspectable, permissively licensed pre-training data and reproducible evidence about its construction. The paper introduces and evaluates The Stack, finding that near-deduplication improves results and that permissively licensed data can match prior benchmark performance.
Problem
Code-LLM research lacks openness and transparency around development and training data, making full reproduction difficult.
Method
The paper collects GitHub source code, constructs a permissively licensed dataset, applies near-deduplication, and evaluates 350M-parameter decoders on HumanEval and MBPP.
Results
37.00% pass@100 on HumanEval and 54.69% pass@100 on MBPP are achieved by the near-deduplicated permissive-license dataset, matching or surpassing prior results.
Takeaways & Limitations
The Stack provides a shared resource for open and responsible code-LLM research, with near-deduplication identified as important for competitive text2code performance.
Takeaways & Limitations
Scraped HTML may not comply with WCAG, potentially introducing accessibility issues in generated HTML code.
Abstract
from arXiv · showhide
Large Language Models (LLMs) play an ever-increasing role in the field of Artificial Intelligence (AI)--not only for natural language processing but also for code understanding and generation. To stimulate open and responsible research on LLMs for code, we introduce The Stack, a 3.1 TB dataset consisting of permissively licensed source code in 30 programming languages. We describe how we collect the full dataset, construct a permissively licensed subset, present a data governance plan, discuss limitations, and show promising results on text2code benchmarks by training 350M-parameter decoders on different Python subsets. We find that (1) near-deduplicating the data significantly boosts performance across all experiments, and (2) it is possible to match previously reported HumanEval and MBPP performance using only permissively licensed data. We make the dataset available at https://hf.co/BigCode, provide a tool called "Am I in The Stack" (https://hf.co/spaces/bigcode/in-the-stack) for developers to search The Stack for copies of their code, and provide a process for code to be removed from the dataset by following the instructions at https://www.bigcode-project.org/docs/about/the-stack/.
1 Introduction
The paper introduces The Stack as a large, permissively licensed source-code dataset designed to improve openness, transparency, and reproducibility in code-LLM research. It releases the dataset and reports that near-deduplication improves text2code performance.
- Motivation: Code-LLM research is difficult to reproduce because development processes and training data are often undisclosed.Prior systems were released through paid APIs, commercial products, or model weights without training data.
- Motivation: Legal uncertainty around training LLMs on open-source repositories motivates greater openness about pre-training data.The paper describes disagreement over derivative works, license obligations, and copyright exceptions.
- Contributions: The released dataset is intended to make LLM development more reproducible and transparent by enabling shared, inspectable pre-training data.The authors describe collection from GitHub and present evidence that the dataset supports competitive code-LLM development.
- Contributions: The Stack contains 3.1 TB of permissively licensed source code across 30 programming languages and includes a near-deduplicated release.The dataset is publicly released through the BigCode Hugging Face organization.
- Results: Near-deduplication significantly boosts performance in all reported 350M-parameter decoder experiments and enables matching prior text2code results with permissively licensed data.The paper also reports outperforming prior models by a large margin when training on the all-license dataset.
- Data governance: Developers can request removal of their code from The Stack through the project’s data-governance process.The paper acknowledges that some developers do not want their code used for LLM pre-training.
2 Related Work
Related work includes decoder-only code LLMs, GitHub-derived pre-training datasets, function–docstring corpora, and studies showing that duplicate training data can harm performance. The paper positions The Stack against both released and private alternatives.
- Code LLMs: Decoder-only transformer models with causal language modeling objectives have been widely studied for source-code generation.Prior work generally finds that larger models are increasingly capable of synthesizing programs from natural-language descriptions.
- Datasets for pre-training code LLMs: GitHub BigQuery snapshots have supported pre-training for AlphaCode, BLOOM, CodeGen, and InCoder, with a related GitHub-Code dataset publicly available.These resources provide commonly used GitHub-derived training data for code LLMs.
- Datasets for pre-training code LLMs: CodeSearchNet extracts function–docstring pairs in Go, Java, JavaScript, Python, PHP, and Ruby for code-model pre-training.CodeBERT and CodeT5 used this corpus.
- Datasets for pre-training code LLMs: Codex and CodeGen used substantial datasets that were not fully released, limiting reproducibility of their training-data pipelines.Codex did not release its data or disclose licensing, while CodeGen withheld its private Python dataset.
- Deduplication: Recent studies report that near-duplicate or repeated training examples can reduce language-model performance.The cited work motivates deduplication as an important pre-training-data preprocessing step.
3 Dataset
The Stack combines large-scale GitHub collection with license filtering, deduplication, governance measures, and dataset analysis. The resulting permissive dataset contains 3.1 TB across 30 languages and supports code-LLM research, while retaining documented classification and coverage limitations.
- Dataset Collection: The dataset was built from 220.92M unique GitHub repository names extracted from GHArchive events between January 1st, 2015 and March 31st, 2022.GHArchive supplied repository names from public event archives, not the code repositories themselves.
- License Filtering: The permissive subset excludes copyleft licenses such as GPL, while license detection combines GHArchive metadata with go-license-detector predictions.Repositories were classified as permissive only when all detected license predictions were permissive.
- Data Governance: The dataset provides developer removal requests, initially covering all public repositories under a username, with finer-grained scopes planned for future work.Future scopes include specific repositories, files, and contribution types linked through GHArchive.
- Dataset Analysis: 3.1 TB of permissively licensed code across 30 languages was reduced to 1.4 TB after near-deduplication, a reduction of more than 50%.Near-deduplication uses tokenization, MinHash, locality-sensitive hashing, and cluster reduction on top of exact deduplication.
- Dataset Analysis: The four largest permissive-language components are HTML (746 GB), Javascript (486 GB), Java (271 GB), and C (222 GB), together exceeding 55% of dataset size.The all-license dataset contains over 29 TB, whereas permissive filtering retains roughly 10%.
- Comparison with Other Code Datasets: The Stack is more than three times larger than CodeParrot and exceeds it for every individual programming language, while covering 30 languages.CodeParrot includes copyleft files, whereas The Stack's permissive subset does not.
- Python Subset Analysis: Only 0.7% of sampled Python files failed compilation, while docstrings and comments comprised 18% of sampled volume and 94% of detected natural-language files were English.Language detection covered 10,000 Python files and was noted as imperfect because docstrings can contain code examples.
4 Experiments
The experiments evaluate 350M-parameter decoder-only transformers trained on Python dataset variants using HumanEval and MBPP. Near-deduplication substantially improves performance, enabling permissively licensed data to match prior results and outperform CodeParrot.
- Experimental setup: 350M-parameter decoder-only transformers are evaluated on HumanEval and MBPP across all-license, permissive-license, and near-deduplicated Python datasets.The study also includes CodeParrot and detects benchmark contamination by exact prompt matching.
- Near-deduplication: Near-deduplication raises permissive-license HumanEval pass@100 from 27.21% to 37.00% and MBPP pass@100 from 44.99% to 54.69%.The all-license version shows similar gains: HumanEval rises from 36.67% to 44.00%, while MBPP rises from 53.59% to 61.00%.
- Permissive-license performance: 37.00% HumanEval pass@100 after near-deduplication matches Codex and CodeGen performance using only permissively licensed source code.On MBPP, the near-deduplicated permissive dataset reaches 54.69% pass@100 and surpasses CodeGen's 51.80%.
- Comparison to CodeParrot: The released dataset reaches 37.00% HumanEval pass@100, compared with 30.37% for CodeParrot.On MBPP, the released dataset also exceeds CodeParrot, 54.69% versus 45.44% pass@100.
- Data contamination: Removing contaminated files has very little impact on text2python results.HumanEval changes from 37.00% to 36.01% for permissive-license data and from 44.00% to 45.52% for all-license data; all-license MBPP changes from 61.00% to 58.28%.
5 Conclusion and Future Work
The paper releases The Stack, a dataset of more than 3 TB of permissively licensed source code, and reports that near-deduplication supports competitive text2code results. It aims to support open, responsible, reproducible, and transparent code-LLM research while outlining future dataset improvements and removal options.
- Conclusion: The Stack contains more than 3 TB of permissively licensed source code across 30 common programming languages.The authors release both the permissively licensed files and a near-deduplicated version.
- Conclusion: Near-deduplication is an important pre-processing step for achieving competitive results on text2code benchmarks.The conclusion connects this finding to the reported HumanEval results and released dataset variants.
- Future work: Future work includes adding languages, removing PII and malicious code, and enabling developers to remove their data from the dataset.The authors also hope The Stack will support open and responsible research on Code LLMs.
6 Limitations
The Stack aims to improve access, reproducibility, and transparency for code LLM research, while acknowledging risks involving malicious code, bias, licensing, privacy, accessibility, and model scope.
- The Stack is developed within BigCode's Open Science effort for responsible code LLM development.
- Comments may contain harmful or offensive language that models could learn, despite the absence of demographic information in the collected code.
- C and Javascript are overrepresented, while SQL, Batchfile, and TypeScript are less likely to be permissively licensed; Python comments and docstrings are 96% English.
- Potential risks include harmful code generation, over-reliance on generated code, and broader negative effects from increased accessibility.
- The dataset may contain malware, incorrectly classified licenses, personally identifiable information, and non-WCAG-compliant scraped HTML.
- The reported experiments use smaller Python models, leaving larger models and other programming languages for future research.
A Permissive Licenses
The experiments define a permissive-license dataset using specified SPDX identifiers, but later corrected the classification of weak copyleft licenses.
- The permissive-license experiments use the SPDX identifiers BSD-3-Clause-No-Nuclear-License-2014 and MPL-2.0-no-copyleft-exception.
- MPL, LGPL, and EPL files were erroneously labeled permissive because they are weak copyleft licenses.
- The authors removed the weak copyleft files and planned to release an updated version of The Stack.
- Weak copyleft data comprised below 0.5% of the Python subset, so the authors expected the experimental findings to remain unchanged.
B The Stack v1.1
The Stack v1.1 adopts a new license-classification process, expands language coverage, and reports dataset statistics across license and deduplication splits.
- The Stack v1.1 relies on the Blue Oak Council to classify 193 permissive licenses.
- The updated collection process covers 370 programming languages and reports statistics for 30 popular languages.
- Table 7 organizes The Stack v1.1 by all-license, permissive-license, and near-deduplicated permissive-license splits.
- The listed SPDX identifiers include CNRI-Python-GPL-Compatible, BSD-3-Clause-Attribution, BSD-3-Clause-No-Nuclear-Warranty, BSD-3-Clause-No-Nuclear-License-2014, BSD-3-Clause-No-Nuclear-License, and BSD-3-Clause-Modification.
C Excluded file extensions
The dataset collection excludes numerous non-source and binary file extensions to focus the corpus on source-code content.
- The exclusion list was partly taken from the EleutherAI GitHub downloader.
- Excluded extensions include archives, binaries, media, fonts, databases, serialized objects, and other non-source artifacts.
D Included programming language extensions
This section lists programming-language file extensions included in the dataset, drawing the extension list from an external source and spanning diverse language and tooling ecosystems.
- The programming-language extension list is taken from a referenced GitHub gist.The passage identifies the source as a gist by ppisarczyk.
- The list includes extensions associated with languages and environments such as ABAP, Ada, Assembly, AWK, Bash, C, C#, and C++.
- Additional entries cover Fortran, F#, Go, GLSL, GNUplot, GraphQL, Groovy, HCL, HLSL, HTML, and related formats.