Source-linked AI summary
StarCoder 2 and The Stack v2: The Next Generation
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, Harm de Vries
TL;DR
Code LLMs have rapidly emerged as code-writing assistants, motivating responsible development and stronger training resources. The paper introduces The Stack v2 and StarCoder2, then evaluates models across code completion, editing, and reasoning; StarCoder2-3B and StarCoder2-15B are state-of-the-art within their size classes.
Problem
Code LLMs are rapidly emerging as assistants for writing and editing code, creating a need for responsible development and high-quality training resources.
Method
The authors build The Stack v2 from the Software Heritage archive and supplementary code-related sources, then train 3B, 7B, and 15B StarCoder2 models and evaluate their completion, editing, and reasoning capabilities.
Results
StarCoder2-3B and StarCoder2-15B are state-of-the-art within their respective size classes; StarCoder2-15B also matches or outperforms DeepSeekCoder-33B on reasoning, mathematics, and several low-resource languages.
Takeaways & Limitations
The results position StarCoder2-3B and StarCoder2-15B as leading Code LLMs for their respective parameter ranges.
Takeaways & Limitations
StarCoder2-7B underperforms DeepSeekCoder-6.7B, and StarCoder2-7B and StarCoder2-15B have the highest insecure program rates among similarly sized models.
Abstract
from arXiv · showhide
The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size. Although DeepSeekCoder- 33B is the best-performing model at code completion for high-resource languages, we find that StarCoder2-15B outperforms it on math and code reasoning benchmarks, as well as several low-resource languages. We make the model weights available under an OpenRAIL license and ensure full transparency regarding the training data by releasing the SoftWare Heritage persistent IDentifiers (SWHIDs) of the source code data.
1 Introduction
StarCoder2 extends BigCode’s open Code LLM effort with a larger, more diverse training corpus and evaluations across code benchmarks. The 3B and 15B models show strong comparative performance, while the 7B model trails a similarly sized competitor.
- Motivation: BigCode develops StarCoder2 through an open-scientific collaboration and builds The Stack v2 from Software Heritage’s source-code archive plus curated code-related data.The added sources include GitHub issues, pull requests, notebooks, documentation, and natural-language datasets related to math, coding, and reasoning.
- Approach: StarCoder2 models are trained with 3.3 to 4.3 trillion tokens and evaluated across a suite of Code LLM benchmarks.
- Results: StarCoder2-3B outperforms similarly sized StableCode-3B and DeepSeekCoder-1.3B on most benchmarks and matches or surpasses StarCoderBase-15B.
- Results: StarCoder2-15B significantly outperforms comparable-size CodeLlama-13B and matches or outperforms CodeLlama-34B.It also matches or outperforms DeepSeekCoder-33B on low-resource languages, code-execution reasoning, and mathematics, although DeepSeekCoder-33B leads high-resource code completion.
- Results: StarCoder2-7B outperforms CodeLlama-7B but remains behind DeepSeekCoder-6.7B.The report’s authors state that they do not know why its performance is weaker relative to the 3B and 15B models.
2 Data Sources
The Stack v2 combines Software Heritage repositories with curated code, documentation, notebook, issue, pull-request, and intermediate-representation sources. Its collection pipeline applies language detection, license assignment, quality filtering, and source-specific extraction procedures.
- Source Code: The Stack v2 draws on Software Heritage’s deduplicated archive and extracts recent main-branch revisions from GitHub repositories.The source archive is represented as a Merkle DAG, and repository extraction retains the latest revision of the main branch.
- Source Code: 96.93% of repositories lack repository-level license information, so file-level licenses are detected with ScanCode and propagated by base path.Files are then classified as permissively licensed, non-permissively licensed, or unlicensed using the assignment algorithm.
- Source Code: The Stack v2 includes permissively licensed and unlicensed files but excludes commercial and copyleft-licensed code.
- Source Code: 658 unique programming languages are detected in TheStackV2-dedup with go-enry, replacing Stack v1’s file-extension language detection.
- Documentation: The documentation collection crawls package-manager and curated websites, while Table 1 compares Stack v1, Stack v2, and StarCoder2-15B data across 32 programming languages.
- Intermediate Representations: StarCoder2 augments source code with paired intermediate representations, using LLVM because it is widely available and adopted across programming-language compiler frontends.
3 Preprocessing Pipeline
The preprocessing pipeline combines deduplication, privacy protection, benchmark decontamination, malware scanning, and opt-out deletion. These operations are applied selectively across data sources and remove low-quality or sensitive material before training.
- Pipeline Overview: The pipeline includes deduplication, PII redaction, benchmark decontamination, malware removal, and opt-out deletion requests.Processing steps vary by data source and are summarized per source in Table 3.
- Deduplication: Deduplication uses MinHash fingerprints and locality-sensitive hashing to group similar files, retaining one file per duplicate group while prioritizing repository context.Higher star and fork counts, or a newer commit date as a tiebreaker, guide file selection.
- Privacy: StarPII redacts names, emails, keys, passwords, IP addresses, and usernames from several code and discussion sources.Username anonymization substitutes conversation-specific participant counters while preserving speaker identity.
- Decontamination: Training data is decontaminated by removing files containing benchmark docstrings, solutions, questions, or prompts from HumanEval, MBPP, APPS, GSM8K, and DS1000.Whitespace removal during matching increases recall, while docs, LHQ, arXiv, and Wikipedia are excluded from this step.
- Malware Removal: 59,442 files, or 0.009% of 654M files, are removed after malware scanning with ClamAV and additional SaneSecurity signatures.
- Opt-Out Deletion: Opt-out processing removes 1,561 repositories associated with 91 users and organizations, totaling 22,066 source-code files.
4 Data Composition
The Stack v2 combines Software Heritage code with curated supplementary sources, using model-size-specific datasets that balance language diversity and model capacity. The full variant spans 619 programming languages, while the small subset already provides over 525B unique tokens.
- Data composition per model size: Model-size-specific data composition tailors dataset diversity to model capacity, because smaller models have limited capacity and languages can compete for it.The 3B, 7B, and 15B models use distinct data compositions.
- Downsampling languages: The source-code datasets preserve natural language distributions while downsampling high-volume formats and retaining 254GB of markdown data.HTML is reduced to 100 GB, while JSON, XML, and YAML are subsampled to 8GB.
- Repository-context: Source code is organized into repository-context-aware examples, with each example containing a full repository whose files are arranged in random order.Two Software Heritage dataset variants are constructed: small and full.
- The-stack-v2-train-smol: 525B+ unique tokens are obtained despite limiting the small dataset to 17 widely used programming languages plus documentation and configuration languages.The small variant includes curated documentation and configuration languages alongside the selected programming languages.
- The-stack-v2-train-full: 619 programming languages contribute to the full variant, adding 600+ languages and approximately 250B tokens for 775B+ total tokens.The full variant uses all 619 programming languages.
5 Data Formatting
The formatting pipeline converts repositories, pull requests, issues, notebooks, StackExchange data, and intermediate representations into structured token sequences. It uses repository context, sentinel tokens, anonymization, localized diffs, and controlled source-to-intermediate translation directions.
- 5.1 Source Code: Repository-context formatting groups files from the same repository in random order, replacing StarCoder1’s random file-context training arrangement.Repository metadata and file paths are prepended with 50% probability; otherwise, files use a metadata-free format.
- 5.1 Source Code: Fill-in-the-middle training applies repo-context file-level FIM with 50% repository-candidate probability and 50% chunk-transformation probability.Repository metadata is excluded from the FIM transformation.
- 5.2 Pull Requests: Pull-request diffs use changed-file hunks with 3-10 randomly selected context lines before and after each change, while base files are included entirely with 0.2 probability.The union of file-line changes across commits is used, and up to 32 lines may be added around changes.
- 5.2 Pull Requests: Pull requests are rendered as structured sequences combining titles, descriptions, anonymized participants, review events, comments, base files, and localized diff hunks.Closed pull requests receive a final block indicating closed status and whether they were merged.
- 5.3 GitHub Issues: GitHub issues encode titles, anonymized speakers, comment turns, and optional closure using sentinel tokens.The format distinguishes comments and anonymized participant identifiers within each conversation.
- 5.4 Notebooks: Jupyter and Kaggle notebooks are represented with sentinel-delimited text, code, and outputs, while Kaggle metadata may include dataset descriptions and schema information.Associated dataset titles and descriptions are prepended for 42% of Kaggle notebooks, and schema information for 8.6%.
- 5.5 StackExchange: StackExchange examples begin with a question and append randomly ordered answers with upvote scores and selected-answer markers.Conversation titles are not available for this dataset.
- 5.6 Intermediate Representations: Intermediate-representation training splits examples evenly between code-to-intermediate and intermediate-to-code translation, using size-optimized representations 80% of the time.Separate sentinel tokens indicate translation direction.
6 Model architecture and training details
StarCoder2 introduces architectural changes, a fixed tokenizer, extensive base and long-context training, and explicit emissions accounting. Training documentation also records model-specific carbon estimates and the missing 15B emissions estimate.
- 6.1 Model Architecture: Rotary Positional Encodings replace learned positional embeddings, while Grouped Query Attention replaces Multi-Query Attention with low key-value-head counts.The RoPE base period is θ = 1e5; key-value heads are kept relatively low across model sizes.
- 6.2 Tokenizer: The models use a byte-level BPE tokenizer with a 49,152-token vocabulary, including sentinel tokens, plus digit splitting and GPT-2 regex pre-tokenization.Increasing the vocabulary to 100K did not improve performance in preliminary experiments.
- 6.3 Training Details: Base models train with sequence length 4,096 using Adam, cosine learning-rate decay after 1,000 warmup iterations, weight decay 0.1, and no dropout.StarCoder2-15B was early stopped after 1M iterations instead of its scheduled 1.1M.
- 6.3 Training Details: Long-context pre-training adds 200B tokens at context length 16,384 with a sliding window of 4,096 and FlashAttention-2.The optimizer configuration remains the same while RoPE θ is increased.
- 6.4 CO2 Emissions: CO2 estimates are calculated from total GPU hours for base training and extrapolated to long-context fine-tuning according to token counts.The calculation uses the Machine Learning Impact calculator.
- 6.4 CO2 Emissions: 16,107.01 kgCO2eq is the estimated total for StarCoder2-3B after adding 1,111.68 kgCO2eq from long-context fine-tuning.The base-model estimate is 14,995.33 kgCO2eq.
- 6.4 CO2 Emissions: 29,622.83 kgCO2eq is the estimated total for StarCoder2-7B after long-context fine-tuning.The base-model estimate is 28,021.6 kgCO2eq, and the fine-tuning stage adds 1601.23.
- 6.4 CO2 Emissions: The paper will soon be updated with emissions estimates for StarCoder2-15B.The 15B estimate is not yet provided in this section.
7 Evaluation
Across code-completion, editing, reasoning, multilingual, and repository-level benchmarks, StarCoder2 generally outperforms comparable open models. StarCoder2-15B is especially competitive with models more than twice its size, while performance varies by model size, language, and task.
- HumanEval, MBPP, and EvalPlus: 60.2% and 32.4% improvements over StarCoderBase-3B occur on HumanEval+ and MBPP+, respectively.
- HumanEval, MBPP, and EvalPlus: 46.3 versus 37.8 on HumanEval separates StarCoder2-15B from CodeLlama-13B, while EvalPlus gains are 47.7% and 17.0% over StarCoderBase-15B and CodeLlama-13B.
- StarCoder2-15B is the strongest large model across DS-1000, HumanEvalFix, CanItEdit, and CRUXEval, often outperforming larger or instruction-tuned alternatives.
- Repository-level training consistently improves StarCoder2 over StarCoderBase, while Issue prompting provides a viable alternative to Commit prompting.
8 Search Index and Attribution Tools
The paper extends open-data inspection with membership checking, attribution, and search tools for The Stack v2. These tools support exact matching, lightweight inspection, and fuller fuzzy retrieval.
- Membership checking tools: 4 times larger than StarCoderBase, The Stack v2 adds non-code sources that make accessible data inspection more important.The expanded dataset includes documentation, textbooks, and papers alongside source code.
- Membership checking tools: Bloom filters power a Data Portrait that matches file contents, including prose sources such as documentation, textbooks, and papers.The system supports username- and repository-level inspection and exact matching of long strings.
- Attribution: The 70GB inspection system checks whether user material occurs verbatim in the data and enables rapid first-pass attribution checks for coding tools.It provides exact matches for long strings, while the full search index supports additional analysis.
- Search index: The Elasticsearch search index supports full-text fuzzy matching and retrieval over The Stack v2 source-code subset.It is made available through the BigCode search-v2 interface.
9 Social Impact and Limitations
The project frames StarCoder2 as open, auditable, and responsibly governed, while acknowledging risks from openness, privacy, security, bias, and uneven language representation. These risks require continued safeguards and downstream evaluation.
- Ethical data sourcing: The project applies curation, validation, decontamination, malware removal, license filtering, opt-out processing, PII removal, and dataset documentation.It states that training-data transparency was provided, including for data sourced under license from Software Heritage.
- Auditable: StarCoder2’s model, pre-training dataset, and supporting artifacts are accessible for independent audits.This auditability is presented as part of the project’s open-science approach.
- Openness and safety risks: Open development enables community research, democratized access, and audits, but greater access can increase misuse risks and make released weights difficult to retract.The paper positions responsible AI practices and guardrails as central to open model development.
- Security: Open access to weights, data, and training artifacts lets actors run or fine-tune the model cheaply, and license restrictions cannot prevent malicious attempts to cause harm.The paper identifies this as a security risk of open scientific model releases.
- Societal bias: Generated code can reflect stereotypes involving gender, race, emotion, class, and names, requiring further evaluation and guardrail mitigations.The limitation concerns downstream use cases rather than a single benchmark result.
- Representation bias: Training data are richer for popular languages than niche languages, and predominantly English data may limit fairness and effectiveness across coding tasks and environments.The paper notes that non-English-context code generation is not guaranteed to work equally well.
10 Conclusion
The paper introduces StarCoder2 and The Stack v2, combining a substantially expanded Software Heritage-based corpus with models trained and evaluated for code generation, editing, and reasoning. The reported evaluations place the 3B and 15B models among the strongest models in their size classes while transparent release is intended to support trust and reuse.
- Conclusion: 67.5 TB raw data became an approximately 3TB training set containing 900B+ tokens after cleaning, filtering, subsampling, and adding code-related datasets.The Stack v2 is described as ten times larger than its predecessor.
- Conclusion: StarCoder2 models with 3B, 7B, and 15B parameters achieved state-of-the-art performance within their respective size classes across code completion, editing, and reasoning evaluations.The conclusion summarizes extensive Code LLM evaluations across these capabilities.
- Conclusion: Releasing model weights with transparent training-data information is intended to increase trust and empower engineering teams and scientists to build upon the work.The Stack v2 is built on the foundations of the Software Heritage archive.
A Data Curation
The data-curation appendix provides license-detection resources centered on recognized license filenames and regular-expression matching. It also lists file-extension categories included in the curation process.
- Data curation: The appendix enumerates heterogeneous file-extension categories, including project files, configuration formats, notebooks, and data or media formats.The supplied list includes formats such as CSV, G-code, Jupyter Notebook, SVG, TSV, and WebVTT.
- License detection: The license detector defines a list of recognized license-related filenames, including readme, guidelines, and notice.The list also includes names such as legal, copying, unlicense, and common license abbreviations.
- License detection: A case-insensitive regular expression matches filenames built from the recognized license-name list with optional surrounding text and separators.The expression permits prefixes, suffixes, and common punctuation or spacing around names.
A.4 Permissive licenses
This section enumerates SPDX-recognized and ScanCode-specific license identifiers, while excluding certain license amendments and agreements from labeling. It also references pull-request dataset volume tables.
- The license inventory includes SPDX-recognized identifiers such as Apache, BSD, and Artistic licenses.
- ScanCode-specific identifiers are represented using the LicenseRef-scancode naming scheme.
- Contributor license agreements, warranty disclaimers, and other amendments were not considered during license labeling.
- Table 24 reports pull-request dataset volume under sequence-length restrictions, while Table 25 lists base-file sizes and change ranges for the top 20 languages.
A.6 StackOverflow
The supplied material describes a response-evaluation rubric, Kaggle notebook template removal, and tables covering pull-request and potentially malicious-file data. These elements define evaluation and preprocessing references rather than StackOverflow-specific findings.
- Responses are scored on a 10-point scale ranging from entirely off-topic or inaccurate to exemplary and fully informative.
- Higher rubric levels emphasize helpfulness, comprehensive coverage, organization, and an AI-assistant perspective.
- The evaluation instruction requires a final line containing the assigned score followed by brief reasoning.
- Standard Kaggle notebook templates are removed when they appear at the beginning of notebooks, including boilerplate imports and input-directory traversal.
- Tables identify top detected malware signatures and languages ranked by potentially malicious files.
C Data Composition
The data composition lists configuration languages and configuration filenames included in the dataset. The inventories span build systems, package manifests, dependency files, and project settings.
- Configuration languages include Ant Build System, CMake, Dockerfile, Go Module, Gradle, INI, and Java Properties.
- Configuration filenames include build, dependency, packaging, and project files such as Makefile, Cargo.toml, Gemfile, and Package.swift.
- The inventory also includes Python, JavaScript, TypeScript, Kotlin, Composer, Conda, Docker Compose, Go, and Maven configuration files.
C.2 TheStackV2-train-full
This section summarizes subsampling volumes for languages in the Stack v2 dataset. Table 28 is the referenced source for the language-level volume summary.
- Table 28 summarizes subsampling volumes for languages in the Stack v2 dataset.
- The table is intended to report language-specific data volumes after subsampling.
- The supplied passage points to Table 28 for the full subsampling-volume breakdown.