PaperlayerView exact revision on arXiv

Source-linked AI summary

StarCoder 2 and The Stack v2: The Next Generation

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, Harm de Vries

arXiv:2402.19173v1cs.SEcs.AI

TL;DR

Code LLMs need continued development and evaluation across programming tasks and model scales. This paper introduces StarCoder2 and The Stack v2, finding that the 3B and 15B models are state-of-the-art within their size classes.

  • Problem

    Code LLMs have emerged as powerful assistants for writing and editing code, motivating continued development and evaluation across programming tasks and model scales.

    Source: §1 · p. 2

  • Method

    The authors construct The Stack v2 from Software Heritage and curated code-related sources, then clean, filter, and use it to train 3B, 7B, and 15B StarCoder2 models.

    Source: §1 · p. 2, §4 · p. 16

  • Results

    StarCoder2-3B and StarCoder2-15B are state-of-the-art within their respective size classes across code completion, editing, and reasoning evaluations.

    Source: §10 · p. 39

  • Takeaways & Limitations

    StarCoder2-3B and StarCoder2-15B offer strong Code LLM performance at their respective parameter scales.

    Source: §10 · p. 39

    StarCoder2-7B has the highest insecure program rate among similarly sized models, except for StarCoder2-3B.

    Source: §7.7 · p. 34

Abstract

from arXiv · show

The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size. Although DeepSeekCoder- 33B is the best-performing model at code completion for high-resource languages, we find that StarCoder2-15B outperforms it on math and code reasoning benchmarks, as well as several low-resource languages. We make the model weights available under an OpenRAIL license and ensure full transparency regarding the training data by releasing the SoftWare Heritage persistent IDentifiers (SWHIDs) of the source code data.

1 Introduction

This report presents BigCode’s open development of The Stack v2 and StarCoder2, motivated by the expanding role of Code LLMs and the need for transparent training data. It introduces a 900B+ token dataset and 3B, 7B, and 15B models, with results showing strong performance across code, reasoning, mathematics, and low-resource-language benchmarks.

Motivation

  • Code LLMs increasingly support software development beyond code generation, including implementation, quality assurance, debugging, maintenance, and migration.GitHub Copilot had over 1.3 million paying subscribers and more than 50,000 enterprise organisations as of January 30, 2024.Source: §1 · p. 2

Openness

  • Open-weight models enable local use and fine-tuning, but undisclosed training data limits scrutiny of provenance, bias, toxicity, benchmark contamination, and reproducibility.Fully open projects such as OLMo, Pythia, and BLOOM release training data, frameworks, and evaluation suites.Source: §1 · p. 2

BigCode background

  • BigCode’s open collaboration previously released The Stack v1, SantaCoder, and StarCoder, whose 15B-parameter model was the best open-access Code LLM at release.The Stack v1 contained 6.4 TB of permissively licensed code in 384 languages and included opt-out and provenance-governance tools.Source: §1 · p. 2

The Stack v2 and training

  • The Stack v2 contains 900B+ unique tokens from Software Heritage and other curated sources, processed with deduplication, quality filtering, PII redaction, malicious-code removal, and opt-out handling.The dataset is 4× larger than the first StarCoder dataset, and StarCoder2 uses two-stage training with 4k then 16k context windows.Source: §1 · p. 2

Benchmark results

  • StarCoder2-3B outperforms similarly sized StableCode-3B and DeepSeekCoder-1.3B on most benchmarks and matches or surpasses StarCoderBase-15B, while StarCoder2-7B beats CodeLlama-7B but trails DeepSeekCoder-6.7B.The report does not determine why StarCoder2-7B performs less strongly than the 3B and 15B models relative to their sizes.Source: §1 · p. 3
  • StarCoder2-15B significantly outperforms CodeLlama-13B, matches or surpasses CodeLlama-34B, and exceeds DeepSeekCoder-33B on reasoning, mathematics, and several low-resource languages.DeepSeekCoder-33B remains best on code completion for high-resource languages, while StarCoder2-15B matches or surpasses it for D, Julia, Lua, and Perl.Source: §1 · p. 3

2 Data Sources

This section describes the diverse, curated data sources assembled for The Stack v2, spanning deduplicated source code, programming materials, structured code representations, math and coding datasets, and filtered natural-language text. It also covers the filtering, transformation, licensing, and de-duplication processes used to improve data quality and coverage.

2.1 Source Code

  • The Stack v2 uses Software Heritage’s deduplicated graph archive, retaining latest main-branch revisions, reconstructing repository paths, and downloading files under a 10MB compressed-size limit.Repositories are deduplicated by content hashes, with main branches identified from GHArchive metadata or branch names.Source: §2.1 · p. 3
  • Source-code curation includes license detection, permissive and unlicensed files, classifier-based language identification, community inspection, and quality filters, yielding 619 programming languages.The pipeline detects 658 languages initially, excludes 130 extensions and 39 languages during inspection, and excludes commercial and copyleft-licensed code.Source: §2.1 · pp. 3–5

2.2 Github Issues

  • GitHub issues are cleaned by removing automated, short, bot-generated, and low-quality conversations, while retaining qualifying single-user discussions and anonymizing participants.The filters remove 17% for message-quality constraints, 3% for bot comments, and 38% of remaining issues for conversation-quality criteria.Source: §2.2 · pp. 6–7

2.3 Pull Requests

  • Pull requests combine GHArchive review events with Software Heritage code, reconstructing diffs from base and head files before extensive PR, file, text, comment, and repository-level filtering.The collection contains approximately 300M PRs, including 215M closed PRs from around 24M repositories, before subsequent filtering and subsampling.Source: §2.3 · pp. 7–8

2.4 Notebooks

  • The notebook corpus combines Jupyter notebooks from Software Heritage with Apache 2.0 Kaggle notebooks, processed into script and structured formats.Jupyter processing preserves Markdown structure, while Kaggle provides Markdown and code cells but no output cells.Source: §2.4 · p. 8, §2.4.1 · p. 8, §2.4.2 · p. 8

2.4.1 Jupyter Notebooks

  • Jupyter processing converts 11M notebooks into 4M scripts after parsing and near-deduplication, and produces 4.6M structured notebooks as code-text instruction pairs with outputs.Language metadata or Guesslang identifies notebook languages, while structured examples retain preceding Markdown and formatted cell outputs or an <empty_output> token.Source: §2.4.1 · p. 8

2.4.2 Kaggle Notebooks

  • Cleaning drops notebooks shorter than 100 characters, with syntax errors, templated openings, or near-duplicates; these steps remove 18%, then near-duplicate removal eliminates 78%, leaving 580k notebooks.Metadata is available for 42% of notebooks, while Kaggle API retrieval successfully retrieves datasets for 8.6% of notebooks; retrieved datasets provide schema and sample-row information.Source: §2.4.2 · pp. 8–9

2.5 Documentation

  • Documentation is collected from package managers, PDFs, websites, and existing web crawls, then converted to text or Markdown, near-deduplicated, and assembled into cohesive documents.Website processing extracts HTML content, removes recurring navigation, clusters similar pages with MinHash at a 0.7 threshold, and preserves crawl order.Source: §2.5 · pp. 9–10
  • The documentation data includes 3,541 free programming books in multiple languages, with document languages identified through rules or guesslang9.The books were downloaded as PDFs and converted to text before parsing.Source: §2.5 · p. 10

2.6 Intermediate Representations

  • LLVM pairs source code with intermediate representations to improve understanding of low-resource programming languages, using self-contained compilable solutions from about 4M sources.LLVM was selected for its broad adoption and availability across compiler frontends; most pairs use size-optimized IR, while 20% use performance-optimized IR.Source: §2.6 · pp. 10–11

2.7 LHQ19

  • LHQ19 contributes curated math, coding, multilingual programming, and proof datasets, including 110M+ DeepMind Mathematics examples and over 200k MultiPL-T examples.It also includes APPS, Code Contest, GSM8K variants, Rosetta Code, and proofsteps datasets for Lean and Isabelle.Source: §2.7 · p. 12

2.8 Other Natural Language Datasets

  • Other natural-language sources include 11.4M filtered Stack Overflow questions, roughly 30B ArXiv tokens, 6B Wikipedia tokens, and almost 15B OpenWebMath tokens.Stack Overflow quality filtering combines Llama-2 scoring, classifier-based question-answer matching, and a 0.1 average-score threshold.Source: §2.8 · pp. 12–13

3 Preprocessing Pipeline

The preprocessing pipeline applies source-dependent deduplication, PII redaction, benchmark decontamination, malware removal, and opt-out deletion. It uses specialized matching, anonymization, scanning, and governance procedures across the training data.

3.1 Removing Near-Duplicates

  • Near-duplicate removal uses SantaCoder’s MinHash and LSH pipeline with 5-grams and 0.7 Jaccard similarity, retaining one file per group while prioritizing repository context.Files from repositories with more stars or forks are preferred, with latest commit date as a tiebreaker; natural-language datasets are not deduplicated.Source: §3.1 · p. 13

3.2 PII Redaction

  • StarPII redacts names, emails, keys, passwords, IP addresses, and usernames from code and conversational sources, with source-specific entity coverage and participant-counter pseudonyms.References to participating usernames are removed, while non-participating users’ mentions remain unaffected.Source: §3.2 · p. 13

3.3 Decontamination

  • Decontamination removes benchmark files or text matching HumanEval, MBPP, APPS, GSM8K, and DS1000 content, improving recall by stripping whitespace during matching.Docs, LHQ, arXiv, and Wikipedia are excluded from this step.Source: §3.3 · pp. 13–14

3 Preprocessing Pipeline

  • The pipeline applies deduplication, PII redaction, benchmark decontamination, malware removal, and opt-out deletion differently across data sources, as summarized in Table 3.Not all preprocessing steps apply to every source.Source: §3 · p. 13, Table 3 · p. 14

3.4 Malware Removal

  • 59,442 files, or 0.009% of 654M files, are removed as possible malware using ClamAV 1.2 with additional SaneSecurity signatures while excluding high-false-positive signatures.The scan covers source code, pull requests, notebooks, and issues.Source: §3.4 · p. 14

3.5 Removing Opt-outs

  • After a November 20, 2023 opt-out deadline, the project removed 1,561 repositories associated with 91 users and organizations, totaling 22,066 source-code files.The process updated the “Am I in the stack” governance tool and excluded issues and pull requests from the file count.Source: §3.5 · p. 14

4 Data Composition

StarCoder2 uses model-size-specific data compositions, pairing a 17-language subset with smaller models and all 619 programming languages with the 15B model. The resulting datasets contain 622B+, 658B+, and 913B+ unique tokens for the 3B, 7B, and 15B models, respectively.

Model capacity

  • The 3B and 7B models use a 17-language SWH subset, while the 15B model uses all 619 programming languages to match dataset diversity to model capacity.The 3B model additionally excludes OpenWebMath, Wikipedia, and Arxiv from supplementary data.Source: §4 · pp. 14, 16

Downsampling languages

  • Java and JavaScript are downsampled to 200GB, markdown is retained at 254GB, HTML is reduced to 100GB, and JSON, XML, and YAML are subsampled to 8GB.These choices aim to balance high-resource languages and retain documentation-rich markdown while limiting webpage-heavy HTML and other formats.Source: §4 · pp. 14–15

The-stack-v2-train-smol

  • 525B+ unique tokens remain in the small repository-context dataset despite restricting it to 17 widely used programming languages and selected documentation and configuration languages.Each example is a full repository with files arranged in random order.Source: §4 · p. 15

The-stack-v2-train-full

  • The full repository-context dataset covers all 619 programming languages, adding 600+ languages and about 250B tokens for 775B+ total tokens.Both the small and full variants are compiled from Software Heritage source code with files arranged in random order within complete repositories.Source: §4 · p. 15

Data composition per model size

  • 913B+ unique tokens train the 15B model, compared with 658B+ for 7B and 622B+ for 3B; the 15B dataset is 4× StarCoderBase’s training-set size.The 15B composition combines the full SWH dataset with all supplementary sources, whereas the 3B composition excludes three supplementary sources.Source: §4 · p. 16

5 Data Formatting

The data-formatting scheme serializes diverse code, collaboration, notebook, discussion, and representation-translation sources with dedicated metadata, delimiters, sentinel tokens, and probabilistic variants. Source code receives repository context and file-level infilling, while other sources preserve their structures, comments, outputs, votes, and translation directions.

5.1 Source Code

  • Source code groups files by repository in random order, optionally prepends repository and path metadata, and applies repo-context file-level FIM with 50% selection and transformation probabilities.Metadata is omitted in the alternative format, and FIM does not modify repository metadata.Source: §5.1 · p. 16

5.2 Pull Requests

  • Pull requests serialize titles, descriptions, base files or changes, head diff hunks, commit discussions, reviews, comments, statuses, and merge outcomes in structured blocks.Base files are shown entirely with 0.2 probability or as localized changes with surrounding context; diff hunks include 3–10 randomly selected context lines.Source: §5.2 · pp. 16–17

5.3 GitHub Issues

  • GitHub issues use sentinel tokens for issue starts, titles, anonymized participants, comment turns, and optional closure markers.Comment and speaker identifiers distinguish conversation turns, using participant counters such as username_1.Source: §5.3 · p. 18

5.4 Notebooks

  • Notebooks are formatted as scripts or structured cell sequences, while Kaggle metadata, dataset schemas, text, code, outputs, and final scripts receive specialized tokens.Structured Jupyter notebooks preserve text, code, and outputs; structured Kaggle notebooks omit output cells, and dataset information is included for 42% of samples and schema details for 8.6%.Source: §5.4 · pp. 18–19

5.5 StackExchange

  • StackExchange concatenates each question with randomly ordered answers, preserving answer upvotes and selected-answer status without conversation titles.The format uses issue-like comment delimiters and anonymized participant identifiers.Source: §5.5 · p. 19

5.6 Intermediate Representations

  • Intermediate-representation examples are split evenly between code-to-intermediate and intermediate-to-code translation, using size-optimized representations 80% of the time and performance-optimized ones 20%.Separate sentinel tokens indicate translation direction.Source: §5.6 · p. 19

6 Model architecture and training details

StarCoder2 modifies StarCoderBase’s architecture and tokenizer, then uses dedicated base and long-context training procedures. Reported training emissions total 16,107.01 kgCO2eq for 3B and 29,622.83 kgCO2eq for 7B, while 15B estimates are pending.

6.1 Model Architecture

  • RoPE replaces learned positional embeddings, while GQA replaces MQA with 2, 4, and 4 key-value heads for the 3B, 7B, and 15B models.The RoPE base period is θ = 1e5, and the change was supported by preliminary ablation gains; other architectural hyperparameters are summarized in Table 6.Source: §6.1 · p. 20

6.2 Tokenizer

  • The tokenizer is byte-level BPE with a 49,152-token vocabulary, digit-splitter and GPT-2 regex pre-tokenization, retaining sentinel tokens from Table 5.Increasing the vocabulary to 100K did not improve preliminary performance.Source: Table 5 · p. 20, §6.2 · p. 21

6.3 Training Details

  • 16,384-token long-context pretraining uses 200B tokens, a 4,096-token sliding window, FlashAttention-2, increased RoPE θ, and the base optimizer configuration.Base models use 4,096-token sequences, Adam with β1 = 0.9, β2 = 0.95, ϵ = 10^-8, weight decay 0.1, no dropout, cosine decay, and 1,000 warmup iterations.Source: §6.3 · p. 21

6.4 CO2 Emissions

  • CO2 estimates use the Machine Learning Impact calculator and total GPU hours for base training, extrapolating long-context emissions by token count.The 3B estimate uses 97,120 A100 SXM4 80 GB hours, while 7B uses 145,152 H100 hours.Source: §6.4 · pp. 21–22
  • 16,107.01 kgCO2eq is estimated for 3B training and 29,622.83 kgCO2eq for 7B, including long-context additions of 1,111.68 and 1601.23 kgCO2eq, respectively.The 15B model’s emissions estimates were not yet available.Source: §6.4 · p. 22

7 Evaluation

This evaluation compares StarCoder2 with open Code LLMs across code completion, bug fixing, editing, repository-level completion, and math and code reasoning benchmarks. It examines performance across tasks, languages, and prompting formats, while also identifying limitations in fill-in-the-middle performance, security vulnerability rates, and harmful-generation differences.

7.1.1 HumanEval, MBPP, and EvalPlus; 7.1.2 MultiPL-E: Multilingual Code Completion; 7.1.3 DS-1000: Data Science Tasks in Python

  • The evaluation uses robustified EvalPlus tests, multilingual translation through MultiPL-E, library-based DS-1000 tasks, and prompt-sensitive bug-fixing and editing protocols.EvalPlus expands HumanEval and MBPP with 80× and 35× more tests, while the original benchmarks mainly exercise basic Python built-ins.Source: §7.1.1 · pp. 23–24, §7.1.2 · p. 24, §7.1.3 · p. 25

7.1.1 HumanEval, MBPP, and EvalPlus; 7.1.3 DS-1000: Data Science Tasks in Python; 7.3 Math Reasoning

  • StarCoder2-3B is the strongest small model on HumanEval, MBPP, their EvalPlus variants, and most DS-1000 libraries, while ranking second on GSM8K.It improves over StarCoderBase-3B by 60.2% on HumanEval+ and 32.4% on MBPP+, but StableCode-3B leads on GSM8K.Source: §7.1.1 · p. 23, §7.1.3 · p. 25, §7.3 · p. 28

7.1.1 HumanEval, MBPP, and EvalPlus; 7.1.2 MultiPL-E: Multilingual Code Completion; 7.1.3 DS-1000: Data Science Tasks in Python

  • StarCoder2-7B generally ranks second among medium models, behind DeepSeekCoder-6.7B, while outperforming StarCoderBase-7B and CodeLlama-7B on EvalPlus.Its HumanEval+ and MBPP+ scores exceed StarCoderBase-7B by 19.6% and 15.2%, and CodeLlama-7B by 16.8% and 9.6%.Source: §7.1.1 · p. 23, §7.1.2 · p. 25, §7.1.3 · p. 25

7.1 Code Completion; 7.2 Code Fixing and Editing

  • StarCoder2-15B outperforms CodeLlama-34B on MBPP and MBPP+, is best among large models on DS-1000, and does best on 16 of 18 programming languages.It meets or exceeds CodeLlama-34B on 10/18 programming languages and is competitive with models more than twice its size.Source: §7.1.1 · p. 24, §7.1.2 · p. 25, §7.1.3 · p. 25

7.1.2 MultiPL-E: Multilingual Code Completion

  • On MultiPL-E, StarCoder2-3B leads in 11/18 languages, while StarCoder2-15B leads in 16/18 and matches or exceeds CodeLlama-34B in 10/18.CodeLlama-13B is better on Go and Java, and DeepSeekCoder-33B remains stronger in four listed lower-resource languages.Source: §7.1.2 · p. 25

7.2.1 HumanEvalFix: Fixing Bugs in Six Programming Languages

  • On HumanEvalFix, the base models perform poorly with instruction prompts, but StarCoder2-15B becomes highly competitive with the Issue prompt and nearly reaches instruction-tuned DeepSeekCoder.The Issue prompt also outperforms the compared open models and indicates pull-request pretraining can be a viable alternative to commit pretraining.Source: §7.2.1 · p. 27

7.2.2 Code Editing

  • On CanItEdit, StarCoder2-3B ranks second among small models, StarCoder2-7B leads one instruction type, and StarCoder2-15B beats both the large-model baselines and CodeLlama-Instruct-34B.These results further support the Issue format as an alternative to StarCoderBase’s Commit format.Source: §7.2.2 · p. 28

7.3 Math Reasoning

  • On GSM8K, StarCoder2-15B significantly outperforms large-model baselines and even surpasses CodeLlama-34B and DeepSeekCoder-33B, while StarCoder2-7B ranks second.StarCoder2-7B is close to DeepSeekCoder-6.7B and substantially outperforms CodeLlama-7B and StarCoderBase-7B.Source: §7.3 · p. 29, Table 14 · p. 29

7.4 CRUXEval: Code Reasoning, Understanding, and Execution

  • On CRUXEval, StarCoder2-15B is the best large model, outperforms smaller baselines, and matches or exceeds extra-large models depending on the task.StarCoder2-3B beats other small models on CRUXEval-O, while StarCoder2-7B matches CodeLlama-7B but trails DeepSeekCoder-6.7B; results use pass@1 and pass@5 across CRUXEval-I and CRUXEval-O.Source: §7.4 · pp. 29–30, Table 15 · p. 30

7.5 Fill-in-the-Middle

  • On the FIM benchmark, StarCoder2-3B performs as well as StarCoderBase-15B, while StarCoder2-15B underperforms because an implementation bug made the FIM rate smaller than intended during most training.Source: §7.5 · p. 30, Table 16 · p. 31

7.6.1 RepoBench

  • On RepoBench, repository-level training lets StarCoder2 consistently outperform StarCoderBase, with StarCoder2-15B largely surpassing CodeLlama-13B and matching or exceeding CodeLlama-34B.StarCoder2-3B ranks second among smaller models, while StarCoder2-7B is competitive with CodeLlama-7B but behind DeepSeekCoder-6.7B.Source: §7.6.1 · p. 31, Table 17 · p. 32

7.6.2 CrossCodeEval

  • StarCoder2-15B achieves state-of-the-art repository-level completion among similarly sized models and exceeds models with twice its capacity in Java and C# on CrossCodeEval.Across almost all languages, model sizes, and metrics, StarCoder2 outperforms StarCoderBase, although performance varies substantially by language.Source: §7.6.2 · p. 32, Table 18 · p. 33
  • CrossCodeEval shows that StarCoder2's repository-level gains are not uniform: StarCoder2-15B trails on TypeScript, while other models lead in C# or Java.The results expose substantial performance variance across programming languages despite strong aggregate performance.Source: §7.6.2 · p. 32, Table 18 · p. 33

7.7 “Asleep at the Keyboard” Security Benchmark

  • On the security benchmark, StarCoder2 produces about 90% valid programs, but StarCoder2-7B and StarCoder2-15B have the highest insecure-program rates among similarly sized models.The higher insecure rate may reflect more valid and functionally correct completions, which are more likely to expose vulnerabilities.Source: §7.7 · pp. 33–34, Table 19 · p. 33

7.8 Measuring Harmful Generations

  • Across BOLD, WinoBias, HONEST, and RealToxicityPrompts, the evaluated code-focused models produce roughly the same amount of harmful content.The evaluation uses classifier- or lexicon-based measures of regard, gender bias, hurtfulness, and toxicity across the four benchmarks.Source: §7.8 · p. 34

8 Search Index and Attribution Tools

The work provides open inspection, membership, attribution, and search tools for the much larger and more diverse Stack v2 dataset. These tools range from lightweight exact matching to full-text fuzzy retrieval.

Section motivation

  • The authors expand data inspection and attribution tooling to support open-science access beyond releasing dataset dumps.They motivate these tools by noting the growing need for computational documentation as datasets increase in size.Source: §8 · p. 34

Membership checking tools

  • The updated “Am I in the Stack” tool checks membership at the username and repository levels for the new dataset.The dataset is four times larger than StarCoderBase and includes many additional non-code sources.Source: §8 · pp. 34–35
  • A Bloom-filter Data Portrait enables lightweight file-content matching across code and non-code sources, including documentation, textbooks, and papers.This implements the documentation-artifact recommendation of Marone & Van Durme (2023).Source: §8 · p. 35
  • The approximately 70GB system supports exact long-string checks for verbatim membership, no-code inspection, and rapid first-pass attribution of coding tools.Users can use the full search index when exact matching is insufficient.Source: §8 · p. 36

Search index

  • An Elasticsearch index over The Stack v2 source-code subset provides full-text search with fuzzy matching and retrieval.The index is made available through the BigCode search-v2 Hugging Face Space.Source: §8 · p. 36

9 Social Impact and Limitations

StarCoder2’s community-led approach emphasizes open science, responsible governance, transparent data practices, and independent auditability. The paper also identifies risks involving misuse, privacy, bias, traceability, unequal language performance, and employment disruption.

9.1 Project Approach

  • Open governance prioritizes responsible development and use of Code LLMs, even when safeguards introduce limitations that may reduce adoption or constrain future research.The project combines open scientific collaboration, model and artifact access, and the BigCode Open RAIL-M license with use restrictions and deployment guidance.Source: §9.1 · p. 37
  • Training data underwent curation, validation, decontamination, malware removal, license filtering, opt-out handling, PII removal, and transparent documentation, with substantial data sourced from Software Heritage (Software Heritage, 2024a).The model, pre-training dataset, and supporting artifacts are accessible for independent audits.Source: §9.1 · p. 37

9.2 Advancements in Code LLMs

  • Software Heritage metadata and SWHIDs can support more ethical data supply chains and software bill of materials by linking components with provenance, licenses, versions, and source locations.SWHIDs are intrinsically bound to software components and require no central registry, although tracing components remains difficult for downstream developers.Source: §9.2 · pp. 37–38, §9.3 · p. 39

9.3 Challenges and Risks

  • Open model development broadens access, enables community research, and supports auditing, but released weights are difficult to retract and can increase misuse risks without appropriate guardrails (Solaiman, 2023).Open access to weights, data, and training materials can enable low-cost malicious fine-tuning, including sophisticated polymorphic malware, despite license restrictions (Mozes et al., 2023).Source: §9.3 · p. 38
  • Generated code may expose privacy weaknesses, reproduce societal stereotypes, and favor high-resource programming and natural languages, limiting fairness and effectiveness across users and environments.Downstream users should add PII scanning and mitigation; further evaluation and guardrails are needed, while low-resource language performance and traceability remain active challenges.Source: §9.3 · pp. 38–39
  • Code LLMs may both augment and automate work: automation risks are higher for routine tasks, while augmentation benefits emphasize complex problem-solving and STEM expertise.Reported evidence includes 25–34% code acceptance among 10k+ Google-internal developers using code completion and a 52% productivity increase for ServiceNow platform skills.Source: §9.3 · p. 39

10 Conclusion

The paper introduces StarCoder2 and The Stack v2, a substantially expanded Code LLM training corpus built on Software Heritage. Evaluations show StarCoder2-3B and StarCoder2-15B are state-of-the-art in their respective size classes, with transparent data and released weights supporting reuse.

Contributions

  • StarCoder2 introduces 3B, 7B, and 15B-parameter Code LLMs trained using The Stack v2.The Stack v2 is built on the Software Heritage archive and combines cleaned, filtered, subsampled source code with other high-quality code-related datasets.Source: §10 · p. 39

Training data

  • 10× larger than its predecessor, The Stack v2 provides a 67.5 TB raw dataset and an approximately 3TB training set containing 900B+ tokens.Source: §10 · p. 39

Evaluation

  • StarCoder2-3B and StarCoder2-15B are state-of-the-art Code LLMs within their respective size classes across evaluations of completion, editing, and reasoning.Source: §10 · p. 39

Transparency

  • Releasing model weights and complete training-data transparency aims to increase trust and enable engineering teams and scientists to build on the models.Source: §10 · p. 39

A Data Curation … A.3 License detection

The data-curation procedure excludes selected file extensions and programming languages, then detects likely license files using a broad set of license-related name patterns and case-insensitive filename matching.

A.1 Excluded Extensions

  • The excluded-extension list covers numerous special-purpose, configuration, documentation, and language-associated extensions, including C++, Pascal, SQL, TeX, XML, and Yacc formats.The list also includes empty-extension cases and aliases such as project, lock, gitignore, and service.Source: §A.1 · p. 55
  • Several excluded extensions overlap with multiple file-type conventions, such as lock, d, inc, p, w, and sc, reflecting varied mappings across formats.Examples include Elixir and TOML using lock, Makefile and DTrace using d, and OpenEdge ABL and Pascal using p or related extensions.Source: §A.1 · p. 55

A Data Curation

  • The curation scope therefore combines extension-based exclusions, programming-language exclusions, and license-file identification as distinct filtering operations.The supplied curation subsections separately enumerate excluded extensions, excluded programming languages, and license-detection patterns.Source: §A.1 · p. 55, §A.2 · p. 55, §A.3 · pp. 55–56

A.2 Excluded Programming Languages

  • The excluded-language list removes heterogeneous artifacts and formats, including CSV, Diff, Go, Jupyter Notebook, Linux Kernel Module, SVG, TSV, and Unity3D Asset.It also includes project, graphics, archive, log, and data-oriented formats such as COLLADA, KiCad files, IRC logs, Pickle, and Raw token data.Source: §A.2 · p. 55

A.3 License detection

  • License detection recognizes filenames containing patterns for common licenses, including GPL, BSD, MIT, Apache, Artistic, CC0, and related legal terms.The pattern list also includes terms such as license, legal, copyright, unlicense, disclaimer, notice, README, and guidelines.Source: §A.3 · pp. 55–56
  • License-file matching is case-insensitive and permits arbitrary prefixes, suffixes, and separators around recognized names.The regular expression anchors the full filename while allowing empty or extended surrounding components separated by hyphens, underscores, periods, or spaces.Source: §A.3 · p. 56

A.4 Permissive licenses · A.5 Pull Requests

The permissive-license inventory combines SPDX-recognized identifiers with extensive ScanCode-specific identifiers, while excluding non-license agreements and disclaimers from labeling. The pull-request section reports rendering volumes by sequence length and base-file volumes for the top 20 languages.

A.4 Permissive licenses

  • The dataset recognizes a broad catalog of SPDX license IDs, including Apache, BSD, MIT, Creative Commons, and other permissive licenses.The catalog explicitly includes identifiers such as Apache-2.0, BSD-3-Clause, MIT, and CC-BY-4.0.Source: §A.4 · p. 56
  • ScanCode-specific identifiers extend permissive-license coverage across numerous project-, vendor-, historical-, and variant-specific licenses.These identifiers include many BSD, MIT, X11, software-vendor, and project-specific variants.Source: §A.4 · pp. 56–58
  • The ScanCode inventory includes specialized exceptions, attribution terms, notices, and other permissive-use variants alongside standard license families.Examples include patent disclaimers, linking exceptions, attribution licenses, government-use terms, and modified BSD or MIT variants.Source: §A.4 · p. 57
  • License labeling excludes contributor license agreements, warranty disclaimers, and other amendments classified as non-licenses.Excluded references include DCO, generic and Google CLAs, export-compliance terms, trademark terms, and warranty disclaimers.Source: §A.4 · p. 58

A.5 Pull Requests

  • The pull-request data are summarized by the volume of renderings across sequence lengths measured in characters.Table 24 reports these rendering volumes for different sequence lengths.Source: §A.5 · p. 58
  • The base-file volumes for the 20 highest-volume programming languages are listed separately.These language-specific base-file volumes appear in Table 25.Source: §A.5 · p. 58

A.6 StackOverflow … B.1 Malware removal

This section specifies evaluation and preprocessing procedures for StackOverflow and Kaggle data, and reports malware-detection summaries by signature and programming language. StackOverflow answers are scored on a 10-point rubric emphasizing relevance, completeness, clarity, and an AI-assistant perspective.

A.6 StackOverflow

  • StackOverflow evaluation asks whether a candidate answer appropriately responds to a user instruction and assigns a score on a 10-point scale.The prompt requires the evaluator to judge answer quality as an example of how an AI Assistant should respond.Source: §A.6 · p. 58
  • Scores 1–3 denote off-topic, inaccurate, incomplete, or controversial responses that fail to address key parts of the user’s query.The rubric progresses from entirely off-topic or incomprehensible answers to responses that are relevant but incomplete or partly off-topic.Source: §A.6 · p. 58
  • Scores 4–7 represent increasingly helpful answers, from limited depth or clarity to responses addressing most query aspects but possibly lacking depth or AI-assistant framing.Scores 4–6 emphasize helpfulness and coverage, while score 7 notes that an answer may resemble external content rather than an AI Assistant response.Source: §A.6 · p. 58
  • Scores 8–10 require comprehensive, clear, well-organized, and highly informative answers written from an AI Assistant’s perspective.The highest ratings allow only minor improvement areas, with score 10 describing an exemplary answer without discernible shortcomings.Source: §A.6 · p. 58
  • The evaluator must place “Score: <rating>” on the final line and briefly explain the reasoning behind the assigned rating.This defines the required output format for the StackOverflow assessment.Source: §A.6 · p. 59

A.7 Kaggle Notebooks templates

  • Kaggle preprocessing removes specified templates appearing at the beginning of notebooks, including standard imports, input-directory instructions, and file-listing code.The listed templates include Kaggle Docker-image text, common NumPy and pandas imports, traversal of /kaggle/input, and runtime-storage instructions.Source: §A.7 · p. 59

B.1 Malware removal

  • Malware analysis summarizes the top 10 detected malware signatures and the top 10 programming languages by potentially malicious-file count.These summaries are presented in Table 26 and Table 27, respectively.Source: Table 26 · p. 60, Table 27 · p. 60, §B.1 · p. 60

C Data Composition · C.1 TheStackV2-train-smol

TheStackV2-train-smol includes configuration languages and a broad set of configuration and dependency-management files. Its listed formats span build systems, package manifests, environment specifications, and project metadata.

C.1 TheStackV2-train-smol

  • The configuration-language list includes Ant Build System, CMake, Dockerfile, Go Module, Gradle, INI, and Java Properties.Source: §C.1 · p. 60
  • The configuration-file list includes CMakeLists.txt, Cargo.toml, DESCRIPTION, Gemfile, Makefile, Makefile.am, NAMESPACE, Package.swift, Pipfile, and build.gradle.Source: §C.1 · p. 60
  • Additional configuration formats include Maven POM and TOML.Source: §C.1 · p. 60
  • Python and TypeScript project files include pyproject.toml, requirements-dev.txt, requirements-prod.txt, requirements.in, requirements.test.txt, requirements.txt, setup.cfg, and tsconfig.json.Source: §C.1 · p. 60
  • The listed dependency and project files also include yarn.lock, build.gradle.kts, composer.json, conda.yml, configure.ac, and Docker Compose YAML files.Source: §C.1 · p. 60
  • Further formats include go.mod, package.json, pom.xml, and the configuration files docker-compose.yaml and docker-compose.yml.Source: §C.1 · p. 60

C.2 TheStackV2-train-full

This section summarizes the data volumes of language subsamples in The Stack v2 dataset.

C.2 TheStackV2-train-full

Related papers

Loading 2402.19173v1…