Source-linked AI summary
SantaCoder: don't reach for the stars!
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, Leandro von Werra
TL;DR
Code LLM development needs responsible data handling and evidence about architecture and preprocessing choices. This report evaluates 1.1B-parameter multilingual models and finds that SantaCoder can outperform larger open-source multilingual baselines, while GitHub-star filtering harms performance and PII coverage remains incomplete.
Problem
The report addresses how to develop multilingual code LLMs responsibly while choosing architecture and preprocessing methods that preserve benchmark performance.
Method
The authors redact selected PII, ablate MQA, FIM, and four data filters, then train SantaCoder with the strongest choices on three language subsets of The Stack.
Results
SantaCoder obtains comparable or stronger MultiPL-E generation and infilling performance than InCoder-6.7B and CodeGen-Multi-2.7B despite having 1.1B parameters.
Takeaways & Limitations
More aggressive near-deduplication can improve performance, whereas filtering repositories with 5+ GitHub stars consistently decreases performance across benchmarks and languages.
Takeaways & Limitations
The first PII pipeline covers emails, IP addresses, and keys, leaving names, usernames, and passwords for future work.
Abstract
from arXiv · showhide
The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code. This tech report describes the progress of the collaboration until December 2022, outlining the current state of the Personally Identifiable Information (PII) redaction pipeline, the experiments conducted to de-risk the model architecture, and the experiments investigating better preprocessing methods for the training data. We train 1.1B parameter models on the Java, JavaScript, and Python subsets of The Stack and evaluate them on the MultiPL-E text-to-code benchmark. We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at https://hf.co/bigcode.
1 INTRODUCTION
The report presents BigCode’s responsible development of Santa models, covering PII redaction, architecture ablations, and training-data preprocessing. SantaCoder combines these findings into a compact multilingual model that performs strongly on code generation and infilling.
- BigCode is an open-scientific collaboration focused on responsible development of large language models for code.
- The Santa models contain 1.1B parameters, train on Java, JavaScript, and Python subsets of The Stack, and are evaluated on MultiPL-E.
- The report describes a PII benchmark, detection filters, and evaluation of the PII-redaction pipeline on annotated code files.
- MQA and FIM only slightly deteriorate downstream performance relative to baseline models, while MQA can speed inference and FIM enables infilling.
- Filtering repositories with 5+ GitHub stars significantly worsens text2code performance, whereas other preprocessing filters have more modest effects.
- SantaCoder matches or exceeds InCoder-6.7B and CodeGen-Multi-2.7B on MultiPL-E generation and infilling across the three languages despite its smaller size.
2 RELATED WORK
Related work spans transformer architectures for code and benchmarks that evaluate generated programs across languages and tasks. MultiPL-E extends established Python benchmarks while keeping correctness tests hidden during generation.
- Code-generation research includes decoder-only, encoder, and encoder-decoder transformer architectures.
- Generated-code correctness can be evaluated with unit tests, while textual similarity metrics correlate only weakly with correctness.
- HumanEval and MBPP are popular Python benchmarks built from natural-language function descriptions and unit tests.
- Evaluating text-to-code: MultiPL-E extends MBPP and HumanEval to 18 additional languages by compiling prompts and tests into each language.
- Evaluating other tasks: CodeXGLUE evaluates code models on 14 datasets spanning code-to-code, text-to-code, and code-to-text tasks.
3 OPT-OUT PROCESS
The opt-out process allows developers to request removal of their source code from The Stack, and the project honored the requests received before the cutoff.
- 9 opt-out requests covered 299 repositories, of which 138 were present in The Stack v1.0 and were removed.
4 REDACTING PERSONALLY IDENTIFIABLE INFORMATION
The report’s initial PII-redaction pipeline targets emails, IP addresses, and secret keys in The Stack. Evaluation shows strong email and IP detection, but substantially weaker recall for secret keys, while detected PII varies by language.
- The project’s first PII-redaction effort targets source-code entities that may expose personally identifiable information.
- The benchmark annotates names, emails, usernames, passwords, IP addresses, API keys, and SSH keys across 400 pre-filtered code files.
- The first pipeline iteration detects emails, IP addresses, and keys, leaving names, usernames, and passwords for future work.
- PII detection and redaction: Emails are detected with regular expressions and replaced by randomized five-character addresses at example.com.
- PII detection and redaction: IP detection combines IPv4 and IPv6 regular expressions with validity checks and exclusions for likely package versions, private addresses, and popular DNS servers.
- PII detection and redaction: The secret-key detector uses detect-secrets plugins, disables keyword detectors, and applies filters intended to reduce false positives.
- Performance analysis: Email detection exceeds 90% precision and recall, IP detection exceeds 80%, and key detection has almost 80% precision but recall slightly above 50%.
- Performance analysis: Detected PII distributions differ by language: IP addresses are most common in Python, keys in JavaScript, and emails in Java.
5 EXPERIMENTS
The experiments establish the dataset, tokenizer, model, training setup, architecture ablations, preprocessing filters, and evaluation procedures used for SantaCoder.
- 5.1 DATASET, MODEL, AND TRAINING DETAILS: The base dataset contains 268 GB of Python, Java, and JavaScript files from The Stack v1.1 after deduplication, PII redaction, filtering, and benchmark decontamination.
- 5.1 DATASET, MODEL, AND TRAINING DETAILS: The tokenizer uses byte-level BPE with a 49,152-token vocabulary, trained on 600,000 rows balanced across the three languages.
- 5.1 DATASET, MODEL, AND TRAINING DETAILS: The base model is a 1.1B-parameter decoder-only transformer with FIM and MQA, trained for 300K iterations on 118B tokens.
- 5.2 ARCHITECTURE ABLATIONS: The architecture ablations compare FIM with no FIM and MQA with MHA to assess the training objective and attention design.
- 5.3 DATA FILTERING ABLATIONS: The preprocessing ablations test GitHub stars, comment-to-code ratio, near-duplicate filtering, and character-to-token ratio filters.
- 5.3 DATA FILTERING ABLATIONS: 5-grams with a 0.7 similarity threshold balance false positives and false negatives while removing an additional 16%–20% of files.
- 5.4 EVALUATION: Text2code evaluation generates function bodies from prompts and uses hidden tests, while fill-in-the-middle evaluation measures exact reproduction of one masked code line.
6 RESULTS
The experiments compare architectural and data-preprocessing choices and evaluate the final SantaCoder model on multilingual generation and infilling benchmarks. More near-deduplication helps, GitHub-star filtering hurts, and the smaller final model generally outperforms earlier open-source multilingual models.
- Architecture ablations: 1–4% higher HumanEval pass@100 and 1–3% higher MBPP pass@100 make MHA slightly stronger than MQA, although MHA uses 1.3B rather than 1.1B parameters.The comparison is therefore not entirely head-to-head; the authors suggest MQA’s inference speed-ups may outweigh its small performance drop.
- Architecture ablations: 2–4% lower HumanEval pass@100 and 1% lower MBPP pass@100 make FIM slightly worse than No-FIM on left-to-right benchmarks.This small but consistent drop contrasts with prior evidence for a FIM-for-free property.
- Data filtering ablations: 1–3% higher HumanEval and 1–4% higher MBPP performance result from more aggressive near-deduplication across the three languages.The comment-to-code filter improves HumanEval by 0–2% but can reduce MBPP performance, while the tokenizer filter has a neutral text-to-code effect.
- Data filtering ablations: 2–4% higher fill-in-the-middle performance follows from the fertility filter across the three languages, while near-deduplication and comments have mixed language-dependent effects.Those latter filters improve fill-in-the-middle performance for Python but deteriorate it for JavaScript.
- Data filtering ablations: 3–6% lower HumanEval and MBPP pass@100 and 5–11% lower fill-in-the-middle performance result from filtering repositories with 5+ GitHub stars.The filter removes over 60% of the data, but early divergence during training indicates the loss is not attributable only to dataset size.
- Final model: SantaCoder uses MQA, FIM, more near-deduplication, and the comments-to-code filter, trained for 600K iterations on 236B tokens.Doubling training iterations substantially improves MultiPL-E text-to-code performance across benchmarks and languages, but does not improve the reported single-line infilling evaluations.
- Final model: SantaCoder generally outperforms InCoder-6.7B and matches or exceeds CodeGen-Multi-2.7B on MultiPL-E generation and single-line infilling despite being smaller.The comparison covers left-to-right generation and infilling across languages.
7 CONCLUSION
The report concludes that BigCode’s early PII-redaction and ablation studies informed a 1.1B-parameter SantaCoder model that outperforms earlier multilingual code models. It also identifies secret-key detection and scaling model size and data as future priorities.
- PII redaction: Regular expressions are reasonably effective for detecting emails and IP addresses in the initial PII-redaction pipeline.The report identifies improving secret-key precision and recall, and detecting names, usernames, and passwords, as future work.
- Ablation findings: Filtering for GitHub stars consistently decreases performance across all benchmarks and programming languages.This finding comes from the architectural and data-filtering ablations performed on the PII-redacted Stack dataset.
- Final model: SantaCoder is a 1.1B model trained for 236B tokens that outperforms InCoder-6.7B and CodeGen-Multi-2.7B on left-to-right generation and infilling.The authors anticipate stronger multilingual, infilling-capable models from larger architectures and more training data.
8 CONTRIBUTIONS
The contribution list records the named teams responsible for model licensing, PII redaction, dataset construction, tokenization, training and architecture, opt-out work, evaluation, and inference.
- Contributions: Model licensing, PII redaction, and dataset work are attributed to separate named contributor groups.The contribution list distinguishes these three areas explicitly.
- Contributions: Tokenizer and training-and-architecture work are listed as distinct technical contribution areas with named contributors.The list separates tokenizer responsibilities from training and architecture responsibilities.
- Contributions: Opt-out, evaluation, and inference are separately identified contribution areas with their own named contributors.These entries complete the report’s contribution-role breakdown.
A FULL TEXT2CODE RESULTS
The appendix reports full text-to-code results for data-filtering and architecture ablations, with separate HumanEval and MBPP tables and an additional CodeXGLUE table.
- Data filtering ablations: Tables 8 and 9 provide full data-filtering ablation results on HumanEval and MBPP, respectively.The appendix reports these benchmark results separately by dataset.
- Architecture ablations: Tables 10 and 11 provide full architecture-ablation results on HumanEval and MBPP, respectively.The appendix likewise separates the two benchmark suites for architecture experiments.
- Additional evaluation: Table 12 reports CodeXGLUE Python Docstring generation using smoothed 4-gram BLEU scores under the stated zero-shot greedy evaluation setup.The maximum generation length is 128 tokens.
B DOCSTRING GENERATION
The report evaluates Santa models on Python docstring generation using CodeXGLUE. Its 1.1B-parameter variants outperform the larger InCoder model, with differences attributed to training datasets.
- Evaluation: CodeXGLUE evaluates docstring generation by prompting models with a Python function signature and requesting its corresponding docstring.The benchmark is constructed from CodeSearchNet, and evaluation uses the bigcode-evaluation-harness library.
- Findings: All BigCode Santa variants with 1.1B parameters outperform the 6.7B InCoder model.The report attributes this result to differences in the training datasets.
- Findings: Variants trained on more Python perform better: the stars variant contains 32% Python, compared with 28.5% for the tokenizer fertility variant.
- Findings: The bfloat16 and no-fim variants show no notable performance difference at the 1.1B-parameter scale.The no-fim variant was trained in float16, and the report suggests small-scale training instabilities were not encountered.
- Qualitative examples: Table 13 presents qualitative Python docstring-generation examples, including model generations and the correct solution.
C PII
The PII pipeline uses regular expressions and heuristic detectors to identify emails, IP addresses, and other potentially sensitive strings. Detected emails are replaced with randomized example.com addresses, while additional detectors support secret and identifier filtering.
- Regular expressions: Email addresses are detected with a regular expression and replaced by a random five-character string followed by @example.com.
- Regular expressions: IPv4 and IPv6 addresses are detected using regular expressions, with the same regular expression described for IP-address PII detection.
- Data pre-filtering: The annotation dataset is pre-filtered for data containing emails using a dedicated regular expression.
- Heuristic detection: Heuristic detectors identify potential UUIDs, likely identifier strings, templated secrets, and sequential strings.
- Secret detection: The implementation includes detectors for Base64 high-entropy strings, Azure storage keys, and Discord bot tokens.