SantaCoder: don't reach for the stars!

1 INTRODUCTION

• BigCode coordinates dataset, training, evaluation, and ethics work; The Stack v1.1 introduced a source-checking tool and an opt-out form for code removal.The Stack v1.1 contains 6.4 TB of permissively licensed source code in 384 programming languages.Source: §1 · p. 2
• The report describes a PII-redaction pipeline benchmarked on 400 code files, including filters for emails, IP addresses, and secret keys; all experiments use PII-redacted The Stack.Source: §1 · p. 3
• MQA and FIM ablations only slightly reduce downstream performance versus baseline models; MQA can speed inference at larger batch sizes, while FIM enables infilling.MQA: Shazeer (2019), Chowdhery et al. (2022), and Li et al. (2022); FIM: Fried et al. (2022) and Bavarian et al. (2022).Source: §1 · p. 3
• Of four preprocessing filters tested, more aggressive near-duplicate filtering can boost performance, while selecting files from repositories with 5+ GitHub stars significantly deteriorates text-to-code results.The other filters target comments-to-code ratio and character-to-token ratio; their effects are described as modest. Prior work had used GitHub stars as a data-quality proxy (Gao et al., 2020; Xu et al., 2022b).Source: §1 · p. 3
• SantaCoder, a 1.1B-parameter model, achieves comparable or stronger MultiPL-E code-generation and infilling performance than InCoder-6.7B and CodeGen-Multi-2.7B across Python, JavaScript, and Java.It is substantially smaller than both comparison models.Source: §1 · pp. 2–3

Code LLMs

• Code LLM research spans causal decoder-only, encoder, and encoder-decoder models; fill-in-the-middle training reportedly preserves left-to-right generation ability (Bavarian et al., 2022; Fried et al., 2022).Causal masking enables decoder-only infilling.Source: §2 · p. 3
• Multi Query Attention shares key and value embeddings across heads to reduce memory and speed large-batch inference; AlphaCode and PaLM implement it (Shazeer, 2019; Li et al., 2022; Chowdhery et al., 2022).Source: §2 · p. 3

Evaluating text-to-code

• Unit tests approximate semantic equivalence, while textual similarity metrics correlate only weakly with code correctness (Austin et al., 2021; Chen et al., 2021).Source: §2 · p. 3
• HumanEval and MBPP pair function descriptions with unit tests, while MultiPL-E extends them to 18 languages and MBXP uses a similar approach (Chen et al., 2021; Austin et al., 2021; Athiwaratkun et al., 2022).The benchmarks differ in type inference, prompt construction, and evaluation; MBXP exposes the test assertions in prompts, whereas MultiPL-E keeps them hidden. MultiPL-E automatically compiles doctests, signatures, and tests into new languages and replaces Python-specific prompt terminology.Source: §2 · p. 3

Evaluating other tasks

• CodeXGLUE evaluates 14 datasets spanning code-to-code, text-to-code, and code-to-text tasks, with Python and Java the most common languages (Lu et al., 2021).Tasks include clone detection, repair, translation, code search, generation, and documentation generation.Source: §2 · p. 4

• 9 opt-out requests covered 299 repositories before the 31 October 2022 cut-off.Source: §3 · p. 4
• 161 of the 299 repositories were already excluded from The Stack v1.0 for lacking a permissive license, while 138 were included.Source: §3 · p. 4
• BigCode removed the 138 included repositories from The Stack v1.1 and planned to remove requests received after 31 October 2022 before releasing v1.2.Source: §3 · p. 4

4.1 PII BENCHMARK

• The team annotated a benchmark of 400 pre-filtered files selected from 4,000 code files across 11 programming languages.Annotations covered names, emails, usernames, passwords, IP addresses, API keys, and SSH keys; 12 community members annotated one file each, followed by review and corrections for false positives and false negatives.Source: §4.1 · p. 4

4.2 PII DETECTION AND REDACTION

• The first iteration detects and redacts emails, IP addresses, and keys, leaving names, usernames, and passwords for future work.Emails are replaced with a randomized five-character local part at example.com; valid-format IPs are replaced with one of five generated addresses, while private IPs and popular DNS servers are not anonymized. Key detection combines detect-secrets plugins with filters for false positives, gibberish, and hashes, and avoids removing keys from files with frequent hash-related references.Source: §4.2 · pp. 4–5

4.3 PERFORMANCE ANALYSIS

• Emails exceeded 90% precision and recall, IP addresses exceeded 80%, while keys reached almost 80% precision but slightly above 50% recall, with key detection trading recall against false positives.Figure 1 reports these detection metrics; adding plugins or disabling filters found more keys but also increased false positives.Source: §4.3 · p. 5, Figure 1 · p. 5
• Java had fewer secrets per file, nearly 0.1% of files contained about 100 secrets, and detected IP addresses were most common in Python, keys in JavaScript, and emails in Java.JavaScript had significantly fewer files with PII than Python and Java, possibly because of prior line-length and alphanumeric-content filtering; Python and JavaScript had more secrets per file than Java. Files with many secrets sometimes contained repeated PII or long encodings and key-like strings.Source: §4.3 · pp. 5–6, Figure 2 · p. 5, Table 2 · p. 6

5.1 DATASET, MODEL, AND TRAINING DETAILS

• Experiments use 268 GB of filtered, decontaminated Python, Java, and JavaScript data and a 1.1B-parameter decoder-only transformer trained on 118B tokens.The training data comes from The Stack v1.1; the model uses FIM and MQA, and the final model described in Section 6.2 uses twice the compute of each 3.1-day run.Source: §5.1 · p. 6

5.2 ARCHITECTURE ABLATIONS

• Architecture ablations compare the base model’s FIM objective against left-to-right-only training and its MQA against MHA with otherwise identical hyperparameters.The FIM ablation uses random context-level, character-level transformations at a 0.5 FIM rate; the MHA alternative has 1.3B parameters versus the base model’s 1.1B.Source: §5.2 · pp. 6–7

5.3 DATA FILTERING ABLATIONS

• Additional filters target repositories with at least 5 stars, comment-to-code ratios, and low tokenizer fertility, with cutoffs chosen separately by language.The star filter removes more than 60% of data by volume; the comment filter removes 20%, and tokenizer-fertility cutoffs remove roughly 4%–5%. The comment filter removes files below 1% or above 80% comments; fertility cutoffs are 2.5 for Python, 2.9 for Java, and 2.6 for JavaScript, and may be biased against files with non-English comments.Source: §5.3 · pp. 7–8
• 5-grams with a 0.7 similarity threshold removed an additional 16%–20% of files; for documents with similarity above 0.85, the false-negative rate fell from 35% to 24%.Unigrams produced around 20% false positives at a 0.85 threshold; increasing n-gram size reduced false positives but increased false negatives. The additional deduplication was applied to already deduplicated data using the Stack v1 hyperparameters.Source: §5.3 · p. 8

5.4 EVALUATION

• Text-to-code evaluation uses MultiPL-E pass@1, pass@10, and pass@100 on Python, Java, and JavaScript; infilling evaluation measures exact reproduction of a masked code line.For infilling, the masked line is neither blank nor a comment, and benchmark solutions come from code-davinci-002 at temperature 0.8; this evaluation is reported only for data-filtering ablations.Source: §5.4 · pp. 8–9

6.1 ABLATIONS

• MHA improves pass@100 over MQA by 1-4% on HumanEval and 1-3% on MBPP, with noticeable JavaScript text2code gains, but MHA has more parameters (1.3B versus 1.1B).The report notes that the head-to-head comparison may therefore be unfair and that MQA inference speed-ups might outweigh its small performance drop.Source: §6.1 · pp. 9–10
• FIM models have pass@100 performance 2-4% lower on HumanEval and 1% lower on MBPP than No-FIM models, a small but consistent left-to-right drop.This contrasts with Bavarian et al. (2022), who presented evidence for a FIM-for-free property.Source: §6.1 · p. 10
• Near-deduplication improves HumanEval by 1-3% and MBPP by 1-4%; fertility filtering improves fill-in-the-middle by 2-4%, while comment and tokenizer effects vary by benchmark.Comment-to-code filtering improves HumanEval by 0-2% but can lower Java MBPP; near-deduplication and comments improve Python infilling but worsen JavaScript infilling, and tokenizer filtering is neutral on text2code.Source: §6.1 · pp. 10–11
• The GitHub stars filter drops pass@100 by 3-6% on HumanEval and MBPP and fill-in-the-middle performance by 5-11%; early divergence suggests data quality also contributes beyond dataset size.The filter removes over 60% of the data.Source: Figure 3 · p. 10, §6.1 · p. 11

6.2 FINAL MODEL

• Doubling training iterations boosts text2code performance across MultiPL-E benchmarks and languages, but the final run does not improve single-line fill-in-the-middle evaluations.SantaCoder uses MQA and FIM with more near-deduplication and comment-to-code filtering, trained for 600K iterations (236B tokens).Source: §6.2 · pp. 11–12, Figure 4 · p. 11
• SantaCoder outperforms InCoder 6.7B on left-to-right generation and single-line infilling across languages, and matches or exceeds CodeGen-multi 2.7B.The comparison is on MultiPL-E; the report describes SantaCoder as substantially smaller than InCoder.Source: Table 7 · p. 10, §6.2 · p. 12

7 CONCLUSION

• SantaCoder, a 1.1B model trained for 236B tokens, outperformed InCoder-6.7B and CodeGen-Multi-2.7B on left-to-right generation and infilling.The project expects larger architectures and more training data to produce stronger multilingual, infilling-capable models.Source: §7 · p. 12
• Filtering repositories by GitHub stars consistently decreased performance across benchmarks and programming languages.The project conducted architectural and data-filtering ablations using the PII-redacted version of The Stack.Source: §7 · p. 12
• The project’s first PII-redaction steps found regular expressions reasonably effective for detecting emails and IP addresses.Future work should improve precision and recall for secret keys and detect other sensitive information, including names, usernames, and passwords.Source: §7 · p. 12

• Tables 8 and 9 report full results for data-filtering ablations on HumanEval and MBPP, respectively.Source: A FULL TEXT2CODE RESULTS · p. 17, Table 8 · p. 17, Table 9 · p. 18
• Tables 10 and 11 report full results for architecture ablations on HumanEval and MBPP, respectively.Source: A FULL TEXT2CODE RESULTS · p. 17, Table 10 · p. 18, Table 11 · p. 19
• Table 12 reports CodeXGLUE Python docstring-generation smoothed 4-gram BLEU scores using Fried et al.'s methodology (2022), with zeroshot greedy decoding and a 128-token maximum generation length.Source: Table 12 · p. 19

B DOCSTRING GENERATION

• Models receive a Python function signature and generate a docstring on CodeXGLUE code-to-text, built from CodeSearchNet and evaluated with bigcode-evaluation-harness.The cited works are Lu et al. (2021), Husain et al. (2019), Ben Allal et al. (2022), and Gao et al. (2021), respectively.Source: B DOCSTRING GENERATION · p. 19

Findings

• All 1.1B-parameter Santa variants outperform 6.7B InCoder, and the 32%-Python stars variant outperforms the 28.5%-Python tokenizer fertility variant.The authors attribute the InCoder comparison to training-dataset differences and report the Python proportions in Table 3.Source: B DOCSTRING GENERATION · p. 19
• There is no notable performance difference between bfloat16 and no-fim, which differ in using bfloat16 versus float16 training.The authors suggest their 1.1B-parameter scale avoided training instabilities.Source: B DOCSTRING GENERATION · p. 19

Qualitative examples

• The section includes a CodeXGLUE Python docstring-generation prompt, model outputs, and the correct solution as a qualitative example.Table 13 presents the example.Source: B DOCSTRING GENERATION · p. 19, Table 13 · p. 20

C.1 REGULAR EXPRESSIONS

• Email addresses are detected with a regular expression.The passage gives the regex’s purpose but not a readable complete expression.Source: §C.1 · p. 20
• Detected emails are replaced with [random 5 character string]@example.com.Source: §C.1 · p. 20
• Regular expressions detect IPv4 and IPv6 addresses, and the same expression is used for IP-address pre-filtering.The passage identifies the IP-address pre-filtering expression as the one used for PII detection.Source: §C.1 · pp. 20–21
• The annotation dataset is pre-filtered for data containing emails.The passage says a regular expression is used but does not provide its full expression here.Source: §C.1 · p. 21

C.3 DETECT-SECRETS FILTERS

• Secret filters include checks for potential UUIDs, likely ID strings, templated secrets, and sequential strings.Source: §C.3 · p. 21

C.4 DETECT-SECRETS PLUGINS

• The listed secret-detection plugins are Base64HighEntropyString, AzureStorageKeyDetector, and DiscordBotTokenDetector.Source: §C.4 · p. 21