Source-linked AI summary

StarCoder: may the source be with you!

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, Harm de Vries

arXiv:2305.06161v2cs.CLcs.AIcs.PLcs.SE

TL;DR

Open code LLMs raise concerns about copyright, privacy, and openness. This paper develops and evaluates StarCoder and StarCoderBase, finding that they outperform existing open multilingual code LLMs and match or outperform code-cushman-001.

  • Problem

    The rapid adoption of code LLMs has heightened concerns about copyright, privacy, and openness.

    Source: §1 · p. 2
  • Method

    The BigCode community trains StarCoderBase on permissively licensed code data, fine-tunes it on Python, and applies data curation, PII redaction, and attribution tracing.

    Source: §1 · p. 3, §3 · p. 4, §4.3 · p. 14
  • Results

    StarCoder outperforms every open code LLM supporting multiple programming languages, matches or outperforms code-cushman-001, and substantially outperforms Python-fine-tuned alternatives.

    Source: §1 · p. 3
  • Takeaways & Limitations

    StarCoder and StarCoderBase provide publicly released open-access code LLMs with capabilities and release practices designed around responsible development.

    Source: §1 · pp. 2–3

    StarCoder may still produce personally identifiable information, and its PII detection can yield false positives or negatives across data types and programming languages.

    Source: §10.2 · p. 31

Abstract

from arXiv · show

The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python, can be prompted to achieve 40\% pass@1 on HumanEval, and still retains its performance on other programming languages. We take several important steps towards a safe open-access model release, including an improved PII redaction pipeline and a novel attribution tracing tool, and make the StarCoder models publicly available under a more commercially viable version of the Open Responsible AI Model license.

1 Introduction

The BigCode community introduces StarCoder and StarCoderBase as open-access Code LLMs developed through an open-scientific collaboration emphasizing responsible, transparent development. The models combine permissively licensed training data, broad programming-language coverage, and safety-oriented release tools, while achieving leading results across Code LLM benchmarks.

  • The models are developed by a 600-plus-member BigCode collaboration focused on copyright, privacy, transparency, and community-driven model development.The project uses The Stack, whose governance includes inspection tooling and an opt-out process for developers.Source: §1 · pp. 2–3
  • The Stack provides 6.4 TB of permissively licensed source code in 384 languages, alongside GitHub issues and repository metadata, with tools for inspection and removal requests.Its “Am I in The Stack” tool helps developers check dataset inclusion and opt out.Source: §1 · p. 3
  • StarCoderBase and StarCoder are 15.5B-parameter models trained on permissively licensed data, with 8K context, Fill-in-the-Middle infilling, and Multi-Query-Attention inference.StarCoderBase uses 1 trillion tokens from 80+ programming languages and related code artifacts; StarCoder adds 35B Python tokens through fine-tuning.Source: §1 · p. 3
  • StarCoder outperforms every open multilingual Code LLM, matches or outperforms OpenAI code-cushman-001, and substantially outperforms Python-fine-tuned alternatives.The paper presents the most comprehensive Code LLM evaluation to date using diverse benchmarks.Source: §1 · p. 3
  • The release uses a commercially viable OpenRAIL-M license that permits royalty-free access, use, and distribution while imposing restrictions for critical scenarios and encouraging documentation.The license effort promotes transparency through shared AI documentation such as model cards.Source: §1 · p. 3
  • The project adds attribution tracing and improves PII redaction to support safer releases, including a two-step membership-and-BM25 attribution search and a model trained on 22,950 annotated entities.The PII dataset contains 12,000 files, and the attribution tool is integrated into the VSCode demo.Source: §1 · p. 3

2 Related Work

Related work traces language modeling from n-gram and neural architectures to scalable Transformers, then surveys their application to code and the transparency gap between closed- and open-access LLMs.

Language models

  • Language modeling progressed from smoothed n-grams to feedforward and recurrent networks, followed by Transformers that enabled scalable models with predictable scaling relationships.Scaling factors include model size, training tokens, and compute budget.Source: §2 · pp. 3–4

Language Models for Code

  • Code language modeling evolved from small n-gram systems to encoder-only, encoder-decoder, and decoder-only Transformer models trained on text-code mixtures.Most decoder-only models were not fully open; PolyCoder (Xu et al., 2022) and SantaCoder (Ben Allal et al., 2023) released both models and training data but remained relatively small at 2.7B and 1.1B parameters, respectively, using less than 300GB of code.Source: §2 · p. 4

Closed-access LLMs

Open-access LLMs

  • Open-access LLMs generally lagged closed-access models, and their openness varied substantially in weight availability, training-data disclosure, and filtering transparency.The paper defines “open-access LLM” by publicly available model weights; examples range from EleutherAI releases with datasets to models released without training sets or under non-commercial licenses.Source: §2 · p. 4

3 Data Curation and Cleaning

The StarCoderBase training data was curated from permissively licensed GitHub repositories in The Stack v1.2 using heuristic filtering, manual inspection, and specialized processing for programming languages, notebooks, discussions, commits, deduplication, and source weighting.

3 Data Curation and Cleaning

  • The dataset was restricted to The Stack v1.2, which contains permissively licensed GitHub repositories, and was further cleaned through heuristic filtering and manual inspection.At processing time, 44 people had opted out of The Stack.Source: §3 · p. 4

3.1 Programming Languages

  • The team selected 86 of The Stack’s 358 programming languages using data volume, popularity, and dialect criteria, while excluding configuration and unsupported languages.Visual inspection covered 300 extensions and led to excluding 36 extensions and removing the long-line filter for 27 extensions.Source: §3.1 · p. 5

3.2 Jupyter notebooks

  • Jupyter notebooks were converted into 1,432,992 scripts with Jupytext and Guesslang, and into 1,045,605 structured notebooks containing ordered code-text pairs and optional outputs.Guesslang reduced notebooks without identified languages to 6,400 using a probability threshold of at least 0.5.Source: §3.2 · pp. 8–9

3.3 GitHub issues

  • GitHub issues and pull-request conversations were filtered for length, autogenerated text, bots, user participation, and English language, removing 18% of volume, 17% of events, and 14% of issues.Usernames were anonymized by replacing them with participant counters within each conversation.Source: §3.3 · p. 9

3.4 Git commits

  • Git commit data was sampled from single-file commits, filtered with heuristics, and represented using the full file 20% of the time or a 0–32-line changed-region window 80% of the time, yielding 64 GB.Repositories from users who opted out of The Stack were removed.Source: §3.4 · p. 9

3.5 Deduplication

  • Near-deduplication used 5-gram MinHashes, locality-sensitive hashing, and Jaccard similarity 0.7 for programming languages and Jupyter notebooks, but not Git commits or GitHub issues.The exclusions were due to time constraints for commits and an expectation that issues would contain few duplicates.Source: §3.5 · p. 9

3.6 Weighting of data sources

  • Training data sources were sampled proportionally to volume, while JSON and YAML were reweighted to 1 GB and CSS to 3 GB to limit memorization of data-heavy formats.After deduplication, several high-resource languages contained similar volumes of 44–87 GB, supporting the decision not to drastically reweight them.Source: §3.6 · p. 10

4 PII redaction

The PII-redaction pipeline combines a crowdsourced annotated dataset, StarEncoder, and a fine-tuned NER detector with pseudo-labeling and post-processing. The detector surpassed regex baselines, while the pipeline excluded unreliable ID and username detection and applied targeted masking to code and GitHub issues.

4.1 Data collection

  • 1,399 workers from 35 countries annotated 12,000 code files spanning 31 programming languages, producing 22,950 PII entities.The files included 7,100 pre-filtered examples emphasizing rare PII types and 5,100 randomly selected files; annotation covered names, usernames, emails, IP addresses, keys, passwords, and IDs.Source: §4.1 · pp. 10–11
  • The annotation process distinguished voluntary license-header PII, placeholders, and confidential data, and excluded secret IDs from model training because manual inspection found many false positives and negatives.The quality review manually inspected 300 files and calculated precision and recall for each PII type.Source: §4.1 · p. 11, Table 5 · p. 12

4.2 StarEncoder

  • An encoder-only, bidirectionally self-attentive Transformer was trained with BERT’s MLM and NSP objectives for efficient fine-tuning on code- and text-related tasks.Training randomly selected two code snippets from the same source file or distinct documents, masked 15% of input tokens, and ran 100,000 steps with a global batch size of 4,096 sequences and maximum length 1,024.Source: §4.2 · p. 11

4.3 PII detection model

  • Fine-tuning StarEncoder as a six-class NER model achieved F1 above 90% for names, emails, and IP addresses, versus 73.39% for passwords, 56.66% for keys, and 59.39% for usernames.Pseudo-labeling improved all PII categories except usernames, which were excluded from redaction; post-processing filtered short or invalid secrets, keys, and IP addresses before masking.Source: §4.3 · pp. 12–14
  • The PII detection models surpassed enhanced regexes for emails, IP addresses, and keys, while the regex baseline reached 96.83% F1 for email detection and detect-secrets had below 4% overall precision.The regexes supported only emails, IP addresses, and keys; detect-secrets produced many false positives, especially in Go and C-sharp.Source: §4.3 · p. 13, Table 8 · p. 13
  • For GitHub issues, the pipeline used the model to redact names, replaced active participants’ usernames with conversation-local pseudonyms, and preserved speaker identity while masking username mentions.Keys, IP addresses, and emails in issues were handled with regexes, while non-participating users’ mentions were not anonymized.Source: §4.3 · p. 14

5 Model training

StarCoderBase was pretrained on 1 trillion tokens, while StarCoder was fine-tuned from it on Python data. The section details data preparation, model design, training infrastructure, and environmental impact.

5.1 Data formatting

  • Training data formatting adds repository, filename, and bucketed GitHub-star metadata, uses sentinel-token structures for issues and notebooks, and formats commits around changed code.Metadata fields were independently omitted with probability 0.2, while code and other sources used structured sentinel tokens.Source: §5.1 · p. 15

5.2 Training data decontamination

  • Decontamination removed files containing benchmark docstrings, solutions, questions, or prompts; Python had the most matches, with 558 files removed.The affected benchmarks included HumanEval, MBPP, APPS, GSM8K, and DS1000.Source: §5.2 · p. 15

5.3 Tokenizer

  • The tokenizer is a byte-level BPE with a 49,152-token vocabulary, digit splitting, and the GPT-2 regex pre-tokenizer.The vocabulary includes the sentinel tokens used by the training formats.Source: §5.3 · p. 16

5.4 Model Architecture

  • The 15.5B-parameter models use a decoder-only Transformer with multi-query attention, learned absolute positional embeddings, FlashAttention, and an 8K context length.Fill-in-the-middle transformations were applied during training, and FlashAttention reduced attention memory usage.Source: §5.4 · p. 16

5.5 Training details

  • StarCoderBase was trained for 250k iterations on 1 trillion tokens, while StarCoder was fine-tuned for 8,500 steps on Python data for 2 epochs.StarCoderBase used a 4M-token batch; StarCoder used a lower learning rate and the same general settings.Source: §5.5 · p. 16

5.6 Multi-Node GPU Setup

  • Training used 512 A100 80 GB GPUs across 64 nodes with 3D parallelism, 32-fold data parallelism, and a global batch equivalent to 4M tokens.BF16 was used despite 10% lower throughput than FP16 to avoid training instabilities.Source: §5.6 · p. 17
  • Training encountered no significant instabilities apart from a few restarts.This result was reported for the multi-node training setup.Source: §5.6 · p. 17

5.7 CO2 emissions

  • StarCoderBase training consumed 89,671.68 kWh and emitted 16.68 tonnes of CO2eq, while StarCoder fine-tuning added 0.58 tonnes from 3.5% additional training time.The StarCoderBase estimate used 320,256 GPU hours and the AWS us-west-2 carbon intensity.Source: §5.7 · p. 17

6 Evaluation

The evaluation uses a reproducible Code LM Evaluation Harness to assess StarCoder models across Python, multilingual, security, infilling, training-process, and context-length benchmarks. It examines comparative performance, broad task capabilities, training-process errors, and the effects of additional context on repository-based perplexity.

A Code LM Evaluation Harness

  • The Code LM Evaluation Harness enables efficient, centralized, reproducible evaluation through data parallelism and Docker containers across benchmarks including HumanEval, MultiPL-E, and DS-1000.It was developed specifically to evaluate StarCoder and other Code LLMs.Source: A Code LM Evaluation Harness · p. 17

6.1.1 The HumanEval and MBPP Benchmarks

  • StarCoder is the highest-performing open-access model on HumanEval and MBPP, outperforming larger models and code-cushman-001, while StarCoderBase remains competitive with CodeGen-16B-Mono.These Python results compare both open- and closed-access models using pass@1.Source: Table 12 · p. 18, §6.1.1 · p. 19

6.1.2 The DS-1000 Python Data Science Benchmarks

  • StarCoder substantially outperforms all other models on DS-1000 across every data-science library, while StarCoderBase also leads but trails StarCoder slightly; ODEX shows the same dominance on open-domain queries.ODEX also finds both StarCoder models have smaller open-to-closed-domain gaps than other baselines, indicating more generalized coding-query skills.Source: Table 13 · p. 19, §6.1.2 · pp. 19–20, §6.1.3 · p. 20

6.2.1 Evaluation on 19 Programming Languages with MultiPL-E

  • Across 19 programming languages, StarCoderBase outperforms other open-access models, is competitive with code-cushman-001, and StarCoder remains competitive despite Python fine-tuning.Code-cushman-001 exceeds StarCoderBase by more than 5% on C++, Java, Ruby, and Swift, whereas StarCoder exceeds it by more than 5% on Julia; StarCoder also slightly surpasses StarCoderBase on some languages.Source: Table 15 · p. 21, §6.2.1 · pp. 21–22

6.2.2 The “Asleep at the Keyboard” Security Benchmark

  • Among models with more than 95% valid code, StarCoder has the lowest rate of insecure completions, while StarCoderBase has the highest valid-code rate.The security evaluation uses 40 automatically testable scenarios from the Asleep at the Keyboard benchmark.Source: Table 16 · p. 22, §6.2.2 · p. 22

6.2.3 Fill in the Middle Benchmarks

  • StarCoderBase significantly outperforms smaller models on single-line infilling and outperforms InCoder on TypeScript type prediction, while StarCoder models lead Python return-type and docstring generation.The return-type evaluation may overlap with training data, as may the docstring benchmark; TypeScript evaluation uses packages never translated to TypeScript.Source: Table 17 · p. 23, §6.2.3 · p. 23, Table 18 · p. 23

6.3 Performance Improvement Through the Training Process

  • Training longer improves pass@1 mainly for high-resource languages, whereas low-resource languages can stagnate or decline; R drops significantly from the 800B to 1000B checkpoint.Performance improvements occur for languages with roughly at least 1 GB of available data, and per-problem checkpoint variance can be uncorrelated and caused by minor mistakes.Source: §6.3 · p. 24, Figure 2 · p. 25
  • StarCoderBase’s evaluated coding errors include failing to take absolute values for GCD, convert strings to character arrays, and check edge cases.Source: §6.3 · p. 25

6.4 Perplexity With Long Contexts

  • The long-context evaluation compares perplexity using 2K-token and 8K-token windows on held-out GPL repositories across the languages in Table 21.The test predicts the final 1K tokens of 8K-token chunks, conditioning on either the preceding 1K or 7K tokens.Source: §6.4 · p. 25
  • Substantially lower perplexities across all languages show that StarCoderBase benefits from the extra token conditioning provided by its 8K context window.Average perplexity is computed over the predicted 1K-token regions across all chunks.Source: §6.4 · p. 25

7 Natural Language Evaluation

StarCoderBase is evaluated on natural-language reasoning, understanding, generation safety, and HELM tasks, leveraging training data that includes roughly 20% natural-language tokens. It generally outperforms comparable open-access models while showing lower measured bias and toxicity than several baselines.

7 Natural Language Evaluation

  • The evaluation covers natural-language reasoning and understanding tasks that may benefit from StarCoderBase’s combined code-and-text training, plus generation settings relevant to documentation and interactive assistants.Roughly 20% of training tokens are natural-language data, including GitHub issues, Markdown, Jupyter notebooks, and HTML.Source: §7 · pp. 25–26

7.1 Math Reasoning

  • With PAL, StarCoderBase reaches 21.5% GSM8K accuracy versus 8.4% with CoT, outperforming CodeGen-16B-Mono at 13.1% and CodeGen-16B-Multi at 8.6%.It exceeds LLaMA-13B’s 17.8% but trails LLaMA-33B’s 38.7%; majority voting preserves these differences.Source: §7.1 · p. 26, Table 22 · p. 26

7.2 World Knowledge and Reading Comprehension

  • On MMLU, StarCoderBase scores 34.2% versus CodeGen-16B-Multi’s 27.8% and slightly exceeds GPT-NeoX at 32.9%, while both LLaMA models perform better.On CoQA, StarCoderBase outperforms CodeGen-16B-Multi but is outperformed by LLaMA and GPT-NeoX.Source: §7.2 · p. 27, Table 23 · p. 27, Table 24 · p. 27

7.3 Measuring Harmful Generation

  • The harmful-generation evaluation compares StarCoderBase with prior Code LLMs on social-bias and toxicity benchmarks for model-produced text.These benchmarks target potentially harmful open-ended outputs such as code documentation and technical dialogue.Source: §7.3 · p. 27

7.3.1 Social Bias

  • Across four StereoSet bias domains, StarCoderBase has the lowest stereotype scores while retaining competitive language-modeling scores and a high ICAT score.This indicates that its lower stereotype scores are not simply attributable to worse language modeling.Source: §7.3.1 · p. 28, Table 25 · p. 28

7.4 Reasoning Tasks in HELM

  • StarCoderBase generally outperforms other released-weight models on HELM natural-language reasoning tasks and often matches or exceeds much larger models.The evaluation excludes code tasks and ranks models by average task ranking; the authors speculate that mixed code-and-natural-language training contributes to this performance.Source: Table 27 · p. 29, §7.4 · p. 30

7.3.2 Toxicity

  • StarCoderBase produces fewer potentially offensive responses than LLaMA-13B, with 1.12% containing offensive tokens versus 1.43% for LLaMA-13B, although CodeGen-16B-Multi appears less toxic.Toxicity is measured using both a RoBERTa-based classifier and an offensive-word list on generated responses.Source: §7.3.2 · p. 29, Table 26 · p. 29

8 Qualitative Evaluation

The qualitative evaluation highlights StarCoderBase interactions and prompting strategies that expose model capabilities, including technical assistance and improved HumanEval pass@1 performance.

Section E.1

  • Appendix E illustrates interesting StarCoderBase behaviors elicited with Git commit, GitHub issue, and Jupyter notebook templates.These examples are intended as a starting point for researchers and developers exploring the model’s capabilities.Source: §8 · p. 30

Section E.2

  • StarCoder can be prompted to act as a technical assistant without instruction-tuning.Source: §8 · p. 30

Section E.3

  • Combining metadata with natural language prompting yields higher pass@1 performance on HumanEval.Source: §8 · p. 30

9 Attribution Tools

The paper releases two file-content attribution tools—a Bloom-filter membership checker and a BM25 search index—that help users trace model outputs to training files and repositories. The tools are publicly available as standalone sites and integrated into a VSCode demo.

9 Attribution Tools

  • The released attribution tools index training files for content matches, helping users identify copied output and locate its source file and repository.They complement the existing “Am I in The Stack” tool, which operates at the GitHub repository-name level.Source: §9 · p. 30

9.1 Membership Checking

  • The membership checker uses a Bloom-filter portrait of 50-character training-data strings, occupying 26 GB—approximately 3% of the data size.It provides rapid, approximate first-pass overlap checks through the VSCode extension, but requires sufficiently long matches and does not filter common or generic code snippets; users can then consult the full search index.Source: §9.1 · p. 30

9.2 Search Index

  • The search index provides separate Python-subset and full-dataset tools using Elasticsearch 7.17, preprocessing, 3-gram tokenization, BM25 scoring, and searchable username and license fields.Both indexes run in single-node mode on one virtual machine.Source: §9.2 · pp. 30–31

10 Social Impact and Limitations

StarCoder prioritizes responsible open science and governance, accepting adoption and research trade-offs to reduce misuse and protect data rights. Its limitations include licensing, privacy, safety, evaluation, and attribution gaps, while community research highlights the value of transparency and choice.

10.1 Project approach

  • Open governance prioritized responsible decisions over adoption, including withholding an identified malicious-code dataset despite its potential value for security research.The project frames openness as enabling community research and audits but recognizes that released weights are difficult to retract and require guardrails.Source: §10.1 · p. 31

10.2 Limitations

  • The Stack’s license filtering and repository-level opt-out process can misclassify licenses and cannot reliably remove duplicated code, requiring stronger consent and data-control mechanisms.The dataset includes only source code classified as permissively licensed, but detection errors and copied repositories can undermine licensing and opt-out protections.Source: §10.2 · p. 31
  • StarCoder may still reproduce PII or malware and generate inaccurate, offensive, misleading, discriminatory, or stereotyped content despite detection efforts and enforceable OpenRAIL-M restrictions.PII detection can produce false positives or negatives and vary across data types and programming languages; The Stack may contain undetected malicious code.Source: §10.2 · pp. 31–32
  • StarCoder was evaluated only on English-based benchmarks, while its attribution tools cover only the training subset and may produce false positives or miss generic or protected code.Future work should assess other natural languages; attribution tools also require sufficient context and resolution and do not distinguish boilerplate from protected content.Source: §10.2 · p. 32

10.3 Social impact

  • Code LLMs may broaden access to coding and support more robust software, but they also risk security failures, over-reliance, and disruption to software-development employment.The project therefore emphasizes careful consideration of the technology’s broader social impact and security implications.Source: §10.3 · p. 32
  • The project used transparent compensation practices for Toloka annotators, setting a $7.30 hourly rate based on purchasing-power parity with a highest U.S. minimum wage of $16.50.The choice balanced fair compensation, timing, and annotation quality while restricting eligibility to countries meeting the purchasing-power criterion.Source: §10.3 · p. 32
  • Community research with 50 participants found that people valued knowing about data use and having choices, especially governance tools that raise awareness and empower action.Most participants were neutral to positive about permissively licensed data use, but none wanted to opt out; future research should extend beyond North America and Europe.Source: §10.3 · p. 33

11 Conclusion · A GitHub issues filtering

The report presents StarCoderBase and StarCoder as transparent, open-access code models whose evaluation, licensing, and attribution tools support reproducible and responsible use. Its GitHub issues filtering appendix documents regex-based removal of automated text and bot-authored comments.

11 Conclusion

  • StarCoder outperforms CodeGen and CodeGeeX and matches or outperforms OpenAI’s closed-access code-cushman-001 model.The report describes this as the most extensive evaluation of Code LLMs to date.Source: §11 · p. 33
  • The models are released under an Open Responsible AI Model license with use restrictions, alongside open-sourced GitHub code repositories.These measures aim to increase access, reproducibility, and transparency while supporting responsible AI principles.Source: §11 · p. 33
  • The project provides transparency across training data, data curation, PII redaction, and model training.The report frames full-process transparency as part of the BigCode community’s development effort.Source: §11 · p. 33
  • Novel attribution tools help end-users detect and locate model generations that may have been copied from the training set.The tools are presented as part of a broader effort toward a safe model release.Source: §11 · p. 33

A GitHub issues filtering

  • The GitHub issues filtering appendix lists filters and regular expressions used to remove automated text.The implementation includes compiled regular expressions applied with DOTALL behavior.Source: A GitHub issues filtering · p. 44
  • The filtering removes comments from authors whose usernames match listed bot keywords or suffixes.Keyword examples include [bot], botmanager, jenkins, and travis; suffix examples include -automaton, -automation, -ci, and -testing.Source: A GitHub issues filtering · p. 44

B Annotator countries

The PII labeling effort recruited annotators from multiple countries, as listed in Table B.1.

B Annotator countries

  • Table B.1 lists the countries from which annotators were recruited for the PII labeling effort.Source: Table B.1 · p. 45

C Replacements for IP addresses

The listing defines random private IP addresses used to mask original IP addresses, covering both IPv4 and IPv6 replacements.

C Replacements for IP addresses

D Additional Evaluation Results … E.2 Technical Assistant

Additional evaluations compare StarCoder with larger closed models across multilingual coding, security, bias, and scaling settings, while qualitative sections show how pretraining templates and prompting enable technical-assistant behaviors with important limitations.

D Additional Evaluation Results

  • Code-davinci-002 performs best on MultiPL-E HumanEval, despite substantially higher parameter count and inference cost than StarCoder and code-cushman-001.Table D.1 compares pass@1 performance across multiple programming languages.Source: Table D.1 · p. 46
  • Larger code-davinci-002 does not appear to generate more secure code effectively than StarCoderBase on the Asleep at the Keyboard dataset.The security evaluation contrasts StarCoderBase with OpenAI’s code-davinci-002.Source: Table D.2 · p. 46
  • Table D.3 reports R pass counts on selected problems where the 800B and 1000B checkpoints differ by at least 100 passes out of 200 samples.The table focuses on problems with substantial checkpoint-level differences.Source: Table D.3 · p. 47
  • CrowS-Pairs reports stereotype scores across bias domains, with scores closer to 50% indicating less bias.Table D.4 presents results separately for each domain.Source: Table D.4 · p. 47

E.1 Using Pretraining Templates

  • Sentinel-token templates for commits, GitHub issues, and Jupyter notebooks support code modification, technical question answering, natural-language code generation, and forced output prediction.Training on Jupyter cell outputs also enables basic code-output interpretation, with empty-output suppression ensuring a prediction.Source: §E.1 · p. 48

E.2 Technical Assistant

  • An HHH-based prompt with programming examples makes StarCoder a capable but brittle technical assistant without instruction-tuning, using its 8K context for questions, instructions, and technical problems.Examples draw on CoT, least-to-most prompting, StackExchange, PAL, Anthropic’s HHH prompt, and the authors’ work; the assistant can still produce wrong solutions, false facts, or offensive comments.Source: §E.2 · p. 48

E.3 Improving Code Generation with Prompting · F Technical Assistant Prompt

This section examines prompting strategies for code generation and presents a Technical Assistant prompt designed to support conversational, iterative Python assistance through clarification, implementation guidance, and testing.

E.3 Improving Code Generation with Prompting

  • The prompting intervention addresses empty HumanEval solutions such as pass or “Insert code here,” a failure observed across every evaluated model.In IDE practice, programmers commonly respond to this failure by altering the prompt ad hoc.Source: §E.3 · p. 48
  • 40.8% pass@1 on HumanEval rises from 33.6% unprompted when StarCoder receives a filename-and-instruction prefix at temperature 0.1.The prefix uses StarCoder’s <filename> input format.Source: §E.3 · p. 48
  • 28.10% pass@1 results when CodeGen-16B-Mono receives the same temperature and prompt, while the prompt has negligible impact on StarCoderBase.CodeGen cannot use the filename component, and other prefixes might improve its performance.Source: §E.3 · p. 48

F Technical Assistant Prompt

  • The Technical Assistant prompt presents an assistant that is helpful, polite, honest, sophisticated, emotionally aware, humble-but-knowledgeable, and cautious without sacrificing usefulness.It explicitly supports code questions and caveats when uncertain.Source: F Technical Assistant Prompt · p. 51
  • The dialogues demonstrate iterative coding support: writing a function, generating tests, modifying it for uneven-length lists, and explaining or repairing code.Examples include alternating-list functions, factorial initialization, and code-purpose explanations.Source: F Technical Assistant Prompt · pp. 51–52
  • The examples extend to mathematical and practical programming tasks, including k-combinations, prime-number filtering, CSV salary selection, Java translation, and Django environment troubleshooting.The prompt also supports step-by-step arithmetic reasoning followed by Python function generation.Source: F Technical Assistant Prompt · pp. 52–54
  • The assistant declines unrelated preference questions, stating that it is designed for technical support and code-related problem solving.Its stated role includes writing code to solve math problems.Source: F Technical Assistant Prompt · p. 55
  • The assistant provides a Python function that randomly chooses between pizza and a burger, with a 30% probability of selecting pizza.The function uses random.uniform(0, 1) and prints the corresponding lunch suggestion.Source: F Technical Assistant Prompt · p. 55
  • The assistant warns that the lunch function’s output may differ across executions because it is intentionally random.This explicitly qualifies the behavior of the generated code.Source: F Technical Assistant Prompt · p. 55
  • For nested dictionaries, the assistant first clarifies that the user is asking about Python dictionaries whose values can themselves be dictionaries.The exchange establishes the programming context before proposing a solution.Source: F Technical Assistant Prompt · p. 55
  • The assistant characterizes the requested transformation as recursively flattening dictionaries when multiple nesting levels are present.The proposed approach is framed as recursion over several dictionary levels.Source: F Technical Assistant Prompt · p. 55
  • To keep flattened keys unambiguous, the assistant recommends combining outer and inner keys with a separator character.The user suggests using an underscore as the separator.Source: F Technical Assistant Prompt · p. 55
  • The assistant supplies Python code that constructs flattened keys by concatenating nested key components with the chosen separator and updates a result dictionary.The implementation iterates through dictionary items, creates new flattened entries, and merges them into the result.Source: F Technical Assistant Prompt · p. 55
  • The assistant closes by asking the user to try the code on their example and report whether it works or raises further questions.This positions testing and follow-up as part of the interaction.Source: F Technical Assistant Prompt · p. 55
Loading 2305.06161v2…