PaperlayerView exact revision on arXiv

Source-linked AI summary

StarCoder: may the source be with you!

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, Harm de Vries

arXiv:2305.06161v2cs.CLcs.AIcs.PLcs.SE

TL;DR

Code LLM development faces performance, openness, copyright, privacy, and transparency challenges. The BigCode community develops and evaluates open-access StarCoder models while adding release safeguards, finding strong comparative performance across multilingual and Python-focused settings. The models are publicly released with open-scientific documentation and tools for attribution tracing and PII mitigation.

  • Problem

    Code LLM development has raised concerns about copyright, privacy, transparency, and openness while existing models vary in accessibility and disclosure.

  • Method

    The BigCode community trains and evaluates open-access StarCoder models on permissively licensed code, combining broad multilingual pretraining, Python fine-tuning, architectural features, and safety-oriented release processes.

  • Results

    StarCoderBase outperforms every open multilingual Code LLM and matches or outperforms OpenAI code-cushman-001, while StarCoder outperforms models fine-tuned on Python.

  • Takeaways & Limitations

    The StarCoder models provide publicly available Code LLMs accompanied by documentation, attribution tracing, and PII redaction efforts for more responsible use.

  • Takeaways & Limitations

    The deduplication procedure could not be applied to Git commits and was not applied to GitHub issues.

Abstract

from arXiv · show

The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python, can be prompted to achieve 40\% pass@1 on HumanEval, and still retains its performance on other programming languages. We take several important steps towards a safe open-access model release, including an improved PII redaction pipeline and a novel attribution tracing tool, and make the StarCoder models publicly available under a more commercially viable version of the Open Responsible AI Model license.

1 Introduction

The BigCode community introduces open-access StarCoder models designed to address code-LLM performance alongside concerns about copyright, privacy, transparency, and responsible release. The models combine broad training and architectural capabilities with extensive evaluation and safety measures.

  • The project targets copyright, privacy, and transparency concerns through permissively licensed data, open-scientific collaboration, and attention to responsible model development.
  • 15.5B-parameter StarCoder models use 8K context, Fill-in-the-Middle infilling, and Multi-Query Attention for fast large-batch inference.StarCoderBase was trained on 1 trillion tokens from 80+ programming languages and related sources; StarCoder adds 35B Python tokens through fine-tuning.
  • StarCoder and StarCoderBase are open-access Code LLMs trained on 80+ programming languages with a novel combination of capabilities and architectural features.
  • The paper evaluates Code LLMs using a diverse benchmark suite and reports comparative performance across programming-language and fine-tuning settings.
  • StarCoder outperforms every open multilingual Code LLM and matches or outperforms OpenAI code-cushman-001.
  • The release includes an OpenRAIL-M license, attribution tracing for potentially copied generations, and an improved PII redaction pipeline.The PII pipeline used 12,000 files containing 22,950 annotated entities to fine-tune StarEncoder.

2 Related Work

Prior Code LLM research spans n-gram, neural, encoder-only, and encoder-decoder approaches, while access and transparency vary substantially across released and unreleased models.

  • Early Code LLMs used comparatively small-scale n-gram models, followed by encoder-only and encoder-decoder neural architectures.
  • Closed-access LLM providers generally release neither model weights nor comprehensive information about their development methodology.
  • Open-access LLMs make model weights publicly available, but differ in training-data disclosure, filtering transparency, and licensing restrictions.

3 Data Curation and Cleaning

StarCoderBase training data is curated from permissively licensed GitHub repositories through language selection, quality filtering, notebook transformation, conversation and commit processing, and code deduplication. The resulting corpus combines multiple data sources while applying targeted heuristics and inspection to improve quality.

  • Source selection: The Stack v1.2 supplies permissively licensed GitHub data, with 44 people opted out before further heuristic filtering and manual inspection.The selected corpus is restricted to data from permissively licensed repositories.
  • Programming-language selection: 86 programming languages were selected from 358 using data volume, popularity rankings, and dialect inclusion, while configuration and obsolete languages were excluded.JSON and YAML were included with separate volume limits.
  • Quality filtering: 30,000 files per language were sampled for inspection, and annotators retained files judged to be human-written code rather than text, data, or autogenerated lines.The inspection sampled by extension and capped retained files at 1,000 per extension.
  • Jupyter notebooks: 1,432,992 Jupyter scripts were created with Jupytext, using notebook metadata and Guesslang to recover missing language information.A probability threshold of at least 0.5 reduced unidentified notebooks to 6,400.
  • Jupyter notebooks: Jupyter notebooks were also converted into 1,045,605 structured examples containing temporally ordered Markdown-code pairs and formatted outputs when available.These pairs provide instruction-like context for Python code.
  • Additional sources and deduplication: GitHub issues, pull requests, and commits were filtered, truncated or sampled, while commit data was reduced to 64 GB and code files were deduplicated with MinHash and LSH.Deduplication used 5-grams with a Jaccard similarity threshold of 0.7; Git commits and issues were excluded from that deduplication process.

4 PII redaction

The paper constructs a PII dataset and trains a StarEncoder-based detector to redact sensitive information from code and related data. Annotation quality is uneven, so the pipeline excludes unreliable IDs, adds post-processing, and compares the detector with regex baselines.

  • Data collection: 1,399 crowd-workers from 35 countries annotated PII in source code, producing 12,000 files and 22,950 annotated entities.Targeted categories included names, usernames, emails, IP addresses, keys, passwords, and IDs.
  • Data collection: PII was distinguished by context, separating license-header attribution, placeholders, and confidential data for names, emails, and usernames.License-header information may be voluntarily provided for attribution, while placeholders are not real secrets.
  • Annotation quality: Secret IDs were excluded from model training because annotation was particularly difficult and produced many false positives and false negatives.Dataset quality was assessed by manually inspecting 300 files and calculating precision and recall by PII type.
  • StarEncoder: StarEncoder uses bidirectional Transformer pretraining with masked language modeling and next-sentence prediction before PII-specific fine-tuning.The model is an encoder-only architecture designed for code- and text-related tasks.
  • PII detection model: The PII detector adds a token-classification head with six target classes and uses balanced training and test splits of 7,878 and 4,000 examples.IDs were excluded, and license-header versus placeholder distinctions were not modeled because of poor performance.
  • PII detection model: F1 exceeded 90% for names, emails, and IP addresses, reached 73.39% for passwords, and fell to 56.66% for keys and 59.39% for usernames.The paper attributes weaker key performance to only 308 labeled instances and username errors to confusion with decorators and path values.
  • PII detection model: The pipeline improves detection with pseudo-labels, regex comparison, and post-processing rules that remove short or implausible secrets and constrain full-name detection.The enhanced email regex reached 96.83% F1, while the PII models still surpassed regexes for emails, IP addresses, and keys.
  • PII redaction: GitHub issues use regexes for keys, IP addresses, and emails, while the PII model redacts names and replaces active participants’ usernames with conversation-local pseudonyms.The pseudonyms preserve speaker identity, and mentions of participating usernames are also redacted.

5 Model training

StarCoderBase is trained on one trillion tokens, while StarCoder is its Python-fine-tuned version. The models use structured data formatting, decontamination, and a 15.5B-parameter architecture designed for long-context and efficient code generation.

  • Model variants: StarCoderBase is trained on 1 trillion tokens, while StarCoder is fine-tuned on an additional 35B Python tokens.StarCoder starts from StarCoderBase and is trained for two epochs on Python data.
  • Data formatting: Training data is formatted with sentinel tokens for code, commits, issues, and structured Jupyter notebooks.Templates encode repository metadata, changed code regions, issue turns, and notebook text-code-output sequences.
  • Data preparation: The code corpus is decontaminated by removing files containing benchmark solutions, docstrings, questions, or prompts.Python produced the largest reported number of matches, with 558 files removed.
  • Architecture: Both models use 15.5B-parameter decoder-only Transformers with Multi-Query Attention, Fill-in-the-Middle training, and an 8K-token context length.FlashAttention reduces attention memory and supports scaling to the 8K context length.
  • Training process: Training used 512 A100 80 GB GPUs across 64 nodes with 3D parallelism and a 4M-token global batch size.The training process had few restarts and no significant instability; reported carbon emissions totaled 16.68 tonnes of CO2 for StarCoderBase and 0.58 tonnes for StarCoder fine-tuning.

6 Evaluation

The evaluation covers Python, realistic data-science workflows, open-domain coding, multiple programming languages, validity, security, and long-context behavior. StarCoder and StarCoderBase generally lead open-access comparisons, while benchmark choice affects observed rankings.

  • Python benchmarks: StarCoder is the highest-performing open-access model on HumanEval and MBPP, and outperforms OpenAI’s code-cushman-001.StarCoderBase is competitive with the similarly sized Python-fine-tuned CodeGen-16B-Mono.
  • DS-1000: StarCoder substantially outperforms all other models on DS-1000 data-science problems across every evaluated library.StarCoderBase also outperforms every other model but trails StarCoder slightly.
  • Benchmark interpretation: HumanEval and MBPP performance does not always correlate with DS-1000 performance, underscoring the importance of evaluating realistic workflows.CodeGen-Mono performs relatively well on HumanEval and MBPP but significantly worse on DS-1000.
  • Multilingual evaluation: Across all 19 programming languages, StarCoderBase outperforms other open-access models, sometimes by more than 2×.StarCoderBase is competitive with code-cushman-001 on most languages, with stated exceptions including C++, Java, Ruby, Swift, and Julia.
  • Multilingual evaluation: StarCoder remains competitive across most languages after Python fine-tuning and can slightly outperform StarCoderBase on some languages.The paper states that the reason for this behavior remains speculative.
  • Validity and security: StarCoderBase has the highest rate of valid code, while StarCoder has the lowest insecure-completion rate among models producing more than 95% valid code.These findings concern separate validity and security outcomes.

7 Natural Language Evaluation

StarCoderBase is evaluated beyond code generation on reasoning, language understanding, reading comprehension, social bias, and toxicity. Results show strengths relative to several code and general-purpose baselines, alongside remaining gaps.

  • Math reasoning: 21.5% PAL accuracy versus 8.4% CoT accuracy shows stronger StarCoderBase performance with program-aided reasoning on GSM8K.StarCoderBase exceeds both CodeGen-16B variants with PAL but trails LLaMA-33B.
  • World knowledge and reading comprehension: 34.2% versus 27.8% 5-shot MMLU accuracy places StarCoderBase ahead of CodeGen-16B-Multi.Both evaluated LLaMA models outperform StarCoderBase on MMLU.
  • World knowledge and reading comprehension: StarCoderBase performs better than CodeGen-16B-Multi on CoQA but is outperformed by LLaMA and GPT-NeoX.CoQA evaluates passage-based conversational question answering in a zero-shot setting.
  • Social bias: Across four StereoSet bias domains, StarCoderBase has the lowest stereotype scores while retaining competitive language-modeling scores.The paper interprets the combination as evidence that lower stereotype scores are not simply caused by worse language modeling.
  • Toxicity: 1.12% of StarCoderBase responses contain potentially offensive tokens, compared with 1.43% for LLaMA-13B.CodeGen-16B-Multi appears less toxic than StarCoderBase in this evaluation.
  • HELM reasoning tasks: StarCoderBase generally obtains substantially stronger performance than other models with released weights across the evaluated HELM reasoning tasks.The comparison ranks models by average task ranking.

8 Qualitative Evaluation

The qualitative evaluation illustrates how StarCoder models can be prompted through training-data templates and metadata. It also reports a prompting strategy that improves HumanEval pass@1 performance.

  • Qualitative examples: Examples demonstrate model behavior using templates for Git commits, GitHub issues, and Jupyter notebooks.These interactions are presented as starting points for further exploration of model capabilities.
  • Prompting: StarCoder can be prompted to act as a technical assistant without instruction tuning.The paper also combines metadata and natural language to obtain higher HumanEval pass@1 performance.

9 Attribution Tools

The paper provides public tools for inspecting whether model generations overlap with StarCoder’s training data and for tracing possible attribution. These tools support rapid first-pass checks and deeper dataset searches, but have defined coverage and matching limitations.

  • Attribution infrastructure: The Bloom-filter artifact occupies 26 GB, approximately 3% of the data size, and is hosted publicly alongside other documentation.It enables fast and lightweight membership inference over the training data.
  • Attribution infrastructure: The project combines a Bloom-filter-based Data Portrait with public search indexes to inspect training-data membership and attribution.The Data Portrait uses strings of length 50 characters, while Elasticsearch indexes support Python-subset and full-dataset queries.
  • Membership checking: The VSCode extension provides a rapid, first-pass check for approximate overlap between generated code and the training corpus.Users can follow an initial check with the full search index for further attribution assessment.
  • Limitations: The first-pass method requires matching strings to exceed a minimum length and does not filter common or generic code snippets.These constraints limit how confidently a quick match can establish attribution.

10 Social Impact and Limitations

StarCoder’s open-science project emphasizes responsible governance, but acknowledges limitations involving licensing, opt-out control, privacy, malicious code, model behavior, and evaluation scope. Community engagement informed concerns about data use and consent.

  • Open-science and governance: The project prioritized responsible governance, including withholding an identified malicious-code dataset despite its potential value for security research.This decision illustrates that open governance sometimes imposed limitations on adoption or future research.
  • Dataset and licensing: The Stack’s license detector may have incorrectly classified some repositories, limiting confidence that all training code was permissively licensed.The dataset was filtered for permissive licenses, but the detector remains fallible.
  • Opt-out process: The repository-level opt-out process cannot reliably remove duplicated copies of code, leaving large-scale data control and consent mechanisms incomplete.Permissively licensed or copyleft code may be duplicated across repositories before an individual repository is removed.
  • PII detection: StarCoder may still produce personally identifiable information because PII detection can yield false positives and false negatives across data types and programming languages.The detection model may require further validation and fine-tuning for specific use cases.
  • Malicious code: The Stack may contain undetected malicious code, and StarCoder might generate malware despite safety-oriented license restrictions.A detection tool identified 654 unsafe files before release, but residual risk remains.
  • Model limitations: StarCoder retains typical LLM risks, including inaccurate, offensive, misleading, discriminatory, or stereotyped outputs that require deployment-specific safety testing.The paper points to red-teaming and adversarial testing as examples of further safeguards.
  • Evaluation scope: The evaluation used only English-based benchmarks, leaving performance and limitations in other natural languages for future research.This bounds the reported evidence about broader language accessibility.
  • Code attribution tools: Attribution tools cannot detect excluded training code, may produce false positives, and do not distinguish generic boilerplate from protected content.The membership and BM25 tools cover only the training subset used for the project, while Portraits-based matching has minimum-resolution and context requirements.

11 Conclusion

The report presents StarCoder and StarCoderBase as transparent, open-access code models and evaluates them broadly against other Code LLMs. It concludes that the release combines competitive performance with reproducibility, responsible-use restrictions, and attribution tooling.

  • Conclusion: StarCoder and StarCoderBase are open-access 15.5B-parameter code LLMs developed by the BigCode community.The report describes the models as trained on code and released through an open-scientific collaboration.
  • Conclusion: The report provides transparency into training data, data curation, PII redaction, and model training, while open-sourcing model-building repositories.These materials are intended to increase access, reproducibility, and transparency for researchers and developers.
  • Conclusion: StarCoder outperforms CodeGen and CodeGeeX and matches or outperforms OpenAI’s closed-access code-cushman-001 model in the reported evaluation.The comparison is presented as the outcome of the report’s most extensive Code LLM evaluation to date.
  • Conclusion: The Open Responsible AI Model license includes use restrictions requiring model modifications and applications to adhere to responsible-AI principles.The release also includes attribution tools intended to help end users detect and locate model generations related to training data.

B Annotator countries

The appendix identifies the countries from which annotators were recruited for the PII-labeling effort.

  • Annotator countries: Table B.1 lists the countries from which annotators were recruited for the PII-labeling effort.

C Replacements for IP addresses

This appendix lists the private IP-address replacements used to mask IPv4 and IPv6 addresses.

  • The replacement mapping includes five private IPv4 addresses.
  • The replacement mapping includes five private IPv6 addresses.
  • Listing C.1 is titled “Replacements for IP addresses.”

D Additional Evaluation Results

This section presents additional evaluation tables and describes pretraining templates and technical-assistant prompting. The reported results cover multilingual performance, secure code generation, pass-count differences, and bias domains.

  • Table D.1 reports MultiPL-E HumanEval pass@1 for StarCoder and two closed-access models.Code-davinci-002 performs best, but has substantially higher parameter count and inference cost than StarCoder and code-cushman-001.
  • Table D.2 compares StarCoderBase and code-davinci-002 on secure-code generation using Asleep at the Keyboard.The caption states that code-davinci-002’s larger size does not appear to improve secure-code generation.
  • Table D.3 reports R pass counts out of 200 samples for selected problems where checkpoint differences reach at least 100.
  • Table D.4 reports CrowS-Pairs stereotype scores across bias domains, with scores nearer 50% indicating less bias.
  • Pretraining templates use sentinel tokens for git commits, GitHub issues, and formatted Jupyter notebooks to support targeted prompting.The formats prompt code modification, technical-question responses, and code generation from natural-language descriptions.
  • Prompting with an HHH-derived template made StarCoder a somewhat capable but brittle technical assistant without instruction-tuning.The authors report using its 8K context length to answer questions, follow instructions, and help solve technical problems.

E.3 Improving Code Generation with Prompting

The authors investigate prompting strategies for improving code generation and illustrate StarCoder’s technical-assistant behavior through example interactions. A filename-based prefix raises StarCoder’s HumanEval pass@1, while analogous prompting can hurt CodeGen-16B-Mono.

  • E.3 Improving Code Generation with Prompting: The prompting study addresses empty HumanEval solutions such as pass or “Insert code here,” a failure observed across all evaluated models.
  • E.3 Improving Code Generation with Prompting: 40.8% pass@1 on HumanEval follows a filename-based prefix at temperature 0.1, versus 33.6% unprompted.
  • E.3 Improving Code Generation with Prompting: The same temperature and prompt reduce CodeGen-16B-Mono’s performance to 28.10%, while the prompt has negligible impact on StarCoderBase.
  • F Technical Assistant Prompt: The Technical Assistant prompt combines helpfulness, politeness, honesty, sophistication, emotional awareness, humility, and caution about uncertain answers.
  • F Technical Assistant Prompt: Examples show the assistant generating code for alternating lists, CSV salary lookup, factorial-based combinations, translations, and other programming tasks.
  • F Technical Assistant Prompt: One interaction shows a refusal to provide test cases, despite nearby examples where the assistant supplies solutions and tests.
  • F Technical Assistant Prompt: The interactions include explanations and repairs for code, such as identifying factorial’s initial value and explaining a recursive power function.
Loading 2305.06161v2…