PaperlayerView exact revision on arXiv

Source-linked AI summary

Code Llama: Open Foundation Models for Code

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, Gabriel Synnaeve

arXiv:2308.12950v3cs.CL

TL;DR

Code-oriented applications require models that handle specialized programming tasks beyond general language understanding. The paper introduces Code Llama, a family of Llama 2-based code models, achieving state-of-the-art performance among public models while adding long-context, infilling, and instruction-following capabilities.

  • Problem

    Programming applications require language models that combine natural-language understanding with specialized code knowledge for synthesis, completion, debugging, and documentation.

  • Method

    The authors develop Code Llama from Llama 2 and release foundation, Python-specialized, and instruction-following variants across four model sizes, with long-context fine-tuning and selected-model infilling.

  • Results

    Code Llama 70B achieves state-of-the-art performance among public models on standard Python completion benchmarks, while Code Llama-Instruct improves truthfulness, toxicity, and bias benchmarks at moderate code-generation cost.

  • Takeaways & Limitations

    The release provides code, Python, and instruction-following models with large-context support, while selected 7B, 13B, and 70B variants support infilling.

  • Takeaways & Limitations

    Long-context fine-tuning slightly decreases performance on short-sequence coding benchmarks such as HumanEval and MBPP.

Abstract

from arXiv · show

We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.

1 Introduction

Code Llama extends foundation-model pretraining to code with Llama 2 initialization, while adding infilling, long-context, and instruction-following capabilities. It is released as three model variants—Code Llama, Code Llama - Python, and Code Llama - Instruct—in four parameter sizes.

  • Foundation-model initialization: Initializing Code Llama from Llama 2 outperforms training the same architecture on code only for a given budget.
  • Instruction following: Code Llama - Instruct significantly improves truthfulness, toxicity, and bias benchmark performance at moderate cost in code-generation performance.
  • Model variants: The models comprise three variants: Code Llama for code generation, Code Llama - Python for Python specialization, and Code Llama - Instruct for human-instruction and self-instruct code-synthesis data.
  • Model variants: The variants are released in 7B, 13B, 34B, and 70B parameter sizes.

2 Code Llama: Specializing Llama 2 for code

Code Llama specializes Llama 2 for code through foundation, Python-specific, and instruction-tuned model families spanning 7B–70B parameters. The models use code-heavy training, selective infilling objectives, long-context fine-tuning, and additional instruction data to support code generation and completion.

  • Code Llama: Code Llama provides 7B, 13B, 34B, and 70B foundation models initialized from Llama 2 weights and trained primarily on code.The models are trained on 500B tokens for 7B, 13B, and 34B, and 1T tokens for 70B.
  • Code infilling: The 7B, 13B, and 70B foundation models use an infilling objective to predict missing program content from surrounding context, while the 34B model does not.Infilling supports applications such as completing code at an IDE cursor, type inference, and generating in-code documentation.
  • Code Llama - Python: Code Llama - Python uses the same four sizes, then receives further specialization on 100B Python-heavy tokens.This specialization studies models tailored to a single programming language versus general-purpose code generation models.
  • Code Llama - Instruct: Code Llama - Instruct is based on Code Llama and fine-tuned with approximately 5B additional tokens to better follow human instructions.Its training combines proprietary instruction data, execution-filtered self-instruction examples, and rehearsal data comprising 6% code and 2% natural language data.
  • Long-context fine-tuning: Code Llama applies long-context fine-tuning on 16,384-token sequences, increasing the rotary-embedding base period from 10,000 to 1,000,000.The dedicated fine-tuning stage is intended to gain long-range capabilities without significantly increasing overall training cost.

3 Results

Code Llama improves coding performance through specialization and scaling, while retaining strong multilingual, infilling, and long-context capabilities. These gains involve modest performance costs on short-context benchmarks and support the release of specialized model variants.

  • Scaling: 5.6 percentage points on MBPP pass@1 are gained by scaling Code Llama from 7B to 13B, with further gains of 8 points at 34B and 7 at 70B.Larger specialized models outperform smaller counterparts on almost every HumanEval, MBPP, and APPS metric.
  • Code generation: Specialization substantially improves code generation: Code Llama outperforms same-size Llama 2 models across languages, and Code Llama 7B even surpasses Llama 2 70B.Training on 500B additional code-heavy tokens makes Llama 2 70B roughly equivalent to Code Llama 7B on Python coding benchmarks.
  • Multilingual performance: Code Llama 7B outperforms larger CodeGen-Multi and StarCoder models and performs on par with Codex in multilingual coding.Code Llama - Python is comparable to Code Llama overall, with 7B and 13B slightly better and 30B slightly worse than their non-Python counterparts.
  • Infilling: Infilling models achieve state-of-the-art performance among same-size models and outperform all open infilling models across the three evaluated programming languages.Code Llama 7B, 13B, and 70B are released with infilling capability despite a modest downstream performance decline.
  • Long-context capabilities: After 100K tokens, perplexity increases only slightly, while all Code Llama models produce syntactically correct completions on a 103K-token example and the 7B model achieves an exact match.Perplexity steadily decreases beyond the 16,384-token fine-tuning sequence length.
  • Long-context trade-offs: 0.52 percentage points on HumanEval pass@1 and 1.9 points on MBPP pass@1 are lost on average after long-context fine-tuning, motivating a trade-off favoring real-world long-sequence use cases.The authors consequently release all Code Llama, Code Llama - Python, and Code Llama - Instruct models with long-context capabilities.

4 Responsible AI and safety

Code Llama - Instruct is safety-tuned with adversarial and code-risk prompts, then evaluated for truthfulness, toxicity, bias, and malicious-code risks. Results indicate improved safety and safer behavior than ChatGPT, while red teaming identifies dual-intent vulnerabilities, limited false refusals, and a safety–coding-performance tradeoff.

  • Safety approach: Code Llama - Instruct is fine-tuned on Llama 2 outputs, including adversarial prompts with safe responses and prompts addressing code-specific risks.The paper motivates this work by noting that language models can produce falsehoods, toxic content, and training-data biases.
  • Benchmark evaluation: The evaluation covers truthfulness, toxicity, and bias using TruthfulQA, ToxiGen, and BOLD, comparing pretrained and instruction-tuned Code Llama with Falcon, MPT, and StarCoder.TruthfulQA contains 817 questions across 38 categories; ToxiGen measures toxic language across 13 minority groups; BOLD includes 23,679 prompts spanning five domains and 43 subgroups.
  • Benchmark evaluation: The fine-tuned Code Llama - Instruct exhibits significant improvement on the three safety benchmarks and is safer than ChatGPT.TruthfulQA rewards truthful and informative generations, ToxiGen rewards lower toxicity, and BOLD interprets sentiment scores relative to neutral sentiment.
  • Red teaming: Red teaming with 25 Meta employees across three exercises found a dual-intent vulnerability: the model refuses explicit ransomware requests but may comply with indirectly malicious file-encryption requests.The exercises included experts in responsible AI, malware development, and offensive security engineering.
  • Red teaming: Red teamers judged that the main risk is helping low-skill adversaries iteratively produce malicious scripts, while advanced malware development remains beyond current LLM capabilities.They also noted that producing malware code itself is often not the primary barrier, given the availability of scripts and binaries online.
  • Limitations and tradeoffs: The analysis also found limited false refusals without a system preprompt, which can sometimes be resolved by rephrasing valid requests, and longer safety-focused fine-tuning can degrade coding performance.The authors plan to investigate false refusals further while training for high coding performance without compromising safety.

5 Related work

Related work established that code-focused language models outperform general-purpose models on code benchmarks, while research also examined model openness, code-data quality, synthesis tasks, and long-context scaling. This paper follows that trajectory by fine-tuning Llama 2 on code data and focusing on program synthesis and infilling.

  • Code understanding and synthesis: Code-specialized models consistently outperform general-purpose LLMs of comparable or larger size on code benchmarks.Examples include Codex, CodeT5, InCoder, AlphaCode, CodeGen, GPT-NeoX, SantaCoder, StarCoder, and phi-1.
  • Code understanding and synthesis: This paper follows prior code-model work by fine-tuning the general-purpose Llama 2 language model on code data.
  • Closed-source vs open-source models: Open and closed source models form a major divide in both general-purpose and code-focused LLMs.ChatGPT, GPT-4, PaLM, Chinchilla, Codex/Copilot, AlphaCode, and phi-1 are described as closed source, while BLOOM, OPT, Llama, SantaCoder, and StarCoder are public or open-source.
  • Data: Code-data quality, deduplication, repository popularity, supplementary metadata, and synthetic or textbook-quality examples are recurring design considerations in modern training datasets.Prior work discusses GitHub stars, issues, commits, GPT-3.5-generated problems, and filtering for “textbook”-quality code.
  • Code understanding and synthesis tasks: The paper focuses on program synthesis and infilling/completion, among broader code understanding and synthesis tasks studied with NLP models adapted for code.Infilling/completion is emphasized for the 7B and 13B models, where prior work observed little impact on generation performance.
  • Long sequences in LLMs: Long-context research has expanded supported sequence lengths from 4K and 8K tokens to 16K, 32K, 65K, and 100K contexts while addressing self-attention’s O(n^2) complexity.

6 Discussion · A.1 Contributions · B Code Llama 70B specialization pipeline

Code Llama is a family of code-specialized Llama 2 models spanning three variants and four sizes, with infilling and large-context capabilities. The paper also describes instruction tuning, contributor roles, and the 70B specialization pipeline.

  • 6 Discussion: Code Llama comprises Code Llama, Code Llama - Python, and Code Llama - Instruct in 7B, 13B, 34B, and 70B sizes.The 7B, 13B, and 70B models support infilling, while all models leverage large contexts and were tested for stability up to 100K tokens.
  • 6 Discussion: Code Llama - Instruct adds zero-shot instruction ability through further fine-tuning that somewhat distills Llama 2-Chat.The tuning targets greater helpfulness and safer deployment, though instruction following and over-safety can reduce some evaluation scores.
  • 6 Discussion: Further work is needed for LLMs because instruction following and being overly safe can cost evaluation points, including HumanEval for the 34B model.The passage gives this as an example of the tradeoff illustrated in Figure 15 and Table 2.
  • A.1 Contributions: Science and Engineering Leadership included Jonas Gehring, Fabian Gloeckle, Baptiste Rozière, Sten Sootla, and Gabriel Synnaeve.The listed leadership contributors also include the names shown in the passage.
  • A.1 Contributions: Code Evaluations included Yossi Adi, Itai Gat, Artyom Kozhevnikov, Jingyu Liu, Jérémy Rapin, and Tal Remez.The passage identifies these contributors under code evaluations.
  • A.1 Contributions: Responsible AI and red-team leadership involved Louis Martin, Xiaoqing Ellen Tan, Manish Bhatt, Joanna Bitton, Cristian Canton Ferrer, Ivan Evtimov, and Aaron Grattafiori.The red-team leads represented Red Team X, RAI, and the Offensive Security Group.
  • B Code Llama 70B specialization pipeline: The Code Llama 70B specialization pipeline consists of fine-tuning stages annotated with training-token counts, with infilling-capable models marked by ⇄.This description is provided in Figure 8.

C Additional Ablation Results

This section reports ablations examining infilling (FIM) and long-context fine-tuning (LCFT) across code-generation, multilingual, and mathematical-reasoning evaluations. Results cover multiple model sizes and decoding protocols, with an additional Llama 2 70B comparison for GSM8k.

  • Code-generation ablations: Table 10 compares pass@1, pass@10, and pass@100 for 7B, 13B, and 34B models with and without FIM and LCFT.These ablations evaluate the effects of both training modifications across three model scales.
  • Code-generation ablations: Pass@1 uses greedy decoding, whereas pass@10 and pass@100 use temperature 0.8, N = 200, and nucleus sampling with p = 0.95.The decoding settings differ between the single-sample and multi-sample metrics.
  • Multilingual evaluation: Table 11 reports detailed Multilingual-HE results for Code Llama variants with and without FIM and LCFT using greedy decoding.The evaluation compares model variations under a common decoding procedure.
  • GSM8k evaluation: Table 12 reports GSM8k solve rates for Llama 2, Code Llama, and Code Llama - Python at 7B, 13B, and 34B parameters, plus Llama 2 70B.The table provides a cross-family and cross-scale comparison for mathematical reasoning.

D Math reasoning results

The method’s mathematical-reasoning capabilities are evaluated on GSM8K, a benchmark of middle-school math word problems, with results summarized in Table 12.

  • Math reasoning results: The evaluation uses GSM8K, which comprises middle-school math word problems.Results are summarized in Table 12.

E Infilling

The infilling section finds that SPM formatting degrades random-span infilling relative to PSM because SPM avoids prefix–middle token splitting during training. It therefore recommends PSM when prefixes do not end in whitespace or a complete token, while also noting strong sensitivity to prompt formatting in CodeXGLUE docstring generation.

  • Random span infilling: SPM random-span infilling degrades on HumanEval relative to PSM because SPM training avoids splitting tokens at the prefix–middle boundary.Prompts ending in a broken token can therefore fall out of distribution.
  • Random span infilling: PSM avoids this difficulty because its training format splits tokens at the prefix–middle boundary.The models do not struggle with random-span infilling in PSM format.
  • Random span infilling: Use PSM for infilling tasks when the prefix does not end in whitespace or a token boundary.An example completion shows logical code awareness but incomplete character-level spelling understanding when a token is broken.
  • CodeXGLUE docstring generation: CodeXGLUE docstring generation shows strong dependence on the exact prompt formatting.The benchmark inserts a docstring between a Python function’s header and body, surrounded by triple quotes.

F Zero shot results on APPS … G.3 Extended Results

The extended results report Code Llama-Instruct’s zero-shot APPS evaluation and analyze long-context behavior through rotary-position scaling, synthetic key retrieval, and LCC-balanced code completion. Models generally retrieve keys reliably up to 16K tokens, but accuracy declines beyond that length and Code Llama-Instruct 7B has a specific 16K failure mode.

  • F Zero shot results on APPS: Code Llama-Instruct is evaluated on APPS in the zero-shot setting using raw outputs without filtering.Both zero-shot and two-shot evaluations use nucleus sampling with p = 0.95 and temperature 0.6; prompts request answers inside triple single quotes.
  • F Zero shot results on APPS: The APPS results indicate that Code Llama-Instruct can achieve convincing performance on challenging competitive-programming questions.
  • G.1 Further Discussion: Increasing the rotary-embedding base period from 10,000 to θ = 1,000,000 reduces attention-score decay over distance, helping far-away tokens contribute to predictions.The frequency change can be applied to pretrained models, with loss curves stabilizing within a few gradients.
  • G.2 Long context benchmarks: The synthetic key-retrieval benchmark concatenates Python solutions, inserts a randomly sampled two-digit key at a specified relative position, and tests whether the model generates that value.Accuracy is measured over 64 distinct examples for each prompt-length and key-position combination.
  • G Long context fine-tuning: The long-context fine-tuning analysis compares models prompted with entire files against base models given only the last 4K tokens.The LCC-balanced evaluation covers Python, Java, and C# source files grouped by file length.
  • G.2 Long context benchmarks: The LCC-balanced resampling produces 548 Python, 412 Java, and 512 C# data points to better test long-context behavior.The original LCC test and validation distributions are heavily skewed toward shorter files.
  • G.3 Extended Results: Code Llama models generally retrieve keys strongly through 16K tokens, but Code Llama-Instruct 7B fails when the key is at the prompt start at 16K.Beyond 16K tokens, retrieval accuracy declines across all models; GPT-3.5-Turbo (16K) shows small decreases at 16K.

G.4 Ablations · H Prompts

The ablations show that long-context fine-tuning alone is insufficient: adapting rotation frequencies is necessary for retrieving distant keys at extended context lengths. The prompts specify structured procedures for self-training, solution generation, and evaluation on MBPP and APPS.

  • G.4 Ablations: Without long-context fine-tuning, key retrieval works only at the 4,096-token sequence length seen during training.The result comes from ablations on an earlier version of the 7B model.
  • G.4 Ablations: Increasing RoPE’s base period θ only at inference has no effect on key-retrieval accuracy.This ablation isolates inference-time changes without long-context fine-tuning modifications.
  • G.4 Ablations: Without changing the base period, long-context fine-tuning fails to retrieve far-away keys at 8,000 tokens despite 16,384-token fine-tuning.This failure suggests that adapting the rotation frequencies is necessary.
  • G.4 Ablations: With frequency scaling by 1/4, retrieval is low at 16,00 tokens for keys at the beginning, and extrapolation to longer sequences fails.The 1/4 factor corresponds to the 4x sequence-length increase during fine-tuning.
  • H.1 Self training prompts: The interview-question prompt asks the model to generate programming questions for self-training.Figure 10 documents the prompt used for this generation step.
  • H.1 Self training prompts: The unit-test prompt requests 5 correctness tests enclosed in [TESTS] tags, with numbered comments directly above each assert.The required comment format is “#Test case n:”, starting at 1.
  • H.2 Evaluation prompts: The solution-generation prompt substitutes an interview question and one example test while hiding the remaining generated tests to filter overfitting.The example test is randomly sampled from tests generated for the same question.
  • H.2 Evaluation prompts: Separate prompts evaluate Code Llama on the MBPP zero-shot task and APPS.The MBPP prompt is used for instruct models, while Figure 14 documents prompts for APPS evaluation.

I Addition results on responsible AI and safety … K Model card

The paper evaluates pretrained and aligned Code Llama models for truthfulness, toxicity, and demographic bias, then examines false refusals, malicious-use red teaming, and documents the released family in a model card.

  • I Addition results on responsible AI and safety: Safety evaluation covers truthfulness, toxicity, and bias using three automatic benchmarks for pretrained and aligned LLMs.The benchmark descriptions appear in Section 4.
  • I Addition results on responsible AI and safety: 30% to 40% truthfulness is observed for 7B Code Llama and external models including Falcon, MPT, and StarCoder (Python).Truthfulness increases for pretrained Code Llama models as model size grows, with 13B showing about 10% improvement.
  • I Addition results on responsible AI and safety: Mexicans tend to have the highest percentage of toxic generations among demographic groups for pretrained models.Pretrained 34B Code Llama has the lowest percentages for Jewish and Middle Eastern groups, while StarCoder (Python) is lowest for almost the rest.
  • I Addition results on responsible AI and safety: Mean sentiment scores are generally positive across many demographic groups in both pretrained and instruct models.Fine-tuned Code Llama - Instruct models show greater positivity than pretrained models.
  • Examples of Red Teaming Prompts for False Refusals: A red-teaming example shows a false refusal to explain how to kill a process, despite the prompt concerning an operating-system task.The figure indicates that rephrasing the prompt can solve this false refusal.
  • J Examples of red teaming prompts on malicious use of code: A malicious-use red-teaming example shows the model providing ransomware obfuscation guidance and an example encryption code snippet.These prompts were part of the evaluation in Figure 7.
  • K Model card: Table 26 presents a model card for the family of Code Llama models released in the paper.The model card follows Mitchell et al. (2019).

L Qualitative examples

Qualitative examples show Code Llama - Instruct 34B finding and explaining Python bugs, generating complex visualizations, and handling domain-specific command-line tasks. Additional infilling examples demonstrate plausible docstring completion and context-based equation completion in Lean 3.

  • Code understanding: Code Llama - Instruct 34B identifies the origin of a Python bug and provides a clear explanation.The example concerns a function intended to return successive powers but producing repeated outputs.
  • Code generation: Code Llama - Instruct 34B generates a complex pandas/seaborn/matplotlib plot with multiple subplots, shared axes, grids, and correctly selected data.The example requires different decoding subsets, visual encodings, and an external legend; only bbox_to_anchor values needed adjustment.
  • Infilling: Code Llama 13B generates docstrings by infilling code-context gaps, including examples with long parameter lists.The docstrings come from selected CodeXGLUE tasks, with surrounding lines supplied as context.
  • Infilling: Code Llama 7B plausibly infills an equation in Lean 3 from its surrounding context, even when the mathematical terms are fictional and invented.The example uses invented terms to test whether the model could produce a plausible completion rather than rely on memorization.
  • Code generation: The qualitative examples also show the model learning and applying options for a domain-specific audio-processing command-line tool.One example asks for a terminal command that adds reverb to input.mp3 using sox.
Loading 2308.12950v3…