Code Llama: Open Foundation Models for Code

Code-training from foundation models

• Initializing from Llama 2 pretrained on general-purpose text and code outperforms training the same architecture on code only for a given budget.Code Llama is developed by gradually specializing and increasing the capabilities of Llama 2 through cascaded training and fine-tuning steps.Source: §1 · p. 1

Infilling and long input contexts

• A multitask objective combines autoregressive and causal infilling prediction for 7B, 13B, and 70B models, while RoPE fine-tuning extends context from 4,096 to 100,000 tokens.These changes support filling missing text using surrounding context and repository-level completion or synthesis; very large contexts have a moderate impact on standard coding benchmark performance.Source: §1 · p. 2

Instruction fine-tuning

• Code Llama - Instruct variants use proprietary instruction data and machine-generated self-instruct coding data, improving safety and helpfulness with moderate code-generation performance cost.The paper reports significant improvements on truthfulness, toxicity, and bias benchmarks.Source: §1 · p. 2

Model family

• The release includes Code Llama foundation, Python-specialized, and Instruct variants in 7B, 13B, 34B, and 70B sizes.The Instruct variant is fine-tuned with human instructions and self-instruct code-synthesis data.Source: §1 · p. 2

Evaluation

• On HumanEval, MBPP, APPS, and MultiPL-E, the best Code Llama models establish a new state of the art among open-source LLMs.The paper describes exhaustive evaluations on these major code-generation benchmarks.Source: §1 · p. 2

2.1 The Code Llama models family

• The family includes 7B, 13B, 34B, and 70B foundation models, Python-specialized variants, and instruction-tuned models; most variants start from Llama 2 weights.The general-purpose models use 500B training tokens, except 70B, which uses 1T; Python models add 100B Python-heavy tokens, and the 7B, 13B, and 34B Instruct models receive about 5B instruction-tuning tokens. Code Llama - Instruct 70B is trained from Code Llama - Python 70B.Source: §2.1 · p. 3

2.2 Dataset

• Code Llama training uses near-deduplicated public code, code-related natural-language samples, and a small amount of natural-language data; preliminary experiments found the latter improved MBPP performance.The natural-language datasets include discussions and code snippets in questions or answers; 8% of samples come from code-related natural-language data.Source: §2.2 · p. 3, Table 1 · p. 5

2.3 Infilling

• For infilling, 7B, 13B, and 70B models learn to predict a missing code span from surrounding context using randomly formatted prefix-suffix-middle or suffix-prefix-middle sequences.The training transformation is applied with probability 0.9; 34B is trained without the infilling objective.Source: §2.1 · p. 3, §2.3 · p. 4

2.4 Long context fine-tuning

• Long-context fine-tuning uses 16,384-token sequences and raises the rotary-embedding base period from 10,000 to 1,000,000, enabling stable behavior on sequences up to 100,000 tokens.This fine-tuning stage increases context length from the 4,096 tokens used for Llama 2 and initial code training without significantly increasing overall training cost.Source: §2.4 · p. 4

2.5 Instruction fine-tuning

• Instruction tuning combines Llama 2’s RLHF V5 dialogue data with execution-filtered self-instruction examples and code and natural-language rehearsal data.The self-instruction dataset contains about 14,000 question-test-solution triplets, selected by retaining generated solutions that pass their tests; rehearsal uses 6% code data and 2% natural-language data.Source: §2.5 · p. 5

2.6 Training details

• Training details specify AdamW with a cosine schedule and 1,000 warm-up steps, while instruction tuning uses a 524,288-token batch and about 5B tokens total.For long-context fine-tuning, the default is 10,000 gradient steps, with different step counts specified for 34B models and Code Llama 7B.Source: §2.4 · p. 4, §2.6 · pp. 5–6

3.1.1 Python code generation

• Code specialization boosts HumanEval and MBPP, while scaling specialized models improves nearly every HumanEval, MBPP, and APPS metric.Python specialization adds 4.3–8.3 percentage points on HumanEval pass@1 and 1.2–6.4 points on MBPP pass@1; Code Llama - Python 7B also outperforms Code Llama 13B on both benchmarks. Scaling Code Llama from 7B to 13B adds 5.6 points on MBPP pass@1, with 8 more points at 34B and 7 at 70B; Python specialization slightly lowers introductory and interview-level APPS performance but improves competition-level problems.Source: §3.1.1 · p. 7

3.1.2 Multilingual evaluation

• On code generation in any language, Code Llama models clearly outperform same-size Llama 2 models, and Code Llama 7B even outperforms Llama 2 70B.Code Llama-Python is comparable overall: 7B and 13B perform slightly better than counterparts without Python fine-tuning, while 30B performs slightly worse.Source: §3.1.2 · pp. 8–9

3.2 Infilling evaluations

• Code Llama infilling models outperform same-size alternatives on HumanEval infilling and open infilling models across three MultiPL-E languages.On the HumanEval-derived random-span task, performance is significantly worse in suffix-prefix-middle (SPM) than prefix-suffix-middle (PSM) format because token healing is not implemented; on multilingual single-line infilling, SPM prompting further improves performance.Source: §3.2 · p. 10, Table 6 · p. 11

3.3 Long context evaluations

• Perplexity decreases beyond the 16,384-token fine-tuning length and rises only slightly after 100K tokens, while retrieval declines when extrapolating beyond 16K.Retrieval is strong within the trained sequence length except for 7B cases where the function is at the prompt beginning.Source: §3.3 · pp. 10–11, Figure 4 · p. 12

3.2 Infilling evaluations

• Infilling training reduces average HumanEval and MBPP pass@1, pass@10, and pass@100 scores by 0.6 points for 7B and 1.1 for 13B.The authors release 7B, 13B, and 70B models with infilling despite the modest decline, citing the capability’s broad applicability.Source: §3.2 · pp. 9–10, Table 5 · p. 11

3.3 Long context evaluations

• Long-context fine-tuning improves single-line completion, but slightly reduces performance on short-context coding benchmarks.On an example with a 103K-token prompt, all models generate syntactically correct completions and 7B produces an exact match. LCFT reduces HumanEval pass@1 by 0.52 percentage points and MBPP pass@1 by 1.9 points on average; the authors nevertheless release all model families with long-context capabilities.Source: §3.3 · pp. 11–12, Table 7 · p. 12

3.4.1 Fine tuning Llama 2 vs. training from scratch on code

• Training Code Llama 7B from scratch reaches the fine-tuned model’s final loss only after about 240B more training tokens.The loss gap between the scratch-trained and fine-tuned models grows over time.Source: §3.4.1 · p. 13

3.4.2 Instruction fine-tuning

• Code Llama - Instruct improves coding ability across model sizes while preserving Llama 2’s general helpfulness performance.The helpfulness comparison uses the Llama 2 reward model and should be treated cautiously. Self-instruct data improves HumanEval and MBPP scores and makes MBPP zero-shot output formatting more reliable; a 34B Python model trained on 15,000 unnatural instructions also shows clear improvements on both coding benchmarks.Source: §3.4.2 · p. 13, Table 8 · p. 14

Responsible AI and safety

• Code Llama - Instruct is fine-tuned on Llama 2 outputs, including adversarial prompts with safe responses and prompts addressing code-specific risks.Safety evaluations cover truthfulness, toxicity, and bias in pretrained Code Llama and fine-tuned Code Llama - Instruct, alongside Falcon, MPT, and StarCoder.Source: §4 · p. 15

Red teaming

• Standard automatic benchmarks do not comprehensively capture model risks, so the authors also conduct red teaming with domain experts in responsible AI, malware development, offensive security, and software engineering.The authors report three exercises involving 25 Meta employees.Source: §4 · pp. 15–16

Benchmark evaluation results

• For 34B, TruthfulQA rises from 34.64% to 47.37% and ToxiGen toxicity falls from 17.62% to 0.00% after instruction fine-tuning.TruthfulQA reports generations that are both truthful and informative; lower ToxiGen toxicity is better.Source: §4 · p. 15, Table 9 · p. 17
• Code Llama - Instruct is the least toxic across Code Llama sizes and ranks second in truthfulness and toxicity behind Llama 2 Chat; it also tends toward more positive BOLD sentiment.BOLD sentiment increases overall for many demographic groups after fine-tuning.Source: §4 · p. 15, Table 9 · p. 17

Red teaming

• Red teamers found that Code Llama may comply with dual-intent code requests, such as encrypting all files, despite refusing explicit ransomware requests.Some experts questioned the risk of malicious code generation, but noted that iterative assistance could help low-skill adversaries produce malicious scripts; another assessment said advanced malware development exceeds current LLM capabilities.Source: §4 · p. 16

False refusals

• Red teamers found limited evidence of false refusals without a system preprompt, and some cases could be addressed by rephrasing the prompt.The authors say they plan to investigate the behavior further.Source: §4 · pp. 16–17

Red teaming

• Code Llama - Instruct models are safer than ChatGPT by the safety reward model, with Code Llama responses concentrated more in the safer score range.The analysis scored responses to ChatGPT (GPT3.5 Turbo) using the Llama v2 70B safety reward model on red-teamer prompts seeking malicious code.Source: §4 · pp. 16–17, Figure 7 · p. 16

Safety and coding performance

• Because safety is prioritized in instruction fine-tuning, longer fine-tuning tends to degrade coding performance; the authors trained models to preserve high coding performance without compromising safety.Source: §4 · p. 17

Related work

• Adding code to training data enabled synthesis with medium-sized LLMs, and code-specialized models later outperformed general-purpose models of comparable or larger size.The paper follows this line by fine-tuning Llama 2 on code data.Source: §5 · p. 17

Closed-source vs open-source models

• 70B-parameter Code Llama exceeds the sizes of GPT-NeoX-20B and StarCoder (15.5B) and achieves state-of-the-art HumanEval, MBPP, and MultiPL-E performance among open-source models.The models are released for commercial use under the same terms as Llama 2.Source: §5 · p. 18

Data

• The paper trains on publicly available code without issues, commits, or synthetic exercises, unlike approaches using repository metadata or generated textbook-quality problems.It avoids synthetic exercises to reduce the risk of narrowing models to simple coding tasks such as HumanEval and MBPP.Source: §5 · p. 18

Code understanding and synthesis tasks

• Among a broad range of code-understanding and synthesis tasks, this work focuses on program synthesis and infilling or completion for 7B and 13B models.The authors note that infilling can be added with little impact on generation performance, as previously observed by Bavarian et al. (2022).Source: §5 · p. 18

Additional modifications to LLM training and inference

• Rather than adding specialized training objectives or execution- and test-based inference schemes, the paper prioritizes improving the base model.The authors describe more elaborate inference schemes as a possible direction to explore on top of Code Llama.Source: §5 · p. 18

Long sequences in LLMs

• For long contexts, the paper modifies Llama 2’s RoPE hyperparameters to enable inference-time extrapolation without architectural changes or hand-crafted sparsity patterns.Unlike Chen et al. (2023b), the approach tailors its hyperparameter modifications for extrapolation at inference time.Source: §5 · p. 19

6 Discussion

• The 70B model is state-of-the-art among public models on standard Python completion benchmarks, while Code Llama 7B outperforms every other public model on multilingual benchmarks.Source: §6 · p. 19
• The family includes Code Llama, Code Llama - Python, and Code Llama - Instruct in 7B, 13B, 34B, and 70B sizes; all models leverage large contexts.The 7B, 13B, and 70B models support infilling, and inference stability was tested up to 100K tokens.Source: §6 · p. 19
• Large-context fine-tuning and infilling come at a cost on short-sequence, left-to-right code-generation benchmarks, though other models are competitive with similarly sized models.Source: §6 · p. 19
• Code Llama - Instruct aims for zero-shot instruction following, greater helpfulness, and safer deployment through further fine-tuning that somewhat distills Llama 2-Chat.Source: §6 · p. 19
• Following instructions and being overly safe can reduce evaluation scores, and the discussion calls for better handling of context and nuance in instructions.The passage gives HumanEval for the 34B model as an example.Source: §6 · p. 19

A.1 Contributions

• The contributions section credits science and engineering leadership, code evaluations, responsible AI, red-team leadership, and other contributors.Source: §A.1 · p. 26

B Code Llama 70B specialization pipeline

• Figure 8 depicts the Code Llama 70B specialization pipeline, labeling fine-tuning stages with training-token counts and marking infilling-capable models.Source: Figure 8 · p. 28

C Additional Ablation Results

• Table 10 reports pass@1, pass@10, and pass@100 for 7B, 13B, and 34B models with and without FIM and LCFT.Pass@1 uses greedy decoding; pass@10 and pass@100 use temperature 0.8, N = 200, and nucleus sampling with p = 0.95.Source: C Additional Ablation Results · p. 28
• Table 11 reports detailed MultiPL-E results for Code Llama variants with and without FIM and LCFT, using greedy decoding.The caption does not provide individual scores or identify a winning variant.Source: Table 11 · p. 29

D Math reasoning results

• GSM8K evaluates middle-school math word problems, and the method’s results are summarized in Table 12.Source: D Math reasoning results · p. 29

• Random-span HumanEval infilling degrades in SPM versus PSM because SPM avoids training on broken prefix-middle tokens, making such prompts out of distribution.Source: E Infilling · p. 30
• Use PSM when prefixes end mid-token, or SPM with token healing; for example, SPM may complete “enu” as “emrate” rather than “merate.”Source: E Infilling · p. 30
• Python CodeXGLUE docstring-generation evaluations showed good results despite no training on specialized datasets aligning code and natural text, such as those used by Li et al. (2023).Prompts used a triple quote followed by a space, removed the closing triple quote, and scored trimmed first nonempty lines against cleaned references with smoothed 4-gram BLEU. The models evaluated may have trained on datasets overlapping the benchmark.Source: E Infilling · p. 30

F Zero shot results on APPS

• Zero-shot and two-shot APPS evaluations use nucleus sampling with p = 0.95 and temperature 0.6, plus guidance tailored to standard-IO or function-signature questions.Prompts ask models to place final code inside triple single quotes for easier extraction.Source: F Zero shot results on APPS · p. 30
• Code Llama - Instruct achieves convincing zero-shot results on challenging APPS questions despite no task-specific fine-tuning or few-shot examples.Results use raw outputs without filtering.Source: F Zero shot results on APPS · pp. 30–31, Table 15 · p. 31

G.1 Further Discussion

• Raising the RoPE base period from 10,000 to θ = 1,000,000 reduces attention-score decay with distance, helping far-away tokens contribute to predictions.The modification can be applied to pretrained models, with loss curves stabilizing within a few gradient steps at low learning rates.Source: §G.1 · p. 31, Figure 9 · p. 32

G.2 Long context benchmarks

• The synthetic key-retrieval benchmark concatenates CodeContest Python solutions, inserts a randomly sampled two-digit value, and measures correct retrieval across prompt lengths and key positions.Each prompt ends with an assertion requesting the inserted value; accuracy is measured over 64 examples per prompt-length and key-position combination.Source: §G.2 · pp. 31–32
• To test long-context code completion, LCC-balanced resamples the short-file-skewed LCC validation and test sets into Python, Java, and C# subsets.The resampled corpus contains 548 Python, 412 Java, and 512 C# data points.Source: §G.2 · p. 32, Table 16 · p. 32, Figure 9 · p. 32

G.3 Extended Results

• Up to 16K tokens, Code Llama models generally retrieve keys strongly, but Instruct 7B fails for keys at the prompt start and all models decline beyond 16K.These results cover the synthetic key-retrieval task.Source: §G.3 · p. 33, Table 17 · p. 33
• At 16K-token prompts, GPT-3.5-Turbo shows small performance decreases; 24K-token prompts exceed its API limit.The paper notes that 16K tokens correspond to 12K tokens with the GPT-3.5 tokenizer.Source: §G.3 · p. 33

G.4 Ablations

• Without long-context fine-tuning, key retrieval was possible only at the 4,096-token sequence length seen during training.The ablations report key-retrieval accuracy for an earlier version of the 7B model in Table 18.Source: §G.4 · p. 33
• Keeping RoPE’s base period unchanged caused far-away key retrieval to fail at 8,000 tokens despite fine-tuning on 16,384-token sequences.This failure suggests adapting the rotation frequencies is necessary; increasing RoPE’s base period for inference alone had no effect. Frequency scaling by 1/4, matching a 4x increase in fine-tuning sequence length, still yielded low retrieval for beginning keys at 16,00 tokens and failed at longer sequences.Source: §G.4 · p. 33

H.1 Self training prompts

• Figure 10 shows the prompt used to generate interview-style programming questions.Source: Figure 10 · p. 34
• Figure 11’s unit-test prompt uses %%%question%%% as a runtime placeholder for an interview-style programming question.Source: Figure 11 · p. 34

H.2 Evaluation prompts

• The solution-generation prompt supplies one sampled test and hides the remaining generated tests to filter solutions that overfit the prompt tests.The prompt uses %%%question%%% and %%%test%%% as placeholders for the question and example test.Source: Figure 12 · p. 35
• Figure 13 gives the prompt used to evaluate instruct models on the MBPP zero-shot task.Source: Figure 13 · p. 35
• Figure 14 shows the prompts used to evaluate Code Llama on APPS.Source: Figure 14 · p. 35

I Addition results on responsible AI and safety

• 34B Code Llama - Instruct reaches 50.92% truthfulness and 96.33% informativeness on TruthfulQA.Instruction-tuned models at all three sizes exceed 90% informativeness; pretrained truthfulness is around 30%–40% for 7B Code Llama and rises with model size.Source: I Addition results on responsible AI… · p. 36, Table 19 · p. 37
• After instruction fine-tuning, Code Llama - Instruct models at all three sizes produce effectively zero toxic generations across demographic groups.Among pretrained models, Mexicans have the highest toxic-generation percentages; pretrained 34B Code Llama is lowest for Jewish and Middle Eastern groups, while StarCoder (Python) is lowest for most others.Source: I Addition results on responsible AI… · p. 36, Table 20 · p. 37
• BOLD sentiment scores are generally positive across many demographic groups, with Code Llama - Instruct scores more positive than pretrained scores.The 13B variants tend toward more neutral scores than 7B and 70B; patterns resemble Llama 2 Chat. Scores vary by domain, including negative scores for Fascism and positive scores for several occupational groups.Source: I Addition results on responsible AI… · p. 36, Table 21 · p. 38, Table 22 · p. 38

Examples of Red Teaming Prompts for False Refusals

• A red-team example shows a false refusal that can be resolved by rephrasing the prompt.Figure 15 presents the initial false refusal and a rephrased prompt as the solution.Source: Figure 15 · p. 40

J Examples of red teaming prompts on malicious use of code

• Figure 16 gives examples of red-teaming prompts for malicious code use that were included in the Figure 7 evaluation.Source: Figure 16 · p. 41

K Model card

• The appendix presents a model card for the released Code Llama family.The model card follows Mitchell et al. (2019).Source: K Model card · p. 42, Table 26 · p. 42

L Qualitative examples

• Code Llama - Instruct 34B identifies a Python bug’s origin and explains it clearly.Figure 17 presents an example of standard Python bugs found and explained by the model.Source: L Qualitative examples · p. 43, Figure 17 · p. 44
• Code Llama 13B generates docstrings from CodeXGLUE task context, including examples with long parameter lists.Figure 20 marks generated docstrings and shows surrounding code as context.Source: L Qualitative examples · p. 43, Figure 20 · p. 47
• In a fictional Lean 3 problem, Code Llama 7B plausibly infills an equation using mathematical context, including invented terms and notation.The example’s context suggests its quantification, variable, and expression of 1-connectedness.Source: L Qualitative examples · p. 43, Figure 21 · p. 48
• For a complex pandas, matplotlib, and seaborn plot, Code Llama generates multiple plots, shared axes, and a grid, while selecting data and using appropriate arguments.Only the legend positioning required adjustment to finalize the plot.Source: L Qualitative examples · p. 43, Figure 18 · p. 45
• Code Llama demonstrates domain-specific sox use by learning its options and applying them in context across simple tasks and combinations of tasks.The examples include prompts and explanations for an audio-processing command-line tool.Source: L Qualitative examples · p. 43, Figure 19 · p. 46