PaperlayerView exact revision on arXiv

Source-linked AI summary

Tulu 3: Pushing Frontiers in Open Language Model Post-Training

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, Hannaneh Hajishirzi

arXiv:2411.15124v5cs.CL

TL;DR

Open post-training recipes have lagged behind proprietary methods despite the importance of post-training data and recipes. Tülu 3 responds with fully open models and a multi-stage recipe, achieving stronger comparisons than same-size open-weight models and matching some closed models. The release is intended to support reproduction and further open post-training research, while its current data is limited for long-context, multi-turn, and tool-use settings.

  • Problem

    Open post-training methods and their underlying data and recipes have been less transparent and often less capable than proprietary approaches.

  • Method

    Tülu 3 combines open data, evaluation, code, infrastructure, and recipes in a multi-stage pipeline using supervised finetuning, preference tuning, and reinforcement learning with verifiable rewards.

  • Results

    Tülu 3 outperforms same-size state-of-the-art open-weight models, and its 70B model matches closed offerings such as Claude 3.5 Haiku and GPT-4o mini.

  • Takeaways & Limitations

    The openly released models and resources provide a basis for reproducing and extending post-training research on open base models.

  • Takeaways & Limitations

    Tülu 3’s collected data is relatively short, lacks long multi-turn data, and does not evaluate models in tool-use or agent frameworks.

Abstract

from arXiv · show

Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and the portion with the least transparency. To bridge this gap, we introduce Tulu 3, a family of fully-open state-of-the-art post-trained models, alongside its data, code, and training recipes, serving as a comprehensive guide for modern post-training techniques. Tulu 3, which builds on Llama 3.1 base models, achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and Claude 3.5-Haiku. The training algorithms for our models include supervised finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR). With Tulu 3, we introduce a multi-task evaluation scheme for post-training recipes with development and unseen evaluations, standard benchmark implementations, and substantial decontamination of existing open datasets on said benchmarks. We conclude with analysis and discussion of training methods that did not reliably improve performance. In addition to the Tulu 3 model weights and demo, we release the complete recipe -- including datasets for diverse core skills, a robust toolkit for data curation and evaluation, the training code and infrastructure, and, most importantly, a detailed report for reproducing and further adapting the Tulu 3 approach to more domains.

1 Introduction

Tülu 3 addresses the gap between increasingly important proprietary post-training recipes and less transparent open alternatives by releasing an open, multi-stage post-training framework. Its models and accompanying resources target diverse skills and achieve strong performance against open-weight and closed-model comparators.

  • Motivation: Open post-training recipes have lagged behind proprietary approaches, while earlier open alternatives often used simpler, cheaper pipelines and became outdated on many metrics.Post-training includes instruction tuning, reinforcement learning from human feedback, and other finetuning techniques.
  • Contributions: Tülu 3 releases open models trained from Llama 3.1 bases together with data, evaluation, training code, infrastructure, and development recipes.The release includes intermediate checkpoints and evaluation code.
  • Evaluation and data: The recipe identifies core skills, evaluates development and unseen tasks, and uses those evaluations to guide model improvement and data curation.The process includes aggressive decontamination of training data against the evaluation suite.
  • Training recipe: The multi-stage pipeline combines supervised finetuning, preference tuning, and reinforcement learning with verifiable rewards, with experimentation used to refine data mixes, methods, and parameters.RLVR rewards completions only when they are verified correct and targets skills such as mathematics and precise instruction following.
  • Results: Tülu 3 models outperform same-size state-of-the-art open-weight models, while the 70B model matches closed offerings such as Claude 3.5 Haiku and GPT-4o mini.The reported comparisons are based on the authors’ evaluation suite.
  • Contributions: The project contributes evaluation tooling, scaled synthetic instruction data, on-policy preference data, verifiable-reward reinforcement learning, and infrastructure for implementing large-model post-training.These components are presented as key contributions of the Tülu 3 development effort.

2 Tülu 3 Overview

Tülu 3 presents an open, multi-stage post-training framework designed to narrow the gap between proprietary and open recipes. It combines curated data, SFT, length-normalized DPO, RLVR, rigorous evaluation, decontamination, and released implementation artifacts.

  • Open post-training recipes have lagged behind proprietary methods despite post-training becoming crucial for frontier language models.
  • Tülu 3 releases an open suite spanning data, recipes, findings, model checkpoints, training code, evaluation code, infrastructure, and tooling.
  • Evaluation and Results: Tülu 3 Eval provides reproducible development and held-out evaluations, recommended settings, and tools for decontaminating training data against benchmarks.
  • Tülu 3 Recipe: The recipe uses a four-stage pipeline that progressively targets different skills and training data formats.
  • Tülu 3 Recipe: RLVR trains on tasks with verifiable outcomes and improves GSM8K, MATH, and IFEval performance.
  • Tülu 3 Recipe: Length-normalized DPO improved preference-tuning performance and was used instead of more costly RL-based methods such as PPO.
  • Evaluation and Results: Tülu 3 outperforms comparable open-weight models across its size categories and matches or rivals selected closed models at 70B and 405B.

3 Tülu 3 Data

Tülu 3 Data combines publicly sourced and synthetic prompts selected for diversity, provenance, licensing, and target skills. The data pipeline uses persona-driven generation and decontamination to support broad, skill-specific post-training.

  • Tülu 3 curates millions of prompts from public datasets and synthetic generation as the starting point for all post-training stages.
  • Dataset selection emphasizes diversity, provenance, licensing, and coverage of target capabilities such as reasoning, coding, and precise instruction following.
  • Synthetic Data: Persona-driven synthesis conditions generation on approximately 250K personas to create prompts targeting precise instruction following, math, and coding.
  • Synthetic Data: Synthetic precise-instruction-following data covers 25 constraint types from the IFEval benchmark using manually authored examples to guide generation.
  • Synthetic Data: The dataset includes math, coding, noncompliance, and safety prompts covering both benign and harmful scenarios.
  • Decontamination: Tülu 3 removes training instances that overlap with evaluation sets to reduce test-set contamination, using n-gram matching as the most useful approach in experiments.

4 Supervised Finetuning

Tülu 3 develops supervised finetuning around balanced mixtures of diverse, skill-specific data, using evaluations and ablations to refine the recipe. The resulting SFT models improve over Tülu 2 and benefit from targeted mathematics, Persona, and real-world chat data.

  • SFT data mixture: Tülu 3 balances diverse core skills through SFT mixture ablations and model-merging experiments.The procedure targets lagging skills while monitoring effects on other evaluations.
  • SFT data mixture: Filtering existing responses and creating new responses form the two main sources of the SFT mixture.Existing human or frontier-model responses are retained selectively, while missing or weaker responses are replaced or supplemented.
  • SFT results: Tülu 3’s final SFT mixtures show substantial improvements over Tülu 2 at both 8B and 70B, and outperform other competitive 8B SFT models on average.The comparison covers SFT-only models trained on Llama 3 8B or 70B.
  • Key data experiments: Removing WildChat causes small but noticeable degradation on most skills, while removing safety data leaves most skills roughly unchanged except the safety average.Contrastive prompts help prevent over-refusal of safe prompts.
  • Key data experiments: Removing Persona datasets lowers HumanEval(+), GSM8K, MATH, and IFEval, while mathematics-specific data substantially improves GSM8K and MATH.These ablations support using diverse, skill-targeted datasets in the final mix.
  • Training scale: Increasing SFT data improves average performance and GSM8K, but TruthfulQA performance drops as the mixture grows.The authors stop increasing SFT data because other prompts are allocated for preference optimization.

5 Preference Finetuning

Tülu 3 evaluates preference finetuning across DPO, PPO, and varied synthetic and on-policy preference mixtures. The experiments favor carefully mixed, unique, and on-policy data, while DPO is more economical than PPO in the reported setup.

  • Algorithms: Tülu 3 explores DPO and its derivatives alongside PPO to improve performance across the entire evaluation suite.The preference-fineting experiments compare direct alignment and reinforcement-learning approaches.
  • Preference data: Prompt duplication does not necessarily improve downstream DPO performance: 383k and 64k preference datasets perform similarly on average.More duplicated prompts slightly degrade DROP, GSM8K, and AlpacaEval, favoring unique prompts and proper mixing.
  • Preference data: On-policy generations improve aggregated downstream DPO performance compared with a completely off-policy preference dataset.The comparison includes combinations of on-policy and off-policy data.
  • Preference data: Tülu 3’s best preference mix significantly surpasses DPO training on UltraFeedback, with larger gains for 70B than 8B: +3.3 versus +1.8.The authors hypothesize that UltraFeedback completions are mainly from weaker models.
  • Data ablations: Only Persona IF improves both average evaluation and targeted IFEval performance; Persona Math and Persona Code do not improve their targeted evaluations and slightly harm average scores.The final mix therefore includes Persona IF but not the other two Persona preference datasets.
  • Data ablations: IF-persona preferences significantly improve IFEval while minimally harming average performance, whereas the verified subset improves IFEval by 1 point and slightly lowers average performance.Combining both gives the best IFEval performance but a slightly lower average, so the final 8B mix uses unverified IF-augmented data with Persona IF.
  • DPO versus PPO: PPO reaches comparable but slightly lower average scores than DPO in this non-tuned setup, while requiring roughly 28 hours versus DPO’s 4 hours.PPO uses two nodes and DPO uses one node in the reported runtime comparison.

6 Reinforcement Learning with Verifiable Rewards

RLVR replaces a learned reward model with deterministic verification for tasks whose outcomes can be checked, then optimizes the resulting rewards with PPO. It improves targeted math and instruction-following evaluations, with the strongest reported gains at 8B.

  • Method: RLVR trains language models on tasks with verifiable outcomes by replacing the reward model with a verification function.The method targets mathematics and verifiable instruction following and provides a constant reward for successful completions.
  • Method: RLVR samples completions, checks correctness with a deterministic verifier, assigns reward α when correct and 0 otherwise, and optimizes with PPO.The verifier operates on prompt-completion pairs and uses answer matching or constraint verification as a binary signal.
  • Training data: RLVR training data covers mathematics and exact instruction following through GSM8K, MATH, and IFEval prompts paired with binary verifiers.The resulting mixture contains roughly 30,000 prompts with ground-truth labels.
  • Ablation results: RLVR improves test performance in all three individual-task settings and consistently increases verifiable rewards on the training data.More KL divergence does not necessarily improve verifiable rewards on GSM8K and MATH.
  • Ablation results: Initializing the value function from a general reward model produces the highest GSM8K test score and higher average scores.The result indicates that the value function affects RLVR training outcomes.
  • Ablation results: Using only verifiable rewards outperforms adding reward-model scores, which introduces more noise, especially in average scores.The comparison favors the verifier-only reward signal.
  • Final results: At 8B, RLVR improves MATH, GSM8K, and IFEval, with some runs reaching GSM8K 89.4% and IFEval 84.8%; at 70B, gains are modest on IFEval and MATH and absent on GSM8K.The 70B GSM8K result is attributed to near saturation at 93.5%, while its KL divergence remains below 1.

7 Tülu 3 Evaluation Framework

Tülu 3 Eval combines reproducible tooling, development and unseen evaluations, and standardized prompting designed to measure core skills and generalization. The analyses show broad unseen-task generalization alongside important overfitting and transfer differences across skills.

  • Evaluation goals: The evaluation framework targets reproducibility, unseen-task generalization, and fair prompting across models.It provides an open toolkit, separate development and held-out suites, and recommended evaluation settings.
  • Development evaluations: “Summarize” zero-shot CoT systematically improves MMLU performance over standard 5-shot multiple choice across tested models.The prompt also maximizes the number of MMLU subjects benefiting from chain-of-thought prompting.
  • Unseen evaluations: Real-usage-aligned prompting often improves DeepMind Mathematics performance despite removing few-shot examples.The authors carry these principles into unseen-task formulation, while noting that development formulations were not updated based on this analysis.
  • Training-pipeline evaluation: Tülu 3’s final checkpoints achieve the best average performance on both development and unseen evaluations across the training pipeline.The comparison covers SFT, DPO, and final checkpoints at 8B and 70B across core skills.
  • Design decisions: Preference-data scaling trends generally transfer to unseen evaluations, but development appears to overfit MATH relative to DeepMind Mathematics.The authors attribute this partly to differing LaTeX formatting requirements and answer-extraction failures.
  • Generalization analysis: Tülu 3 generalizes well across most unseen evaluations, while IFEval scores diverge significantly from IFEval-OOD scores.The authors hypothesize that strong IFEval performance can reflect overfitting to its specific constraint set.
  • Cross-task variation: Knowledge-recall generalization varies by benchmark, and instruction-following performance shifts across evaluation categories.Tülu 3 70B outperforms Llama 3.1 70B Instruct on 5 of 11 HREF subtasks.

8 Discussions

The discussion examines scaling the recipe to 405B, methods excluded from the final recipe, and skills deferred to future work. Results are competitive at large scale, but compute constraints and coverage gaps limit the scope of the conclusions.

  • Scaling to 405B: Training Tülu 3 405B required 32 nodes and 256 GPUs, with synchronization issues, hardware failures, and semi-frequent restarts.These infrastructure challenges were especially pronounced during reinforcement learning.
  • Scaling to 405B: Hyperparameter tuning at 405B was limited by computational costs, and asynchronous RL constraints restricted training to 75 steps.The final run’s reward, KL divergence, and response length are reported in Figure 25.
  • Scaling to 405B: The 405B RLVR stage used only MATH training data because GSM8K was saturated after SFT and DPO, while IFEval provided little initial benefit.Even 25 RLVR steps improved MATH performance by over 5 points.
  • Scaling to 405B: Tülu 3 405B achieves competitive or superior performance to DeepSeek-V3 and GPT-4o while surpassing prior same-size open-weight models on many benchmarks.The 405B model generally improves over Tülu 3 70B.
  • Methods not retained: Online DPO produced little or no GSM8K improvement and degraded MATH performance in the reported mathematical-reasoning experiment.The experiment used 200K episodes with reward-model feedback.
  • Methods not retained: Rejection sampling delivered minimal gains relative to its compute cost, and stronger judges were important for selecting responses.Including the original response among candidates performed better than selecting only from newly generated responses.
  • Future work: Tülu 3 leaves long-context and multi-turn capability for future work because its data are relatively short and average 2.4 turns.Most samples are under 2,048 tokens.
  • Future work: The work focuses on English data and evaluations, leaving multilingual post-training for future investigation.The authors note that multilingual post-training may require cross-lingual alignment and careful data balancing.

9 Related Work

Open post-training recipes initially lagged behind closed-lab developments, motivating broader, more complex recipes that combine multiple data sources and training methods. Tülu 3 advances this direction with a large pipeline and online RL using verifiable rewards.

  • Evolution of post-training: Post-training evolved from instruction tuning toward preference-based and multi-round training with human and synthetic preference data.Recent methods also use varied training algorithms and multiple training rounds.
  • Open post-training gap: Open post-training recipes lagged behind closed-lab developments, while early open efforts focused mainly on instruction tuning.Combining datasets could yield strong performance, but open recipes remained behind more advanced closed approaches.
  • Tülu 3 approach: Tülu 3 builds a larger, more complex open recipe intended to match or beat strong closed recipes while releasing its artifacts for study and use.The pipeline includes almost a million instruction-tuning samples, hundreds of thousands of preference pairs, and online RL.
  • Training on verifiable rewards: RLVR uses an existing PPO framework online with binary rewards, distinguishing it from related methods that train on reasoning traces or code feedback.The approach is positioned alongside STaR, TRICE, VinePPO, and code-feedback methods.
  • Training on verifiable rewards: The related work frames RL-based training as a growing approach for improving language-model reasoning.The cited methods use ground-truth answers, reasoning traces, binary correctness rewards, or code feedback as training signals.

10 Conclusion

Tülu 3 presents a fully open, multi-stage post-training framework with released models, data, code, evaluation, and recipes. Its models outperform comparable open-weight systems and match closed offerings at 70B, while the released resources support further multi-objective research.

  • Open framework: Tülu 3 is a family of fully open state-of-the-art language models built around open data, evaluation, code, and development recipes.The framework includes Tülu 3 Data, Tülu 3 Eval, Tülu 3 Code, and Tülu 3 Recipe.
  • Open framework: The release includes Llama 3.1-based final models, intermediate checkpoints, training data, training code, and evaluation code.These artifacts constitute complete pipelines for finetuning open base models.
  • Implications: The released resources support further post-training research using multi-objective and multi-stage training frameworks.The authors state that others can finetune open base models for high performance across diverse tasks.

B.2 Contamination in Public Datasets

The authors identify substantial evaluation contamination in several public datasets and provide decontaminated training versions for some of them. They recommend checking and removing benchmark overlap before using such datasets for training.

  • Contamination findings: Public datasets containing realistic API-model uses, including ShareGPT, WildChat, and LMSys Chat, are likely to overlap with existing benchmark test sets.The contamination analysis reports significant overlap for some publicly available datasets.
  • Contamination findings: Datasets with more than 5% evaluation overlap were identified using heuristics described in Section 3.2.The overlap percentage measures evaluation instances matching dataset instances.
  • Mitigation: The release includes decontaminated versions of the first five contaminated datasets, while the last four were excluded from training.The authors recommend decontamination before using overlapping datasets as training data.

C Supervised Finetuning Data Details

The supervised-finetuning data details reference prompts used to create persona datasets for instruction-following, mathematics, and code.

  • Persona data: Persona-data construction used dedicated prompts for instruction-following, mathematics, and code datasets.The prompts are provided in Figures 30–36.

D Preference Tuning Data Details

The report provides the system prompt and response-rating templates used for LLM-as-a-judge evaluation, including variants for particular aspects.

  • The LLM-as-a-judge system prompt is provided in Figure 37.
  • The template for rating model responses is given in Figure 38.
  • Additional response-rating variants target particular aspects and appear in Figures 39–42.

E.1 Testing Generalization to Target Evaluations

Tülu 3 evaluates RLVR training against target evaluations at intermediate checkpoints to assess generalization and detect overfitting. The report also documents the training setup and notes that the best configuration remains unresolved.

  • RLVR runs are evaluated on target evaluations throughout training to verify that training behaves as expected.
  • Intermediate-checkpoint evaluation tests whether RLVR generalizes beyond similar but non-identical training data and detects possible overfitting.
  • Training runs on GSM8K, MATH, and IFEval use both DPO-trained and SFT-trained initializations for the value functions.
  • The reported configurations show generalization, but the exact best configuration remains to be found.
  • The report details hyperparameters for the Tülu 3 8B reward model used to initialize the RLVR value model.

F.1 Detailed Safety Results

The safety and evaluation analysis covers refusal behavior, instruction-tuned evaluation practices, and chain-of-thought prompting. Implicit CoT improves Tülu 3 across more MMLU subjects than explicit CoT, although prompt wording affects results.

  • Safety results: Safety results are broken down by risk category for Tülu 3 and peer 8B models on HarmBench and WildGuardTest.
  • Instruction-tuned evaluation: Instruction-tuned models are compared on MATH, DROP, and GSM8K using base-model evaluation adaptations versus practices designed around real usage.
  • Prompting strategy: Zero-shot CoT is preferred because an additional in-context example does not consistently improve scores for instruction-tuned models.
  • Prompting strategy: Implicit CoT improves an average of 78% of MMLU subjects for Tülu 3, while explicit CoT benefits at most 53%.
  • Prompting caveat: The tested MMLU CoT formulations preserve the instructions’ overall meaning, yet their observed performance varies.
  • RLVR behavior: RLVR can overoptimize its verifiable reward function and produce no meaningful content in Prompts with Constraints examples.

F.3 IFEval Out-of-Distribution Constraints

The IFEval out-of-distribution evaluation uses unseen WildChat prompts augmented with a broad taxonomy of constraints. The taxonomy spans six categories and includes diverse counting, formatting, ratio, sentence, word, and custom requirements.

  • Taxonomy: The IFEval OOD taxonomy contains 52 human-written constraint types divided into count, format, ratio, sentence, words, and custom categories.
  • Taxonomy: Each category contains 3–12 representative constraints, and non-custom types have at least 5 final prompts sourced from unseen WildChat data.
  • Constraint categories: Count constraints test properties such as unique-word counts, word-count ranges, French-word placement, conjunctions, countries, numbers, names, pronouns, and punctuation.
  • Constraint categories: Format and ratio constraints require structures such as camelCase code, emojis, indentation, lists, nested punctuation, overlaps, sentence-type ratios, and balanced sentence composition.
  • Constraint categories: Sentence and word constraints regulate alliteration, sentence-length increments, keyword placement, word initials, syllables, palindromes, repetition, and vowel usage.
  • Constraint categories: Custom constraints include reversed-letter responses and comma-delimited city data with seven rows.

F.4 Subtask-level breakdown of HREF results

Table 48 compares Tülu 3 with Hermes 3 Llama 3.1 and Llama 3.1 Instruct models at 8B and 70B scales on HREF. It reports subtask-level win rates against Llama 3.1 405B Instruct under multiple evaluation setups.

  • Table 48 compares Tülu 3, Hermes 3 Llama 3.1, and Llama 3.1 Instruct models at 8B and 70B scales on HREF.
  • The table breaks down HREF performance by instruction-following subtask using win rates against Llama 3.1 405B Instruct.
  • Evaluation setups are labeled L, LH, or E, distinguishing LM-as-a-judge, reference-augmented judging, and embedding-based similarity.
Loading 2411.15124v5…