Tulu 3: Pushing Frontiers in Open Language Model Post-Training

Contributions

• Tülu 3 closes the open–closed post-training gap by releasing models alongside data, training recipes, code, infrastructure, and an evaluation framework.The framework comprises Tülu 3 Data, Tülu 3 Eval, and Tülu 3 Recipe, integrating proprietary-method details, novel techniques, and established research.Source: §1 · pp. 5–6

Data and infrastructure

• Tülu 3 improves post-training through permissively licensed skill-specific synthetic data, on-policy preference generations, aggressive decontamination, and principled optimization of data mixes and parameters.Its contributions also include extensive evaluation and recipe-design tooling, scaled synthetic instruction datasets, and advanced infrastructure for large models.Source: §1 · pp. 5–6

Training recipe

• The recipe uses staged supervised finetuning, preference tuning, and RLVR, with each stage emphasizing prompt completions, preferences, or verifiable rewards.RLVR provides rewards only for verified correctness and is tailored to skills such as mathematics and precise instruction following.Source: §1 · pp. 5–6

Results

• Tülu 3 outperforms same-size state-of-the-art open-weight models and matches Claude 3.5 Haiku and GPT-4o mini at 70B, while competing with DeepSeek v3 and GPT-4o at 405B.The evaluation overview covers 8B and 70B models and reports comparisons with open-weight and closed systems.Source: Table 2 · p. 6, §1 · p. 6

Evaluation

• Tülu 3 Eval measures core skills on development and unseen tasks while supporting standardized evaluation and training-data decontamination.The targeted skills include reasoning, math, coding, safety, precise instruction following, and knowledge recall.Source: §1 · p. 5, Table 3 · p. 7

2 Tülu 3 Overview

• Tülu 3 releases model checkpoints, SFT and DPO datasets, training recipes, evaluation tools, and infrastructure so the community can reproduce and extend open post-training.The authors frame the release as a comprehensive suite of data, code, findings, and tools rather than a standalone model artifact.Source: §2 · p. 7, §2.3 · pp. 9–10, §2.4 · p. 11

2.1 Tülu 3 Data

• Tülu 3 targets knowledge recall, reasoning, mathematics, coding, instruction following, general chat, and safety with curated public and synthetic data across training stages.The data collection uses multiple formats, provenance analysis, decontamination, and targeted prompts for core skills, while real-world queries such as WildChat improve general chat capabilities.Source: §2.1 · pp. 7–8, §2.3 · p. 9

2.2 Tülu 3 Evaluation

• Tülu 3 Eval provides a unified toolkit, standardized development and held-out evaluations, recommended settings, and benchmark decontamination for reproducible model development.The unseen set was withheld during model development, enabling assessment of overfitting to particular evaluations, data mixtures, algorithms, and hyperparameters.Source: §2.2 · p. 8

2.3 Tülu 3 Recipe

• The four-stage recipe combines data curation, supervised finetuning, length-normalized DPO, and RLVR, with experiments guiding data mixtures, methods, and parameters.RLVR supplies rewards only for verified outcomes and improves GSM8K, MATH, and IFEval; the pipeline also includes on-policy preference data, scalable asynchronous RL, and 354,192 preference instances.Source: §2.3 · pp. 8–10

2.4 Evaluation and Results

• Tülu 3 outperforms all other open-weight models in its size categories, surpasses GPT-3.5-Turbo-0125 and GPT-4o-mini at 70B, and approaches Claude 3.5 Haiku.At 70B and 8B, Tülu 3 also surpasses the listed Llama, Qwen, Nous Hermes, Gemma, Ministral, and Nemotron instruct or finetuned models.Source: Table 4 · p. 8, Table 5 · p. 9, Table 6 · p. 10

3 Tülu 3 Data

• Millions of curated prompts form the foundation for all Tülu 3 post-training stages, with later training data selected from this collection.The prompt collection supports supervised finetuning and preference tuning, while curation and decontamination aim to prevent evaluation leakage.Source: §3 · p. 11

3.1 Prompt Curation

• Tülu 3 curates diverse, high-quality prompts from publicly available datasets with clear provenance and synthesizes data to fill skill-specific gaps.Selection emphasizes diversity, core capabilities, and licensing; included sources target reasoning, coding, instruction following, multilinguality, science, and table processing.Source: §3.1 · p. 11, §3.1.1 · pp. 11–13

3.1.1 Sourcing from Public Datasets

• The curation process excludes datasets with questionable legal provenance and retains only datasets with clear licenses after manually tracking source subsets.ShareGPT is excluded, WildChat is used instead, and affected UltraFeedback and Helpsteer2 data are removed because of ShareGPT usage.Source: §3.1.1 · p. 13

3.1.2 Synthesizing for Target Skills

• Persona-driven synthesis conditions generation on approximately 250K personas to create prompts for precise instruction following, mathematics, coding, noncompliance, and safety.The method addresses mode collapse while targeting diverse perspectives and use cases; If-Persona-Sft contains 29,980 verifiable instruction-response pairs, and IF-augmented prompts are reserved for DPO and RLVR.Source: §3.1.2 · pp. 13–14

3.2 Prompt Decontamination

• Tülu 3 detects contamination with 8-gram matching over prompts or user turns, marking significant overlap when more than 50% of test tokens match one training instance.Training sets contaminated with unseen evaluations are removed, while development-set contamination is handled by dataset- or instance-level removal; released decontaminated versions are listed in Table 8.Source: §3.2 · pp. 14–15

4.1.1 From Prompts to SFT Data

• Tülu 3 builds SFT data by retaining high-quality human or frontier-model responses, filtering undesirable examples, generating missing or weak responses with GPT-4o, and hand-writing hardcoded prompts.The procedure combines filtered existing responses with newly created responses.Source: §4.1.1 · p. 15

4.1.2 The Tülu 3 SFT Mix

• The final mixture combines skill-specific datasets and iteratively adds or removes data, decontaminates evaluations, and downsamples large datasets to improve lagging capabilities.Skill-specific mixtures were first optimized independently to approximate evaluation-specific upper bounds before constructing the preview and final mixtures.Source: §4.1.2 · p. 16

4.2 Key Data Experiments

• Removing WildChat causes small degradation across most skills, while removing safety data largely leaves general skills unchanged and contrastive prompts reduce over-refusal of safe prompts.The reported effects identify diverse real-world chat data as broadly useful and safety data as largely orthogonal to general performance.Source: §4.2 · p. 16, Table 10 · p. 17

4.1.2 The Tülu 3 SFT Mix

• The final Tülu 3 SFT mix substantially improves over Tülu 2 at both 8B and 70B sizes and achieves a higher average score than comparable SFT mixes.Table 9 compares the final models against SFT-only baselines trained on Llama 3.0 or 3.1.Source: §4.1.2 · p. 16, Table 9 · p. 17

4.2 Key Data Experiments

• Removing Persona data lowers HumanEval(+), GSM8K, MATH, and IFEval, while mathematics-specific data substantially improves GSM8K and MATH.These ablations support constructing diverse datasets targeted at specific skills.Source: §4.2 · p. 17, Table 10 · p. 17

4.3.1 Key Training Experiments

• Larger base models and skill-specific pretraining data substantially improve GSM8K and MATH, while SFT performance varies by random seed and the best model soup does not always beat the best single run.The authors therefore use the best single SFT run for each model size as the final model.Source: §4.3.1 · p. 18, Table 12 · p. 18, Table 14 · p. 19

4.3.2 Batch Aggregation

• Mean-loss aggregation can change effective sample weighting under gradient accumulation or distributed training, so Tülu 3 uses sum loss; 5e-6 and two epochs performed best in validation.Sum loss weights tokens equally, and longer training did not yield further improvements.Source: §4.3.2 · pp. 19–20, Figure 5 · p. 20

5 Preference Finetuning

• Tülu 3 explores DPO, its variants, PPO, and synthetic preference data to improve performance across its evaluation suite.The preference-finetuning study covers training algorithms, on-policy and off-policy data, and skill-specific preference construction.Source: §5 · p. 20

5.1.1 Setup

• Preference data pairs prompts with chosen and rejected responses, while reward modeling learns their preference difference to train policies.The reward difference represents the log-likelihood that the chosen response is preferred over the rejected response.Source: §5.1.1 · pp. 20–21

5.1.2 Policy Optimization

• Tülu 3 uses length-normalized DPO because it avoids reward-model training and policy sampling while mitigating preference length bias more simply and quickly than PPO.Length-normalized DPO was found to work best among the explored DPO variants and was used for preference-mixture tuning and generation methods.Source: §5.1.2 · p. 21

5.2.1 From Prompts to Preference Data

• The synthetic pipeline selects prompts, randomly samples four models to generate responses, includes Tülu SFT-model completions, and uses GPT-4o-2024-0806 to rate responses from 1 to 5 across four aspects.Preference annotation uses an LLM-as-a-judge to create (preferred, rejected) pairs.Source: §5.2.1 · pp. 21–22

5.2.2 The Tülu 3 Preference Mix

• The final preference mixes for the 8B and 70B models maximize average development-evaluation performance while excelling at targeted skills, using prompt sources including SFT data, WildChat, and Persona.The preference base contains 96,911 off-policy and 19,444 on-policy instances; additional prompt sources were ablated for effects on instruction following, math, and general chat performance.Source: §5.2.2 · p. 22

5.3 Key Findings of Data Ablations

• Increasing unique prompts improves downstream DPO performance, whereas duplicating prompts produces similar average performance and slight degradation on DROP, GSM8k, and AlpacaEval.The final mixes therefore prioritize collecting unique prompts and proper mixing over scaling through prompt duplication.Source: §5.3 · pp. 23–24, Figure 8 · p. 23, Figure 9 · p. 23
• Unused prompts slightly outperform reused SFT prompts, on-policy data improves aggregated DPO performance, and GPT-4o leads similar judges slightly on aggregate.GPT-4o-2024-0806 was selected for ease of use, lower request cost, and batch-inference speed.Source: §5.3 · pp. 24–25, Figure 10 · p. 25, Figure 11 · p. 25
• The best preference mix surpasses UltraFeedback, while Persona IF improves instruction following; math and code persona data do not improve targeted evaluations and can harm average performance.Regenerated preferences outperform original Helpsteer2, UltraFeedback, and MultiPref datasets, and combining IF-persona with IF-augmented data gives the best IFEval performance despite a slightly lower average.Source: §5.3 · pp. 25–26, Figure 12 · p. 26, Figure 14 · p. 27

5.4.1 Hyperparameter and Algorithm Design

• Length-normalized DPO was the only tested algorithm to outperform the base checkpoint overall, and it became the final preference-tuning choice.The final DPO training used the length-normalized variant proposed in Meng et al. (2024).Source: §5.4.1 · p. 27, Table 18 · p. 28, Table 20 · p. 28
• For final 70B DPO models, a learning rate of 2.0 × 10-7 was selected after higher tested rates outperformed a lower rate depending on the preference-data mix.The 70B search compared learning rates across two preference mixes, with 2.0 × 10-7 or 5.0 × 10-7 performing better than a lower learning rate.Source: §5.4.1 · p. 28, Table 19 · p. 28
• PPO achieved comparable but slightly lower average scores than DPO in the controlled non-tuned setup, while requiring roughly 28 hours versus DPO’s 4 hours.Because PPO iteration was more expensive and reward-model evaluation was subtle, the authors used DPO for most preference-tuning experiments and reserved PPO primarily for RLVR.Source: §5.4.1 · p. 29, Figure 16 · p. 30

5.4.2 Infrastructure for Scaling DPO

• Caching reference-policy log probabilities and separately forwarding chosen and rejected completions reduced DPO’s GPU footprint by avoiding the reference model and doubled forward-pass batch allocation.The techniques were designed to reduce GPU memory requirements for 70B DPO training.Source: §5.4.2 · pp. 29–30
• On Llama 3.1, the two memory optimizations produced near-identical training losses and used less GPU memory on an 8xH100 setup.Figure 17 reports the peak GPU-memory reduction from caching reference-policy log probabilities and separate chosen/rejected forward passes.Source: §5.4.2 · p. 30, Figure 17 · p. 30

6 Reinforcement Learning with Verifiable Rewards

• RLVR replaces the learned reward model with a deterministic verifier that gives reward α for verifiably correct completions and 0 otherwise, optimized with PPO.The method is applied after preference finetuning, with α = 10 based on pilot experiments.Source: §6 · pp. 30–31, Figure 18 · p. 31

6.1 RLVR Data

• RLVR training uses roughly 30,000 labeled prompts spanning mathematics and exact instruction following across GSM8K, MATH, and IFEval.Verification extracts answers for GSM8K and MATH, while IFEval uses constraint-specific verification functions.Source: §6.1 · pp. 31–32

6.2 RLVR Recipe and Analyses

• The PPO recipe uses implementation safeguards including deterministic no-dropout computation, checkpoint selection on development evaluations, non-EOS penalties, and advantage normalization.The experiments sweep KL coefficients across tasks and model initializations, including weaker SFT starting points.Source: §6.2 · p. 32

6.2.1 Key Findings

• RLVR improved targeted test performance and verifiable training rewards in GSM8K, MATH, and constraint-following settings, but greater KL did not consistently improve rewards.Initializing the value function from a general reward model produced the highest GSM8K test and average scores, whereas adding reward-model scores introduced noise.Source: §6.2.1 · pp. 33–34

6.3 RLVR Infrastructure

• Distributed PPO scales RLVR to 405B parameters by using dedicated inference GPUs, Zero Stage 3, PagedAttention, and asynchronous training with second-latest inference data.Asynchronous execution reduces GPU idle time but can introduce stale data and reproducibility concerns.Source: §6.3 · pp. 34–35

6.4 Final Experimental Results

• RLVR improved all three evaluated benchmarks at 8B, while 70B gains were modest on IFEval and MATH and absent on GSM8K.Some 8B runs reached 89.4% on GSM8K and 84.8% on IFEval; the 70B GSM8K baseline was already near saturation at 93.5%.Source: §6.4 · p. 36, Table 23 · p. 36

7 Tülu 3 Evaluation Framework

• The framework targets reproducible evaluations, unseen-task generalization, and prompting that is fair across diverse models.It separates development evaluations from held-out unseen evaluations and recommends standardized settings summarized in the Tülu 3 Evaluation Regime.Source: §7 · pp. 36–37

7.1 Open Language Model Evaluation System (OLMES)

• OLMES releases the evaluation code with broad model and task support, flexible configurations, exact task formulations, and instance-level prediction outputs for reproducible analysis.The system is designed to standardize and reproduce the evaluations used in this work.Source: §7.1 · pp. 37–38

7.2 Tülu 3 Evaluation Suite - Development

• The development suite adapts prompts, sampling, and answer extraction to benchmark characteristics, including zero-shot “summarize” CoT for MMLU and flexible extraction for MATH.For MMLU, the “summarize” prompt systematically improves performance over standard 5-shot multiple choice across tested models, while MATH’s flex extraction can improve scores by up to 10 points.Source: §7.2 · p. 38

7.2.1 Safety Evaluation

• The safety suite measures refusal, benign compliance, and jailbreak robustness across multiple benchmarks, using WildGuard-based accuracy or response-to-attack metrics and macro-averaging scores.XSTest and WildJailbreak additionally test compliance with benign requests, while harmful and adversarial prompts assess safeguards against unsafe behavior.Source: §7.2.1 · pp. 39–40

7.3 Tülu 3 Evaluation Suite - Unseen

• The unseen suite independently designs realistic, concise, and format-tolerant tasks, and these human-aligned formulations generally preserve or improve exploratory-model performance.For MMLU-Pro, the shorter zero-shot formulation is generally as effective as traditional 5-shot CoT, while DeepMind Mathematics often benefits from chat-style evaluation over base-adapted few-shot prompts.Source: §7.3 · pp. 40–41

7.3.1 New Evaluation: IFEval-OOD

• IFEval-OOD introduces 52 verifiable constraints across six categories combined with quality-checked unseen WildChat prompts to distinguish general constraint following from overfitting to IFEval.The dataset emphasizes broad coverage, including manually written custom tasks such as CSV generation.Source: §7.3.1 · pp. 42–43

7.3.2 New Evaluation: HREF

• HREF provides automatic instructability evaluation across 11 instruction-following tasks using human-written references and win rate against a fixed baseline, while exploring alternative LM judges and computation methods.The task categories include brainstorming, question answering, extraction, generation, rewriting, summarization, classification, numerical reasoning, synthesis, and fact checking.Source: §7.3.2 · p. 43
• The HREF procedure uses Llama 3.1 70B Instruct as judge, embedding similarity for Open QA and Fact Checking, and human references for seven other subtasks.The composite procedure agrees with human judgments 69.4% of the time, versus 67% inter-human agreement.Source: §7.3.2 · pp. 43–44

7.4 Evaluating the Development Process Using the Unseen Suite

• The unseen suite measures whether development decisions overfit by evaluating corresponding core skills on tasks withheld from development.The framework evaluates design checkpoints on unseen tasks and compares final checkpoints with public models.Source: §7.4 · p. 44

7.4.1 Evaluating the design decisions

• The final Tülu 3 checkpoints achieve the best average performance on both development and unseen evaluations, with later training improving harder unseen Reasoning and Coding tasks.Table 31 compares SFT, DPO, and final 8B/70B checkpoints across one development and one unseen evaluation per core skill.Source: §7.4.1 · p. 44, Table 31 · p. 44
• SFT data choices generalize on average, but overfit development evaluations in Precise Instruction Following and, to some extent, Knowledge Recall and Reasoning.DPO scaling trends also generalize on average, whereas Math results expose overfitting caused partly by LaTeX-format differences between MATH and DeepMind Math.Source: §7.4.1 · pp. 44–45, Table 32 · p. 45, Figure 24 · p. 45

7.4.2 Comparison with public models

• Across unseen evaluations, Tülu 3 is generally comparable to Hermes 3 and Llama 3.1, but models overfit IFEval and show recipe-dependent knowledge-recall generalization.Tülu 3 70B outperforms Llama 3.1 70B Instruct on 5 of 11 HREF subtasks, while relative instruction-following performance varies between AlpacaEval and HREF.Source: §7.4.2 · pp. 45–46, Table 33 · p. 46

8.1 Scaling Tülu 3 Recipe to Llama 3.1 405B

• Training Tülu 3 405B required 32 nodes and 256 GPUs, with synchronization failures, hardware faults, frequent restarts, and limited hyperparameter tuning complicating scaling.RLVR used 16 GPUs for vLLM inference and 240 for training; inference, weight transfer, and training typically took approximately 550, 25, and 1,500 seconds, respectively.Source: §8.1 · p. 46
• After only 25 RLVR steps, MATH performance improved by over 5 points and continued increasing with additional training, although the 405B run stopped after 75 steps.The 405B recipe used only the MATH training set for RLVR after removing GSM8K and IFEval data that offered little benefit in initial runs.Source: §8.1 · p. 47, Figure 25 · p. 48
• Tülu 3 405B achieves competitive or superior performance to DeepSeek-V3 and GPT-4o while surpassing prior open-weight 405B post-trained models on many standard benchmarks.The model generally improves over Tülu 3 70B, but RLVR ended early because of compute constraints, and MATH performance had not saturated.Source: §8.1 · p. 47

8.2 Insights from the Unfruitful

• Online DPO produced no or little GSM8K improvement and degraded MATH performance after 200K math episodes, limiting its value for general and targeted capabilities.The experiments varied sampling temperatures and KL penalties, and used reward-model feedback to construct online preference pairs.Source: §8.2 · p. 48
• Rejection sampling delivered minimal gains relative to its compute cost, and strong judges were necessary; including the original response improved selection over judging only new generations.Publicly available models sometimes struggled to choose the best response among candidates.Source: §8.2 · pp. 48–49

8.3 Future Work

• Future Tülu iterations should address long-context and multi-turn interactions, multilinguality, and tool use or agent frameworks.The current data averages 2.4 turns, is mostly under 2,048 tokens, and focuses on English; tool access may improve reasoning and mathematical skills.Source: §8.3 · p. 49

9.1 The Evolution of Post-training Recipes

• Post-training progressed from instruction tuning for zero-shot generalization to preference finetuning, direct preference training, and mixtures of human and synthetic data across multiple rounds.Early instruction-tuning datasets emphasized traditional NLP tasks, whereas later recipes incorporated preference data and varied algorithms.Source: §9.1 · pp. 49–50
• Open post-training recipes have lagged closed systems in data scale, training complexity, and performance, with many relying on limited data, few rounds, and DPO-based AI feedback.Open efforts progressed from instruction tuning to preference finetuning, but no open-recipe models in the cited LMSYS ChatBotArena top 50 had released their post-training data as of November 20, 2024.Source: §9.1 · p. 50
• Tulu 3 advances open post-training with nearly a million instruction-tuning samples, hundreds of thousands of preference pairs, online RL, and release of its code, models, and data.The recipe is described as significantly larger and more complex than prior open work and intended to match or beat strong closed recipes.Source: §9.1 · p. 50

9.2 Training on Verifiable Rewards

• RLVR differs from STaR, TRICE, Quiet-STaR, and VinePPO by using online PPO with binary rewards, extending beyond mathematics to precise instruction following.The work also ablates value-model initialization and combining a general reward model with verifiable rewards.Source: §9.2 · p. 50

10 Conclusion

• Tülu 3 introduces a fully open, state-of-the-art language-model family built around a modern post-training framework.Source: §10 · p. 50
• The release includes Tülu 3 Data, Tülu 3 Eval, Tülu 3 Code, Tülu 3 Recipe, final Llama 3.1-based models, intermediate checkpoints, training data, and evaluation code.Source: §10 · p. 50
• Tülu 3 aims to bridge open and closed post-training methods and establish a milestone for open post-training research.Source: §10 · p. 50
• The released resources support finetuning open base models for high performance across diverse tasks and advancing multi-objective, multi-stage training frameworks.Source: §10 · pp. 50–51

A Additional Hyperparameters

• Table 36 reports the hyperparameters used for reward model training.Source: A Additional Hyperparameters · p. 60

B.1 Extra Distribution Plots

• Figure 26 shows token-length distributions for the publicly available Tülu 2 SFT Mix and OpenHermes 2.5 datasets.Source: §B.1 · p. 60

B.2 Contamination in Public Datasets

• Datasets using realistic API-model conversations can overlap with benchmark test sets, so practitioners should decontaminate them before training.The contaminated sources include ShareGPT, WildChat, and LMSys Chat.Source: §B.2 · p. 60

B.3 Chat Template Implementation

• Figure 27 provides the implementation of the chat template used by the models.Source: §B.3 · p. 60

B.4 RLVR IFEval overoptimization

• Higher-KL RLVR IFEval runs produced some outputs judged to be overoptimized.The authors checked the outputs of their RLVR IFEval experiments for this behavior.Source: §B.4 · p. 60
• Figures 28 and 29 show overoptimized and non-overoptimized RLVR IFEval samples, respectively.Source: §B.4 · p. 60

B.2 Contamination in Public Datasets

• Table 37 identifies public datasets with significant contamination, defined as more than 5% evaluation overlap under the stated heuristics.The table reports overlap as the percentage of evaluation instances overlapping with dataset instances.Source: Table 37 · p. 61
• The training data included decontaminated versions of the first five contaminated datasets and excluded the last four.Source: Table 37 · p. 61

C Supervised Finetuning Data Details

• The supervised-finetuning data details include prompts for creating persona data.Source: §C.1 · p. 61

C.1 Prompts

• The prompts cover instruction-following data, identified as IF data.Source: §C.1 · p. 61
• The IF-data prompts are provided in Figures 30, 31, and 32.Source: §C.1 · p. 61
• The prompts also cover math data.Source: §C.1 · p. 61
• The math-data prompts are provided in Figures 33 and 34.Source: §C.1 · p. 61
• The prompt collection additionally covers code data, with examples provided in Figures 35 and 36.Source: §C.1 · p. 61

D Preference Tuning Data Details

• The preference-tuning data details provide the system prompt used for the LLM-as-a-judge in Figure 37.Source: D Preference Tuning Data Details · p. 61
• Figure 38 presents the template used to rate model responses.Source: D Preference Tuning Data Details · p. 61
• Additional response-rating variants target particular aspects of model outputs.Source: D Preference Tuning Data Details · p. 61
• These aspect-specific rating variants are provided in Figures 39, 40, 41, and 42.Source: D Preference Tuning Data Details · p. 61

E.1 Testing Generalization to Target Evaluations

• RLVR training runs are evaluated on target evaluations throughout the report to verify that training behaves as expected.The report identifies target-evaluation performance as an important metric for assessing RLVR training.Source: §E.1 · p. 61
• The Tülu 3 chat template is specified exactly as part of the evaluation and implementation details.Figure 27 presents the exact chat-template implementation used for Tülu 3 and refers to further discussion in Sec. 4.3.1.Source: Figure 27 · p. 62, §E.1 · p. 62
• Generalization appears across all tested configurations on GSM8K, MATH, and IFEval during intermediate-checkpoint evaluation.The runs use GSM8K, MATH, and IFEval prompts and initialize value functions from both DPO- and SFT-trained models.Source: §E.1 · p. 62
• The best RLVR training configuration remains unresolved despite observed generalization across configurations.The evaluation examines multiple intermediate checkpoints because over-fitting can occur when training data resembles, but does not exactly match, evaluation data.Source: §E.1 · p. 62

E.2 RM Training Hyperparameters

• The Tülu 3 8B reward model is used to initialize the value model for RLVR training.The report provides the reward-model training hyperparameters in Table 36.Source: §E.2 · p. 62
• Table 36 records the hyperparameters used to train the Tülu 3 8B reward model.These hyperparameters concern the reward model used for RLVR training initialization.Source: §E.2 · p. 62

F.1 Detailed Safety Results

• Safety results provide risk-category breakdowns of refusal rates for Tülu 3 and peer 8B models on HarmBench and WildGuardTest.The breakdowns are presented in Tables 39 and 40.Source: §F.1 · p. 62, Table 39 · p. 64, Table 40 · p. 64

Experimenting Principles for unseen suite on development tasks

• For unseen-suite development tasks, evaluation practices aligned with real usage favor natural instructions and simple zero-shot CoT over stitched few-shot examples in several settings.The passage specifically reports that zero-shot CoT is retained because an additional example does not always improve scores, and that a one-turn zero-shot CoT setup performs better for DROP than stitched in-context examples.Source: Experimenting Principles for unseen… · pp. 62–63

Prompts used for unseen evaluation tasks

• The unseen evaluation suite uses a 0-shot reasoning prompt for multiple-choice tasks including AGIEval English, MMLU-Pro, and GPQA, with a separate prompt for DeepMind Mathematics.The multiple-choice prompt requests concise reasoning and ends with a standardized answer-letter format.Source: Prompts used for unseen evaluation tasks · p. 63, MMLU Chain-of-Thought Prompting · p. 77, Figure 44 · p. 77

MMLU Chain-of-Thought Prompting

• MMLU CoT evaluation uses explicit and implicit reasoning formulations, answer-format instructions, heuristic extraction, and exact matching against gold answers.The implicit variant asks models to summarize reasoning before giving an answer prefixed by “Therefore, the answer is”.Source: MMLU Chain-of-Thought Prompting · p. 63
• Implicit CoT consistently improves Tülu 3 and most peer models over 5-shot no-CoT MMLU, with gains across both 8B and 70B models.The comparison is reported in Tables 45 and 46.Source: MMLU Chain-of-Thought Prompting · p. 64, Table 45 · p. 66, Table 46 · p. 69
• Implicit CoT improves an average of 78% of MMLU subjects for Tülu 3, compared with at most 53% for explicit CoT.It is also described as improving reasoning without penalizing subjects that do not require explicit step-by-step reasoning.Source: MMLU Chain-of-Thought Prompting · p. 64
• Small wording changes among CoT prompts produce variable performance, indicating that prompt wording requires careful attention in heterogeneous evaluations such as MMLU.The three formulations preserve the overall meaning of the instructions despite their differing results.Source: MMLU Chain-of-Thought Prompting · p. 64

Dataset taxonomy

• 52 human-written constraint types span six categories—count, format, ratio, sentence, words, and custom—with 3–12 representative constraints per category.All categories except custom include at least five final prompts sourced from unseen WildChat data.Source: §F.3 · p. 79

Count constraints

• Count constraints test quantities and lexical properties, including conjunctions, countries, numbers, unique words, word-count ranges, and French words at fixed intervals.Examples require exactly {N} numbers, between {min_n} and {max_n} words, or every {N}th word in French.Source: §F.3 · p. 79

Format and ratio constraints

• Format and ratio constraints regulate output structure and textual relationships, such as camelCase code, emojis, indentation, nested punctuation, and trigram overlap.The taxonomy includes format requirements and ratio constraints over reference-text overlap or sentence composition.Source: §F.3 · pp. 79–80

Sentence and word constraints

• Sentence and word constraints impose sequential or lexical rules, including increasing alliteration, sentence-length increments, alphabetic starts, palindromes, prime-length words, and vowel restrictions.These constraints govern relationships across sentences or properties of individual words.Source: §F.3 · p. 80

Prompt construction

• IFEval constraints are appended to unseen WildChat prompts to form final prompts, except for the custom instruction group.The custom group instead contains standalone task-style instructions, including reverse-order responses and CSV generation.Source: §F.3 · p. 80, Table 47 · p. 81

• Table 48 compares Tülu 3, Hermes 3 Llama 3.1, and Llama 3.1 Instruct at 8B and 70B scales.Source: §F.4 · p. 81
• The HREF breakdown reports instruction-following subtask win rates against Llama 3.1 405B Instruct.Source: Table 48 · p. 82
• Evaluation setups include Llama 3.1 70B Instruct judging, judging with human-written references, and embedding-based similarity with human-written references.These setups are denoted L, LH, and E, respectively.Source: Table 48 · p. 82