PaperlayerView exact revision on arXiv

Source-linked AI summary

Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models

Boxin Wang, Chankyu Lee, Nayeon Lee, Sheng-Chieh Lin, Wenliang Dai, Yang Chen, Yangyi Chen, Zhuolin Yang, Zihan Liu, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping

arXiv:2512.13607v2cs.CLcs.AIcs.LG

TL;DR

Heterogeneous response lengths and reward computation complicate joint reinforcement learning across domains. Cascade RL trains sequentially by domain, producing a unified model with thinking and instruct modes while retaining reasoning performance.

  • Problem

    Cross-domain variation in response length and reward computation complicates RL infrastructure, training curricula, and hyperparameter tuning for general-purpose reasoning models.

  • Method

    Cascade RL trains models sequentially across domains instead of jointly blending heterogeneous prompt distributions.

  • Results

    Cascade RL achieves state-of-the-art performance across multiple domains while closing the reasoning-performance gap between unified and dedicated thinking models.

  • Takeaways & Limitations

    Domain-wise RL supports a single model that operates in both instruct and deep-thinking modes without sacrificing reasoning performance.

  • Takeaways & Limitations

    RewardBench is an imperfect proxy for selecting the reward model that produces the best aligned policy, and RLHF introduces additional variance.

Abstract

from arXiv · show

Building general-purpose reasoning models with reinforcement learning (RL) entails substantial cross-domain heterogeneity, including large variation in inference-time response lengths and verification latency. Such variability complicates the RL infrastructure, slows training, and makes training curriculum (e.g., response length extension) and hyperparameter selection challenging. In this work, we propose cascaded domain-wise reinforcement learning (Cascade RL) to develop Nemotron-Cascade, capable of operating in both instruct and deep thinking modes, without any performance gap relative to a thinking-only counterpart. Departing from conventional approaches that blend heterogeneous prompts from different domains, Cascade RL orchestrates sequential, domain-wise RL, reducing engineering complexity and delivering state-of-the-art performance across a wide range of benchmarks. Notably, RLHF for alignment, when used as a pre-step, boosts the model's reasoning ability far beyond mere preference optimization, and subsequent domain-wise RLVR stages rarely degrade the benchmark performance attained in earlier domains and may even improve it (see an illustration in Figure 1). Our 14B model, after RL, outperforms its SFT teacher, DeepSeek-R1-0528, on LiveCodeBench v5/v6/Pro and achieves silver-medal performance in the 2025 International Olympiad in Informatics (IOI). We transparently share our training and data recipes.

1. Introduction

The paper scales cascaded, domain-wise reinforcement learning into an open post-training recipe for Nemotron-Cascade, targeting unified models that support both thinking and instruct modes. Across broad benchmarks, the resulting models achieve state-of-the-art performance, including strong LiveCodeBench results and silver-medal performance at IOI 2025.

  • Motivation: General-purpose RL is difficult because domains differ substantially in response length and reward-verification latency, complicating infrastructure, curriculum design, and hyperparameter selection.The paper motivates sequential domain-wise training as a response to this heterogeneity.
  • Motivation: The work addresses the tension between thinking and instruct models by pursuing a single unified model, despite prior concerns that unified models can lose reasoning benchmark performance in thinking mode.The introduction cites Qwen3’s later separation into dedicated thinking and instruct variants as an example of this challenge.
  • Contribution: Nemotron-Cascade scales Cascade RL across human-feedback alignment, instruction following, mathematics, competitive programming, and software engineering, achieving state-of-the-art results across these domains.The recipe starts from pretrained Qwen3-8B-Base and Qwen3-14B-Base models to support transparent comparison and knowledge sharing.
  • Contribution: RLHF improves overall response quality and reasoning performance, while subsequent domain-specific RL stages rarely degrade earlier benchmark performance and may improve it.Cascade RL also allows hyperparameters and training curricula to be tailored to each domain.
  • Contribution: Nemotron-Cascade-8B enables user control over thinking and non-thinking or instruct modes at each conversational turn, closing the reasoning gap with a dedicated 8B-Thinking model.The comparison uses the same thinking and non-thinking training data for both models.
  • Results: The 14B dedicated Thinking model, using a 64K-token inference budget, outperforms Gemini-2.5-Pro-06-05, o4-mini (medium), Qwen3-235B-A22B, and DeepSeek-R1-0528 on LiveCodeBench v5/v6, and achieves silver-medal performance at IOI 2025.DeepSeek-R1-0528 is identified as the model’s SFT teacher.

2. Main Results

Nemotron-Cascade-8B and Nemotron-Cascade-14B-Thinking achieve best-in-class performance across almost all evaluated benchmarks. The models show particularly strong results on LiveCodeBench and SWE-bench Verified, while the paper also reports stage-wise Cascade RL ablations.

  • Main results: Nemotron-Cascade-8B and Nemotron-Cascade-14B-Thinking achieve best-in-class performance across almost all benchmarks in the main evaluation.Scores are reported as pass@1 averaged over k generations, with k typically ranging from 4 to 64; unified-model reasoning benchmarks use thinking mode, while IFEval and IFBench use the higher thinking or non-thinking score.
  • Ablations: Stage-wise Cascade RL ablations are reported for SFT, RLHF, Instruction-Following RL, Math RL, Code RL, and SWE RL in Tables 2 and 4–8.The ablations evaluate models after each stage of the pipeline illustrated in Figure 2.
  • Main results: 74.3 on LiveCodeBench v5 and 71.1 on v6 are achieved by Nemotron-Cascade-8B, comparable to DeepSeek-R1-0528’s 74.8 and 73.3 despite its 671B size.DeepSeek-R1-0528 served as the SFT teacher and generated the code-prompt responses used for SFT data curation.
  • Main results: 43.1% on SWE-bench Verified is achieved by Nemotron-Cascade-14B, exceeding the specialized DeepSWE-32B’s 42.2% and the model’s 8B counterpart’s 37.2%.Additional details and test-time scaling results are deferred to Section §7.

3. Supervised Fine-Tuning

The SFT stage builds foundational capabilities through a two-stage, multi-domain curriculum and explicit control of thinking versus non-thinking generation. After SFT, the 8B unified model matches the dedicated 8B thinking model on reasoning benchmarks and surpasses it on IFEval.

  • SFT curriculum: The two-stage SFT curriculum covers general domains, math, science, coding, tool use, software engineering, dialogue, question answering, writing, safety, role-playing, and instruction following.Stage 1 uses up to 16K-token sequences and includes general, math, science, and code reasoning data; Stage 2 extends responses to 32K tokens to enhance reasoning.
  • Interaction schema: The interaction schema uses ChatML with /think and /no_think flags appended to each user prompt, enabling consistent global behavior or mode switching within multi-turn conversations.The model also receives tool definitions in the system prompt and emits tool calls inside <tool_call> tags.
  • Data curation: The general-domain corpus contains 2.8M samples and 3.2B tokens, with post-processing that filters incorrect verifiable answers and cross-validates responses without ground truth.Curation also addresses brevity, quality variation, stylistic inconsistency, and data scarcity through response generation and multi-turn augmentation.
  • SFT results: The 8B unified model performs on par with the dedicated 8B Thinking model across reasoning benchmarks and surpasses it on IFEval.Both models use the same SFT thinking data, while the unified model additionally incorporates non-thinking data.

4. Cascade RL

Cascade RL sequentially applies RLHF, instruction-following, math, code, and software-engineering stages, improving specialized capabilities while largely preserving earlier-domain performance. The resulting unified Nemotron-Cascade-8B matches its thinking counterpart on reasoning tasks and performs substantially better on instruction following.

  • RLHF: RLHF significantly improves nearly every benchmark except IFEval by penalizing overly long, verbose, and repetitive generations, especially in thinking mode.The IFEval degradation is attributed to semantic overlap between RLHF prompts and IFEval test prompts.
  • IF-RL: Improved IF-RL substantially raises IFEval and IFBench scores, with only controlled small ArenaHard degradation, while reducing entropy and reasoning-token length.Naive rule-based verification degraded human alignment because it ignored response quality beyond instruction adherence.
  • Math RL: Math RL improves AIME 2024 and 2025 while minimally affecting knowledge-reasoning and alignment benchmarks, and RLHF initialization strengthens and length-efficiencizes math reasoning.A final 40K-token training stage was introduced after performance plateaued below 30% on hard problems, explicitly encouraging fuller context use.
  • Code RL: Code RL gives strong LiveCodeBench gains: 14B-Thinking scores 78.0 on LCB v5 and 74.8 on LCB v6, outperforming DeepSeek-R1-0528 (671B).An execution-free verifier enables scalable code-repair RL without managing numerous Docker instances.
  • SWE RL: SWE RL substantially improves SWE-bench Verified, with 14B-Thinking reaching a 43.1 pass@1 resolve rate and outperforming DeepSWE-32B and SWE-agent-LM-32B.Context extension from 16K to 24K tokens produced steady gains in long-context understanding and cross-file reasoning.
  • Overall Cascade RL: The unified Nemotron-Cascade-8B performs comparably to Nemotron-Cascade-8B-Thinking on reasoning tasks while performing substantially better on instruction-following tasks.

5. Deep Dive on Competitive Coding

Nemotron-Cascade performs strongly on competitive programming benchmarks and achieves silver-medal performance at IOI 2025. Its results reflect both scalable coding capability and a feedback-driven test-time strategy that improves reasoning on harder problems.

  • Competitive programming benchmarks: Nemotron-Cascade-8B significantly outperforms nearly all recently released comparable-size reasoning models and matches the performance of the 32B OpenReasoning-Nemotron baseline.These comparisons cover the latest LiveCodeBench and LiveCodeBench-Pro splits while avoiding problems released before the 08/2024 training-data cutoff.
  • IOI 2025: The IOI pipeline repeatedly generates, filters, selects, and submits candidates, then conditions later rounds on official verdicts and solved-subtask code insights.Each round samples 20 candidates per subtask, filters incomplete or sample-failing code, and maintains up to five recent failed submissions in the prompt history.
  • IOI 2025: 343.37 overall score earns Nemotron-Cascade-14B-Thinking a silver medal at IOI 2025 with at most 1,000 generations and 50 official submissions per problem.The model uses a 128K-token thinking budget within a multi-round test-time scaling pipeline.
  • Code-RL ablations: Higher Code-RL sampling temperatures improve code-reasoning performance and sample efficiency, although they can make training less stable than lower-temperature settings.The ablation compares temperatures 0.6, 0.8, and 1.0 on the unified 8B model.
  • Cascade RL analysis: Across LiveCodeBench v6 difficulty splits, RLHF improves reasoning-token efficiency, Math RL helps medium problems, and Code RL supplies the final lift on medium and hard problems.Easy-problem performance saturates above 99% after the initial stages, while later gains come from expanded reasoning traces on harder splits.

6. Deep Dive on RLHF

This section studies RLHF recipe design, finding that larger reward models improve style-controlled ArenaHard performance and that training in both thinking modes outperforms single-mode training. It also identifies stabilization techniques that extend training under weaker rewards, while stronger reward models can make them unnecessary.

  • Reward model selection: The study attributes stronger reward-model performance to more accurate and consistent reward signals that enable stable RLHF and better performance on other tasks.This conclusion follows the section’s reward-model selection experiments and their reported downstream effects.
  • RLHF training mode: Training the unified 8B model with an equal split of thinking and non-thinking prompts outperforms training exclusively in either mode.ArenaHard, AIME, and LiveCodeBench are evaluated in thinking mode only.
  • Reward model selection: Larger reward models produce the strongest style-controlled ArenaHard performance, while smaller models degrade ArenaHard, math, and code capabilities.The comparison uses AceReason-Nemotron-1.0-7B (Chen et al., 2025) with reward models ranging from 7B to 72B; the 7B model also shows a substantial style-control gap suggestive of reward hacking.
  • RLHF stabilization: For 7B-reward-model RLHF, stabilization techniques extend stable training from 350 to 950 steps and improve ArenaHard scores.The reported techniques include KL-penalty loss and policy-gradient loss aggregation; Table 10 reports the corresponding pre-collapse checkpoints and steps.
  • RLHF stabilization: With a 72B reward model, RLHF is already stable, so omitting the stabilization techniques gives comparable or sometimes slightly better downstream performance.The techniques are therefore recommended as a toolbox when training shows instability; otherwise, the recipe in § 4.3.2 is sufficient.

7. Deep Dive on SWE

The SWE deep dive improves code localization and repair through complementary retrieval-generation, execution-free semantic rewards, longer-context training, and test-time scaling. On SWE-bench Verified, test-time scaling with patch validation substantially improves repair performance as more samples are considered.

  • Code Localization: Code localization combines generation-based ranking from repository structure with NV-Embed-Code retrieval over source contents, fused using reciprocal rank fusion.Retrieval outperforms generation, while aggregating multiple generation rollouts consistently improves localization at top and higher ranks.
  • Reward Design: RL improves code repair, with semantic-similarity rewards outperforming lexical rewards; threshold shaping improves lexical training but not semantic training.Lexical rewards below 0.5 tend to provide unreliable supervision, whereas semantic similarity continues providing meaningful signals without the same shaping benefit.
  • Long-Context Training: Training on longer prompts from 16K to 32K improves repair capability, addressing the need to identify buggy patches within retrieved code contents.Preliminary results show resolve rates drop when prompts exceed 24K alongside 16K responses, motivating longer-context training data.
  • Test-Time Scaling: Nemotron-Cascade-8B reaches 43.6% best@32 and 57.7% pass@32 on SWE-bench Verified, versus 39.2% best@2 and a 15.6-point pass@32–best@32 gap.Best@k consistently outperforms majority@k, while pass@k increases monotonically and majority voting saturates earlier as k grows.

8. Related Work · Appendix

The related work reviews RLHF, RLVR, SFT and distillation, and unified reasoning models. It positions Cascade RL as addressing heterogeneous multi-domain RL while investigating RLHF–RLVR interactions and supporting broad reasoning capabilities.

  • 8.1. Reinforcement Learning for LLMs: RLHF aligns SFT or instruction-tuned LLMs using human-feedback rewards, whereas RLVR uses objective, deterministic verification criteria and has advanced frontier reasoning models.The reviewed RLHF literature includes Bai et al. (2022), Ouyang et al. (2022), Liu et al. (2024), and Wang et al. (2024); RLVR examples include Guo et al. (2025), Kimi-Team et al. (2025), and Yang et al. (2025).
  • 8.1. Reinforcement Learning for LLMs: Cascade RL addresses heterogeneous multi-domain RL, whose joint-prompt design complicates infrastructure, curriculum, and hyperparameter tuning and can produce suboptimal performance.The referenced DeepSeek-R1 and Qwen3 training process uses reasoning-oriented and all-domain stages with diverse prompts jointly trained within each stage.
  • 8.1. Reinforcement Learning for LLMs: The work releases open Cascade RL recipes and datasets spanning math, coding, science, instruction following, software engineering, and general-domain tasks, while systematically studying RLHF–RLVR interplay.The passage characterizes this interaction as underexplored in existing literature.
  • 8.2. Supervised Fine-Tuning and Distillation: SFT prepares pretrained LLMs for conversation and instruction following, while distillation transfers capabilities from RL-trained teachers into compact reasoning models.RL is generally applied to SFT models for state-of-the-art reasoning; prior work found RL can narrow initial SFT performance gaps when exploration and exploitation are appropriately balanced.
  • 8.3. Unified Reasoning Models: Recent thinking models emphasize long chain-of-thought generation for analysis, solution exploration, verification, and answer correction, including OpenAI o1, DeepSeek-R1, Qwen3-Thinking, and other systems.The passage also lists o3, o4-mini, MiniMax-M1, gpt-oss, and Kimi-K2-Thinking among recent dedicated thinking models.
  • 8.3. Unified Reasoning Models: Unified reasoning models combine instruct and thinking modes through global system-prompt control, per-turn user switching, or automatic routing.Examples include Llama-Nemotron, Qwen3, GLM-4.5, DeepSeek-V3.1, and GPT-5.

B. Benchmarks and Evaluation Setups

The evaluation spans knowledge, alignment, math, and code-generation benchmarks, using task-specific modes, sampling protocols, response budgets, and baseline configurations. Metrics include exact-match accuracy, prompt-strict instruction satisfaction, pass@1, and averaged multi-generation performance.

  • Knowledge reasoning: Knowledge evaluation covers MMLU’s 14,079 questions across 57 subjects, with thinking-mode exact-match accuracy from one generation per question.MMLU assesses broad world knowledge and problem-solving ability; both unified reasoning and dedicated thinking models are evaluated in thinking mode.
  • Evaluation protocols: Across benchmark families, Nemotron-Cascade generally uses temperature 0.6 and top-p 0.95, with budgets and YaRN scaling adjusted by task and baseline settings followed when available.Knowledge, math, and LiveCodeBench use 64K-token budgets with YaRN scaling factor 2, while alignment uses 32K-token maximum response length and SWE-bench uses a 32K-token thinking budget.
  • Alignment: Alignment evaluation uses IFEval prompt strict, while IFBench and ArenaHard are evaluated in thinking mode under a 32K-token maximum response length.IFEval contains 541 prompts and 25 verifiable instructions; prompt strict measures the percentage of prompts satisfying all instructions, unlike instruct strict’s instruction-level measure.
  • Math reasoning: Math evaluation uses AIME 2024 and AIME 2025, each containing 30 problems, with Nemotron-Cascade given a 64K-token thinking budget and baselines at least 64K tokens.Nemotron-Cascade uses temperature 0.6, top-p 0.95, and YaRN scaling factor 2; baselines follow recommended inference settings.
  • Code generation: Code evaluation includes LiveCodeBench v5/v6 and Pro plus SWE-bench Verified, reporting pass@1 with avg@8 for recent LiveCodeBench subsets and avg@4 for SWE-bench.SWE-bench Verified contains 500 human-verified samples; LiveCodeBench uses algorithmic coding problems collected from AtCoder and LeetCode.

C. Prompt Templates

The prompt templates generate off-topic inputs that remain superficially similar while requiring different correct responses, then judge whether the original and generated instructions ask the same thing. For IOI 2025 test-time scaling, templates request formatted Python solutions and provide accepted solutions, constraint variants, and submission histories as references.

  • Step 1: Generate offtopic prompts: The generator creates an input highly relevant to but different from the given input, with a superficially similar yet incorrect transferable answer.The generated input should not have the same correct response as the given input.
  • Step 1: Generate offtopic prompts: The generation template supplies the original instruction through the placeholder “Given input: {instruction}”.
  • Step 1: Generate offtopic prompts: A binary judge compares Instruction A and Instruction B and answers “YES” or “NO” on whether they ask the same thing.The template explicitly frames the comparison between the original and generated instructions.
  • C.3. Prompt Templates for Test-Time Scaling on IOI 2025: The IOI 2025 template asks the model to write Python code and place it in a specified solution-code format around the problem statement.
  • C.3. Prompt Templates for Test-Time Scaling on IOI 2025: The IOI template provides accepted solutions under different input constraints as references for deriving insights.
  • C.3. Prompt Templates for Test-Time Scaling on IOI 2025: It also supplies incorrect submission histories with official verdicts and reliable problem conditions so the model can improve its code when useful.

D. Training Hyperparameters · D.1. Multi-Stage SFT

This section documents the multi-stage SFT training hyperparameters for the 8B and 14B models. The unified and thinking models use the same hyperparameters.

  • D.1. Multi-Stage SFT: The hyperparameters are explicitly listed for the 8B model's multi-stage SFT.
  • D.1. Multi-Stage SFT: The hyperparameters are explicitly listed for the 14B model's multi-stage SFT.
  • D. Training Hyperparameters: The section presents the hyperparameters as a tabulated training configuration rather than describing separate settings for the two model variants.
  • D.1. Multi-Stage SFT: Together, the listed settings cover multi-stage SFT across two model sizes and the unified and thinking variants.
  • D. Training Hyperparameters: Table 14 reports the multi-stage SFT training hyperparameters for both the 8B and 14B models.The table covers the training setup for both model scales.
  • D.1. Multi-Stage SFT: The unified and thinking models share the same multi-stage SFT hyperparameters.

D.2. RLHF

This section presents the RLHF training hyperparameters for the 8B and 14B models, noting that unified and thinking models use the same settings. Unified models split prompts equally between non-thinking and thinking modes.

  • RLHF hyperparameters for the 8B and 14B models are reported in Table 15.
  • Unified and thinking models share the same RLHF hyperparameters, while unified models divide prompts equally between non-thinking and thinking modes.

D.3. IF-RL

This section specifies the IF-RL training hyperparameters for the 8B and 14B models, noting that unified models are trained in non-thinking mode.

  • IF-RL training hyperparameters for the 8B and 14B models are provided in Table 16.
  • Unified models are trained in the non-thinking mode during IF-RL.

D.4. Math RL · D.5. Code RL · D.6. SWE RL

The appendix specifies training hyperparameters for Math RL, Code RL, and SWE RL across the 8B and 14B model variants. The tables also clarify when unified models are trained in thinking mode and which variants share hyperparameters.

  • D.4. Math RL: Math RL hyperparameters are provided separately for the 8B and 14B models in Tables 17 and 18.Table 17 covers the 8B models, while Table 18 covers the 14B-Thinking model.
  • D.4. Math RL: Table 17 uses shared hyperparameters for the 8B unified and thinking models, with unified models trained in thinking mode.This configuration applies to the 8B Math RL models.
  • D.5. Code RL: Code RL hyperparameters are listed for the 8B-Thinking, 8B unified, and 14B-Thinking models in Table 19.The 8B unified model is trained in thinking mode.
  • D.5. Code RL: The Code RL setup covers three variants: 8B-Thinking, 8B unified, and 14B-Thinking.These variants are the scope of the hyperparameter specification in Table 19.
  • D.6. SWE RL: SWE RL hyperparameters are provided for the 8B unified, 8B-Thinking, and 14B-Thinking models in Table 20.The accompanying text identifies the same three model variants as the SWE RL training configurations.
  • D.6. SWE RL: In SWE RL, the 8B unified and thinking models share hyperparameters, and unified models are trained in thinking mode.Table 20 records these shared-configuration and training-mode conditions.

E. ELO Rating Analysis

The section estimates Codeforces Elo ratings for Nemotron-Cascade-8B and Nemotron-Cascade-14B-Thinking across 51 recent contests using simulated participation and standard contest scoring. Ratings vary substantially across rounds, revealing inconsistent coding performance.

  • E. ELO Rating Analysis: Ratings are estimated from simulated participation in 51 Codeforces contests, allowing up to N=8 independent submissions per problem and using LiveCodeBench Pro evaluations.Responses are generated with a temperature of 0.6; the supplied passage truncates the remaining sampling configuration.
  • E. ELO Rating Analysis: The procedure ranks model contest performance against human contestants using final scores and applies standard Codeforces or ICPC-style penalty rules.Unsolved problems do not contribute penalties, and the resulting rank is converted into an implied performance rating.
  • E. ELO Rating Analysis: Across contests, Nemotron-Cascade-14B-Thinking’s estimated rating ranges from above 2600 on Round 1015 to below 1000 on Round 1024 Div.1, where it solved no problems.The model used up to 8 attempts yet failed to solve any problems on Round 1024 Div.1.
Loading 2512.13607v2…