PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost

Junkeun Yi; Damon Mosk-Aoyama; Baihe Huang; Ritu Gala; Charles Wang; Sugam Dipak Devare; Khushi Bhardwaj; Abhibha Gupta; Oleksii Kuchaiev; Jiantao Jiao; Jian Zhang; Venkat Srinivasan

Source-linked AI summary

PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost

Junkeun Yi, Damon Mosk-Aoyama, Baihe Huang, Ritu Gala, Charles Wang, Sugam Dipak Devare, Khushi Bhardwaj, Abhibha Gupta, Oleksii Kuchaiev, Jiantao Jiao, Jian Zhang, Venkat Srinivasan

arXiv:2603.21383v1Mar 22, 2026cs.AI

TL;DR

Long-horizon agentic post-training presents a trade-off between SFT’s compute efficiency and E2E RL’s OOD retention and rollout cost. PivotRL operates on SFT trajectories with local rollouts at informative pivots and functional-equivalence rewards. Compared with SFT on identical data, it improves in-domain and OOD accuracy, while matching E2E RL accuracy on agentic coding tasks with fewer rollout turns.

Problem
SFT is compute-efficient but often degrades OOD performance, while E2E RL retains OOD capabilities at high computational cost from repeated multi-turn rollouts.
Method
PivotRL filters SFT-derived intermediate states for mixed-outcome pivots, performs local on-policy rollouts, and rewards functionally equivalent actions with domain-appropriate verifiers.
Results
+4.17 higher average in-domain accuracy and +10.04 OOD retention versus standard SFT on identical data; PivotRL also reaches comparable E2E RL accuracy with 4× fewer rollout turns.
Takeaways & Limitations
PivotRL combines local rollout efficiency with stronger action-space coverage and OOD retention for long-horizon agentic post-training.

Abstract

from arXiv · show

Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers from out-of-domain (OOD) degradation. Conversely, end-to-end reinforcement learning (E2E RL) preserves OOD capabilities, but incurs high compute costs due to many turns of on-policy rollout. We introduce PivotRL, a novel framework that operates on existing SFT trajectories to combine the compute efficiency of SFT with the OOD accuracy of E2E RL. PivotRL relies on two key mechanisms: first, it executes local, on-policy rollouts and filters for pivots: informative intermediate turns where sampled actions exhibit high variance in outcomes; second, it utilizes rewards for functional-equivalent actions rather than demanding strict string matching with the SFT data demonstration. We theoretically show that these mechanisms incentivize strong learning signals with high natural gradient norm, while maximally preserving policy probability ordering on actions unrelated to training tasks. In comparison to standard SFT on identical data, we demonstrate that PivotRL achieves +4.17% higher in-domain accuracy on average across four agentic domains, and +10.04% higher OOD accuracy in non-agentic tasks. Notably, on agentic coding tasks, PivotRL achieves competitive accuracy with E2E RL with 4x fewer rollout turns. PivotRL is adopted by NVIDIA's Nemotron-3-Super-120B-A12B, acting as the workhorse in production-scale agentic post-training.

1. Introduction

Long-horizon agentic post-training must balance SFT’s compute efficiency against E2E RL’s stronger generalization and higher rollout cost. PivotRL uses local on-policy rollouts, informative pivots, and functional rewards to target this trade-off.

SFT is compute-efficient but often degrades OOD performance, whereas E2E RL retains OOD capabilities at high rollout cost.
PivotRL repurposes SFT trajectories through local on-policy rollouts that filter for intermediate turns with mixed success and failure outcomes.These pivot states enable brief partial rollouts instead of exhaustive full trajectories.
PivotRL assigns rewards to functionally equivalent actions rather than requiring exact string agreement with SFT demonstrations.
Theoretical analysis links reward variance to natural-gradient strength and shows functional rewards preserve probability ordering for unrelated actions.These properties support stronger local learning signals and reduced OOD degradation.
+14.111 average in-domain accuracy over the base model exceeds SFT’s +9.94 under identical training data across four agentic domains.PivotRL also reports +0.21 OOD change, while SFT reports −9.48 in non-agentic domains.
4× fewer rollout turns yield competitive accuracy with E2E RL on SWE-Bench.

2. Preliminaries and Motivating Observations

The preliminary analysis identifies inefficient local RL credit assignment as arising from rigid expert anchoring, exact-match rewards, and many turns with no normalized advantage. PivotRL addresses these bottlenecks by focusing local rollouts on informative intermediate states and rewarding acceptable alternatives.

Local RL conditions on intermediate expert states and samples turn-level actions instead of generating full end-to-end trajectories.An action is the complete assistant response at a model-call boundary.
The naive baseline rewards an action only when it exactly matches the demonstrated continuation.
57.34 accuracy for exact-match local RL trails 58.44 for same-data SFT on τ2-Bench.
71% of randomly sampled turns yield zero learning signal because their rewards are uniformly successful or uniformly failed.Under group-normalized RL, all-zero or all-one reward batches produce zero normalized advantage.
Functionally acceptable tool calls, shell commands, and search steps can be incorrectly discarded by exact matching.The miss rate measures strict-reward failures among functionally rewarded actions.
PivotRL filters for mixed-outcome pivots and replaces exact-match credit with verifier-based rewards.

3. PivotRL

PivotRL combines offline pivot selection with local on-policy rollouts and verifier-based rewards to make long-horizon agentic RL more compute-efficient. Its theory links mixed outcomes to stronger local updates and functional rewards to targeted probability shifts that preserve unrelated action orderings.

Method: PivotRL filters SFT-derived turns to retain difficult states whose sampled actions have nonzero reward variance and low reward mean.Uniformly solved or failed turns are removed because they provide little or no learning signal under group-normalized RL.
Method: PivotRL samples short local on-policy rollouts at retained pivots and optimizes a verifier-based GRPO-style objective.Compared with end-to-end RL, training uses only the short rollout needed to score each sampled turn-level action.
Method: Verifier-based rewards credit any action acceptable at the current turn, including functionally equivalent actions rather than only the demonstrated completion.Depending on the domain, verification can use normalized string or schema checks, task-specific equivalence rules, or a lightweight LLM judge.
Theoretical analysis: Only mixed-outcome rollout groups produce nonzero group-normalized updates, because identical rewards yield zero normalized advantages.This provides the direct rationale for filtering uniformly easy or impossible turns before training.
Theoretical analysis: Larger reward variance induces a larger natural-gradient norm and population GRPO score, making mixed-outcome turns stronger local learning states.The result is stated for the idealized KL-regularized statewise update and interprets binary verifier variance as success/failure mixing.
Theoretical analysis: Functional reward-based RL shifts probability mass toward acceptable actions while preserving the reference policy’s conditional ordering within acceptable and unacceptable actions.Under the paper’s task-locality assumption, this preserves relative rankings among task-unrelated actions and helps retain OOD performance.

4. Experiments

Across four agentic domains, PivotRL improves in-domain accuracy over same-data SFT while largely preserving OOD performance, and matches E2E RL on SWE-Bench at lower rollout cost. Ablations and training dynamics support the importance of pivot filtering and functional rewards.

In-Domain and OOD Accuracy: PivotRL improves in-domain accuracy over SFT across the evaluated agentic domains while nearly eliminating OOD degradation.Across four training runs, PivotRL averages +14.11 over Base versus +9.94 for SFT; OOD change averages +0.21 for PivotRL versus −9.83 for SFT.
Comparison to End-to-End RL: ∼4× fewer rollout turns and ∼5.5× less wall-clock time let PivotRL reach the same SWE-Bench accuracy as E2E RL.Both methods use the same number of compute nodes.
Ablation Study: Removing pivot filtering reduces τ2-Bench accuracy from 63.81 to 59.68, while removing functional reward yields 57.34.The ablation isolates each component’s contribution to the full PivotRL result.
Training Dynamics: Pivot sets preserve higher reward variance deeper into training and optimize to higher validation accuracy than random sampling.Random sampling’s per-batch reward variance collapses quickly, reducing useful advantage signals.
Production-Scale Use: PivotRL was used for agentic environments in Nemotron-3-Super’s large-scale post-training pipeline.Other reinforcement-learning environments handled reasoning and chat.

5. Related Works

Related work connects PivotRL to agentic language-model training, reinforcement learning, and SFT-to-RL bridges. The paper positions its contribution as turn-level verifier rewards from supervised trajectories for on-policy optimization without a separate reward model.

Agentic LLMs and Training Recipes: Agentic language models combine natural language with grounded actions across tools, code, and web navigation.The related work frames these systems as multi-turn environment-interacting models.
Agentic LLMs and Training Recipes: RL methods optimize multi-turn exploration and credit assignment, supported by scalable policy optimization, hierarchical modeling, and multi-agent coordination.The paper places PivotRL within this broader RL landscape.
Adapting Behavior Cloning into RL: PivotRL converts supervised trajectories into turn-level verifier rewards, enabling on-policy optimization without a separate reward model.This is the paper’s stated connection between agentic-task training and SFT-to-RL methods.
Agentic LLMs and Training Recipes: Offline SFT can suffer catastrophic forgetting and degraded generalization, while offline imitation suboptimality grows quadratically with task horizon.These limitations motivate environment interaction in related approaches.
Adapting Behavior Cloning into RL: Prior SFT-to-RL approaches use online data, partial trace prefixes, or SFT tokens as rollout rewards; PivotRL specializes this bridge with turn-based pivots and functional rewards.The comparison identifies PivotRL’s specific design emphasis.

6. Conclusion

PivotRL is a turn-level reinforcement-learning algorithm for tractable long-horizon agentic post-training. It improves accuracy and OOD retention over same-data SFT, approaches E2E RL accuracy at lower compute cost, and has been used in Nemotron-3-Super post-training.

Conclusion: +4.17 higher in-domain accuracy and +10.04 higher OOD retention than same-data SFT are reported across four agentic domains.The conclusion describes these as average gains relative to standard SFT.
Conclusion: PivotRL achieves comparable accuracy to E2E RL at a fraction of the compute cost.The conclusion attributes the gain to broader action-space coverage from on-policy rollouts and verifier-derived rewards.
Conclusion: The framework is designed to tractably post-train large language models for long-horizon agentic tasks.The paper identifies turn-level reinforcement learning as its core formulation.

A. Experiment Details

The experiments define pivots and actions operationally, distinguish OOD change from absolute performance, and compare random candidates with a low-reward-mean subset for pivot selection.

Terminology Conventions: A pivot candidate is every extracted assistant turn, while a pivot is the offline-filtered subset used for PivotRL training.This terminology separates all possible turn locations from the selected training set.
Terminology Conventions: An action is the full assistant completion at a model-call boundary, including tool-use turns, coding calls, bash commands, or search steps.Natural-language-only turns also count when the benchmark includes them.
Evaluation Definitions: OOD change is the score difference relative to the base model on benchmarks outside the training domain.This definition applies to the single-domain experiments.
Pivot Selection: The random subset 𝒟cand uses all pivot candidates, whereas the low-reward-mean subset 𝒟adv requires nonzero reward variance and a gap from the reference-policy mean.These are the two subsets used for pivot selection.

A.2. Domain-Specific Training Details

The experiments construct domain-specific pivot datasets and local action/verifier definitions, then compare PivotRL with E2E RL under matched accuracy. Across domains, training uses filtered pivots and brief rollouts rather than full trajectories.

Training pipeline: Pivot candidates are extracted from demonstration traces, profiled offline, and filtered before RL training in all four domains.Training samples only from the retained pivot set.
τ2-Bench: τ2-Bench treats every assistant turn as a candidate and defines an action as the full assistant turn, including natural language, tool calls, or both.The dataset contains 281,774 trajectories across 838 domains.
Terminal-Bench: Terminal-Bench treats each assistant bash action as a candidate and uses the next bash command as the action, with command deduplication applied to the low-reward-mean set.Its final dataset contains approximately 20,000 samples.
SWE-Bench Verified: SWE-Bench Verified treats each non-error tool call as a candidate and defines the action as the next assistant tool call in the coding trace.The local verifier matches tool-call names, while final success is measured by the full SWE-Bench harness.
E2E RL comparison: PivotRL reaches 32.67% accuracy with approximately 133K rollout turns, versus approximately 542K for E2E RL at the same accuracy.Both methods use the same number of compute nodes; PivotRL also requires approximately 5.5× less wall-clock time.
BrowseComp: BrowseComp treats each search-related assistant step as a pivot candidate and defines actions as the next browsing step, such as searching or opening a result.The final dataset contains 13,215 samples.

A.3.1. Effect of Turn Selection Strategy on 𝜏2-Bench

On τ2-Bench, more selective pivot selection improves accuracy, while the theory links mixed-outcome pivots to stronger natural-gradient learning signals through reward variance.

Pivot strategies: Low-reward-mean selection filters mixed-outcome pivots for both nonzero reward variance and a large demonstrated-action reward gap.Random pivots use all pivot candidates without filtering.
Empirical effect: 63.81 accuracy is achieved by low-reward-mean pivots, outperforming random pivots at 59.68 and SFT at 58.44.The comparison uses the best checkpoint over training.
Theory: The proof computes the natural gradient through exponential perturbations and its β-derivative along the KL path.These calculations connect reward variability to the GRPO update analyzed in the section.
Theory: A zero reward standard deviation implies a zero natural gradient for the statewise reward objective.Thus, reward variation is necessary for a nonzero local learning signal in this analysis.

B.2. Proof of Theorem 3.3

Theorem 3.3 characterizes functional-reward RL as a pointwise KL projection: it increases total probability on acceptable actions while preserving conditional distributions within acceptable and unacceptable sets.

Pointwise reduction: The statewise optimization can be solved independently because the objective has no coupling across states.The population-level claim then follows almost surely.
Boundary cases: When the reference policy assigns zero or full mass to acceptable actions, functional reward is constant and the unique minimizer remains the reference policy.These are the two boundary cases ρ = 0 and ρ = 1.
Optimization: For fixed acceptable-action mass q, the KL chain rule reduces the objective to separate conditional-distribution terms and mass-allocation terms.The conditional KL terms are minimized when the conditional distributions match the reference policy.
Solution: The unique KL minimizer is the block-rescaled reference distribution with total acceptable-action mass qβ(s).Acceptable and non-acceptable actions receive separate rescaling factors.
Interpretation: Functional-reward RL preserves the conditional distribution on both acceptable actions and their complement, thereby preserving relative ordering within each set.The ordering statements follow directly from the block-rescaling form.