Source-linked AI summary
From Rollouts to Recipes: Self-Contained Post-Training for LLMs
Yifei Li, Lingling Zhang, Muye Huang, Zihan Ma, Jiashuai Liu, Jun Liu
TL;DR
Uniform post-training recipes overlook different learning states revealed by a model’s rollouts. Self-Routing uses rollout correctness and confidence to select GRPO, OPSD, regularization, or skipping for each sample. It consistently outperforms uniform and fixed-routing alternatives on the evaluated mathematical and general reasoning benchmarks.
Problem
Global recipes cannot adapt to heterogeneous sample-level learning states revealed by current-policy rollouts.
Method
Self-Routing uses correctness and confidence from the model’s on-policy rollouts to route each sample among GRPO, OPSD, regularization, and skipping without extra annotation, external evaluators, or additional sampling.
Results
Self-Routing achieves the highest average score across evaluated backbones and consistently outperforms uniform GRPO, uniform OPSD, fixed mixtures, and simpler routing baselines.
Takeaways & Limitations
Rollout behavior can guide both which samples are trained and which optimization mechanism is applied to them.
Takeaways & Limitations
The study does not yet provide a sufficiently deep mechanistic or theoretical explanation for why rollout patterns align with specific optimization signals.
Abstract
from arXiv · showhide
Post-training large language models usually applies a single training recipe to all samples, even though the model's own rollouts reveal different sample-level learning states. We propose Self-Routing, a behavior-conditioned post-training framework that uses rollout correctness and confidence to decide how each sample should be optimized. Depending on its behavior state, a sample is routed to GRPO, on-policy self-distillation, regularization, or skipping, allowing training to adapt without external teachers, extra annotations, or additional sampling. Experiments on mathematical reasoning across Qwen3 and Qwen3.5 backbones show that Self-Routing consistently improves over uniform GRPO, uniform OPSD, fixed mixtures, and simpler routing baselines. Further analyses show that the routing distribution changes over training and reduces unnecessary updates on low-signal or already stable samples.
1 Introduction
Prior verifier-based post-training commonly applies global recipes or uses rollout behavior mainly for selection, despite rollout signals revealing heterogeneous sample-level learning states. The paper proposes self-contained behavior-conditioned routing, which converts these signals into sample-level optimization recipes and reports improvements over several uniform and selection-based baselines.
- Prior behavior-aware methods mainly use rollout signals for filtering, curricula, fixed recipe mixtures, or inference-time answer selection.
- Global training recipes apply the same optimization mechanism or fixed objective mixture to every sample.
- Current-policy rollouts expose heterogeneous learning states because mixed, failed, solved, and high-confidence behaviors differ in their useful optimization signals.Uniformly failed samples may offer little verifier-based RL gradient, while mixed rollouts can inform RL and reliably solved samples may suit conservative objectives.
- Self-contained behavior-conditioned routing assigns training actions according to each sample’s on-policy rollout behavior rather than applying one global recipe.Its signals come from the model’s own rollouts, avoiding external teachers, offline traces, and fixed reference trajectories.
- Self-Routing routes samples among GRPO, OPSD, regularization, and skipping, and improves over uniform GRPO, uniform OPSD, fixed mixtures, and simpler routing baselines.
2 Related Work
Related work improves verifier-based post-training through global reinforcement-learning objectives, reward and optimization techniques, data selection, curricula, and rollout-based inference adaptation. This paper instead studies which optimization signal should be applied to each sampled prompt given its rollouts.
- Verifier-Based Post-Training: Verifier-based post-training uses outcome rewards from answers, unit tests, or execution feedback on verifiable tasks.
- Verifier-Based Post-Training: Existing verifier-based methods primarily improve global PPO, GRPO, or DAPO procedures through reward shaping, advantage estimation, regularization, sampling, and stabilization.
- Data Selection and Curriculum Learning: Data selection and curriculum methods adjust which samples appear, their order, or their frequency using quality, difficulty, uncertainty, reward, or training stage.
- The paper asks a complementary question: given a sampled prompt and its rollouts, what optimization signal should be applied to it?
- Rollout Uncertainty and On-Policy Signals: Rollout distributions and uncertainty signals have mainly supported inference-time answer selection or dynamic reasoning-computation allocation.
3 Method
Self-Routing collects on-policy rollouts, estimates correctness and confidence, and maps each sample’s behavior state to one disjoint training recipe. The resulting queues apply GRPO, OPSD, regularization, or skipping before aggregating active losses for a policy update.
- 3.1 Overview: Each sample receives on-policy rollouts from the current policy and verifier feedback, which are used to construct training updates under a shared data and rollout budget.
- 3.2 Rollout Collection: Rollout accuracy measures how often the current policy solves a sample under repeated sampling and changes as training proceeds.
- 3.3 Behavior Signals: Accuracy is represented with smooth low-, uncertain-, and high-accuracy membership scores centered on mostly failed, inconsistent, and reliably solved behavior.
- 3.3 Behavior Signals: Confidence is estimated from token-level predictive entropy and calibrated within the current batch to account for prompt length, task type, and batch composition.
- 3.4 Recipe Router: The router converts behavior signals into normalized recipe probabilities and assigns each sample to at most one disjoint objective per iteration.
- 3.4 Recipe Router: Mixed-correctness samples are routed toward GRPO, low-accuracy low-confidence samples toward OPSD, stable solved samples toward REG, and confident failures toward SKIP.
- 3.5 Recipe Assignment and Policy Update: GRPO uses group-relative verifier advantages, while OPSD trains the policy to imitate a target trajectory produced by the same base model conditioned on the problem and answer.OPSD requires target answers but no external teacher or additional chain-of-thought annotation.
- 3.5 Recipe Assignment and Policy Update: REG keeps reliably behaving samples close to a reference policy, while SKIP samples contribute no gradients and are excluded from the loss denominator.
4 Experiments
The experiments evaluate Self-Routing across models, benchmarks, baselines, efficiency, routing strategies, and training dynamics. Self-Routing achieves the strongest reported performance while adapting recipe allocation over training and selectively applying optimization mechanisms.
- Main Results: Self-Routing achieves the highest average score across all evaluated backbones, with Qwen3-4B improving from 61.0 to 73.7.This exceeds Naive-GRPO and Naive-OPSD by 6.9 and 3.3 points, respectively.
- Main Results: On Qwen3.5-4B, Self-Routing reaches 86.6 average score versus 79.8 for Naive-GRPO and 83.0 for Naive-OPSD.
- Main Results: Self-Routing shows the smallest MMLU-Pro degradation among post-training methods and remains second-best after Base on every backbone.GPQA-diamond usually improves after math-oriented post-training, whereas MMLU-Pro drops compared with the base model.
- Main Results: On Qwen3-4B, Self-Routing outperforms DAPO-style RL and PODS, achieving 80.9/71.1/59.3 on ID math, OOD verifiable reasoning, and general evaluation.It also obtains the best average score of 71.1 on SATBench, AutoLogi, and LiveCodeBench-v5.
- Efficiency Analysis: Self-Routing is more expensive than Naive-OPSD but less expensive than applying GRPO to all rollout groups.At G = 8, normalized costs are 64.0 for Naive-GRPO, 24.0 for Naive-OPSD, and 34.7 for Self-Routing.
- Ablation on Routing Strategies: Accuracy-Based Routing outperforms Naive-OPSD but still falls slightly behind Self-Routing across most benchmarks, while random routing baselines perform poorly.The comparison uses Qwen3-4B across ID mathematical reasoning and OOD general reasoning benchmarks.
- Ablation on Routing Strategies: Accuracy is the primary routing signal, while confidence calibration and behavior-conditioned assignment provide complementary gains.
- Training Dynamics: Routing ratios change substantially during training: OPSD dominates early, GRPO peaks mid-training, and Reg becomes dominant later.The reported integrated branch ratios are 30.8% GRPO, 30.4% OPSD, 25.5% REG, and 13.3% SKIP; Skip remains relatively low and decreases near the end.
5 Conclusion
Self-Routing converts the model’s rollout behavior into sample-level training recipes rather than applying one objective uniformly. Across Qwen3 and Qwen3.5 mathematical-reasoning experiments, it consistently outperforms several uniform, mixed-objective, and simpler routing baselines.
- Self-Routing uses rollout correctness and confidence to route samples to GRPO, OPSD, REG, or SKIP.The framework turns the model’s own rollout behavior into sample-level training recipes.
- Self-Routing consistently improves over uniform GRPO, uniform OPSD, fixed mixtures, and simpler routing baselines across Qwen3 and Qwen3.5 backbones.
- Different recipes become useful at different training stages, according to the reported routing dynamics.
- The results support using rollout behavior to guide both sample selection and optimization choice.
Limitations
The study supports behavior-conditioned routing primarily through intuition, empirical gains, and behavioral observations rather than a rigorous account of its optimization dynamics. Its validation is also centered on mathematical and related verifiable reasoning tasks, while the work focuses on routing rather than improving individual algorithms.
- The method lacks a sufficiently deep mechanistic or theoretical explanation for why rollout patterns align with particular optimization signals.The authors identify rigorous understanding of the underlying optimization dynamics as future work.
- Training experiments center on mathematical reasoning, leaving generalization to agent planning or open-ended instruction following unclear.Observed transfer is reported for SATBench, AutoLogi, and LiveCodeBench-v5, which remain related verifiable reasoning tasks.
- The work studies adaptive assignment of optimization mechanisms rather than improving the internal designs of GRPO-style or OPSD-like algorithms.Algorithmic improvements such as reward engineering and trajectory refinement are described as orthogonal to the routing objective.
Ethical Considerations
The implementation uses the same training infrastructure and verifier signals across methods, while routing assigns samples among four disjoint recipe queues. The reported setup uses on-policy rollouts, confidence and accuracy signals, and no external teacher or extra annotation.
- Implementation: The implementation uses the same data loader, rollout interface, verifier calls, tokenizer, and checkpointing code across Naive-GRPO, Naive-OPSD, and Self-Routing.The routing module is inserted between rollout collection and loss construction.
- Signals: For each prompt, the current policy generates G responses, the verifier returns binary outcomes, and the router computes rollout accuracy and entropy-based confidence.The router does not call an external teacher model, reward model, or data filter; OPSD targets are generated once during offline preprocessing.
- Routing implementation: Each batch is partitioned into disjoint GRPO, OPSD, REG, and SKIP queues, and the update applies the corresponding loss.SKIP samples contribute no gradient in the current update.
- Assignment: Recipe assignments are sampled categorically from normalized routing scores rather than selected by an argmax rule.Sampling is intended to prevent borderline samples from collapsing into one branch early in training.
- Reference policy: The REG branch uses a fixed reference policy during each update, with no separate inference service required.
D Training and Evaluation Configuration
Experiments train on DAPO-Math-17K and evaluate across six mathematical and general reasoning benchmarks using a shared runtime stack. Self-Routing assigns recipes by rollout behavior, with measured branch ratios and a reported normalized compute cost, while confidence diagnostics support using entropy as a secondary signal.
- Datasets and evaluation: DAPO-Math-17K is used for training, while evaluation covers GSM8K, MATH-500, AIME24, AIME25, MMLU-Pro, and GPQA-diamond.The reported average is the macro-average over these six benchmarks.
- Baselines: The experiments compare Base, Naive-GRPO, Naive-OPSD, round-wise random routing, fixed-ratio random routing, and accuracy-based routing.
- Routing: Each sample is assigned to GRPO, OPSD, REG, or SKIP using categorical sampling, with branch meanings tied to rollout variation, recoverability, confidence, and training signal.GRPO targets useful relative reward variation; OPSD targets failed but recoverable samples; REG targets confidently solved samples; SKIP targets low-signal behavior.
- Compute: Naive-GRPO, Naive-OPSD, and Self-Routing have normalized costs of 24.0, 24.0, and 34.7, respectively.Self-Routing is not the cheapest recipe in this implementation; expensive update types are assigned to fewer samples.
- Confidence diagnostics: Correct responses often have slightly higher confidence and lower entropy than incorrect responses, with gaps varying by model family and scale.The diagnostics support confidence as a secondary rather than sole routing signal.
H Router Motivation and Ablations
The router is motivated by diagnostic experiments that link rollout behavior to different optimization recipes. These experiments support four behavior-specific branches rather than a theoretically optimal assignment rule.
- Diagnostic setup: For each prompt, eight rollouts provide rollout accuracy and normalized confidence for constructing behavior-specific training subsets.GRPO and OPSD are then applied separately to these subsets.
- Diagnostic results: Table 3 reports the diagnostic results as ID Math / General scores.
- Behavior-to-recipe mapping: Mixed-correctness samples favor GRPO because rollout groups provide reward contrast.
- Behavior-to-recipe mapping: Recoverable low-confidence failures favor OPSD, while stable solved samples show little benefit from aggressive optimization.
- Behavior-to-recipe mapping: Confident failures perform poorly under both active recipes, motivating conservative regularization or skipping branches.
I Additional Router Ablations
Additional ablations test how Self-Routing uses accuracy, confidence, calibration, behavior-conditioned assignment, and conservative branches.
- Routing signals: Accuracy is the strongest routing signal: accuracy-only routing remains relatively close to Self-Routing, whereas confidence-only routing degrades substantially.
- Routing signals: Confidence adds complementary information beyond accuracy for routing samples.
- Design ablations: Removing confidence calibration, behavior-conditioned assignment, or the REG/SKIP branches consistently reduces performance.
- Design ablations: Table 4 presents ablations of the Self-Routing design on Qwen3-4B.
J Additional Baselines and OOD Evaluation
Additional evaluation on Qwen3-4B includes an OOD-V aggregate over SATBench, AutoLogi, and LiveCodeBench-v5.
- Additional evaluation: OOD-V is defined as the average over SATBench, AutoLogi, and LiveCodeBench-v5 in the additional Qwen3-4B evaluation.