PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

Zixun Huang, Kishan Panaganti, Haitao Mi, Leowei Liang

arXiv:2609.03241v1cs.LGcs.AI

TL;DR

FlowBalance addresses the fragility of self-improvement from on-policy reasoning, where verifier feedback is sparse and dense same-model guidance can reinforce false confidence. It learns a verifier-calibrated normalized distribution over complete responses through privileged-hindsight guidance and profiled trajectory balance, improving mathematical-reasoning performance over FlowRL on both Qwen3-4B and Qwen3-8B. The paper’s claims are scoped mainly to mathematical reasoning and a fixed prompt-distribution inner loop.

  • Problem

    Verifier feedback is reliable but sparse, while dense same-model guidance can reinforce false confidence or concentrate learning on narrow reasoning modes.

  • Method

    FlowBalance combines verifier-derived group advantages with privileged-hindsight trajectory guidance and fits the resulting normalized complete-response target using profiled trajectory balance.

  • Results

    FlowBalance improves average mathematical-reasoning performance over FlowRL on both Qwen3-4B and Qwen3-8B, while improving training speed and stability and avoiding direct OPSD’s response-length collapse.

  • Takeaways & Limitations

    Outcome-calibrated self-guidance can support verifier-grounded distributional self-improvement without a separate token-level imitation loss.

  • Takeaways & Limitations

    Experiments focus on mathematical reasoning and isolate the experience-to-policy inner loop on a fixed prompt distribution, so broader domains and outer-loop task evolution remain open.

Abstract

from arXiv · show

A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses. For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains, which are aggregated into a trajectory-level self-guidance score. FlowBalance calibrates this score with the verifier-derived group advantage: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when the rollout group provides no outcome preference. The resulting energy exponentially reweights a reference policy, and profiled trajectory balance fits the normalized target with one log-partition estimate per rollout group. This realizes outcome-calibrated self-guidance via trajectory balance, without a separate token-level imitation loss. Our analysis establishes within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control of target reward, and an exact correction against false-positive self-guidance on rejected responses. On mathematical reasoning, FlowBalance improves average performance over FlowRL on both Qwen3-4B and Qwen3-8B, while also improving training speed and stability, avoiding direct OPSD's response-length collapse, and exhibiting higher correct-strategy diversity in a controlled AIME24 diagnostic.

1 Introduction

FlowBalance frames self-improvement as learning a normalized distribution over on-policy reasoning responses, combining sparse verifier outcomes with dense but imperfect privileged-hindsight guidance. It calibrates self-guidance through verifier advantages and trajectory balance, with theory and mathematical-reasoning experiments showing stronger, faster, more stable, and more diverse improvement than comparison methods.

  • Motivation: RLVR supplies reliable outcome grounding but sparse terminal feedback, while dense self-guidance can reinforce rejected trajectories or narrow solution modes.These limitations motivate a distributional update that uses both signals without treating either as sufficient alone.
  • Method: FlowBalance learns a normalized complete-response distribution from on-policy experience, verifier advantages, and privileged-hindsight trajectory guidance.The method changes the policy-update object from a local signal to an explicitly normalized trajectory distribution.
  • Analysis: The analysis preserves within-group probability contrasts, characterizes a minimum reverse-KL update, and establishes monotonic verifier control of target reward.These are target-level and local-fitting properties, not guarantees of global convergence for arbitrary neural optimizers.
  • Results: FlowBalance improves the core four-benchmark average over FlowRL by 1.67 points on Qwen3-4B and 1.98 points on Qwen3-8B.The paper also reports stronger overall averages than GRPO, OPSD, RLSD, and FlowRL on both model sizes.
  • Results: On Qwen3-8B, FlowBalance reaches 0.5 AIME24 validation accuracy in about 100 steps versus roughly 143 for GRPO and remains stable over 400 steps.It also avoids direct OPSD’s response-length collapse and shows higher correct-only semantic strategy diversity than GRPO and RLSD.
  • Method: Verifier sign gating retains positive guidance, reverses it for negative-advantage responses, and disables it when the rollout group has no outcome preference.This prevents positive self-guidance on failures from becoming self-reinforcing supervision.

2 Preliminaries

FlowBalance starts from on-policy reasoning groups, evaluates terminal correctness, and uses a frozen privileged-hindsight view to score the same sampled tokens. These signals define a normalized target over complete responses while preserving reference support and avoiding a separate token-level imitation loss.

  • On-Policy Rollouts: Each prompt produces a group of responses sampled from the frozen rollout policy, with terminal verifier rewards converted into stopped group-relative advantages.The rollout policy is snapshot at each iteration, and rewards, group statistics, trajectories, and advantages receive no gradient.
  • Privileged Hindsight: A training-only context is scored by two frozen-policy views: the rollout view generates without context, while the privileged-hindsight view scores the sampled tokens afterward.The hindsight view generates no replacement trajectory, receives no gradient, and is unavailable to the deployed policy.
  • Role of Self-Guidance: The privileged-hindsight signal is a stopped dense feature for defining a trajectory-level target, not a separate token-level imitation objective.FlowBalance evaluates the same sampled tokens under the frozen hindsight view and uses the resulting signal only to shape the complete-response distribution.
  • Reference-Supported Target: FlowBalance represents the next policy as a normalized target over complete responses, with the reference fixing support and the stopped trajectory energy determining relative preference.The partition function normalizes the energy-weighted reference distribution, while temperature controls the target construction.

3 FlowBalance: Verifier-Grounded Self-Improvement

FlowBalance combines verifier direction with dense hindsight evidence in a stopped trajectory energy, then fits its partition-normalized complete-response target through trajectory balance. The partition absorbs a common offset while the energy controls relative preference among sampled responses.

  • Target Construction: FlowBalance combines verifier outcomes and dense hindsight scores into a trajectory-level energy that defines the complete-response target.The dense branch shapes which distribution is learned and is not optimized through a separate token-level imitation loss.
  • Target Construction: The clipped token-level hindsight gains are aggregated over the complete response into GH, a stopped feature computed on on-policy experience.No tokens are resampled from the hindsight view, and no gradient passes through πH or GH.
  • Outcome Calibration: Positive-advantage trajectories retain positive guidance, negative-advantage trajectories reverse it, and zero-advantage trajectories disable the dense branch.This sign gating prevents a confidently scored failure from becoming self-reinforcing supervision.
  • Target Construction: The reference policy retains support, the verifier advantage supplies outcome direction, and GH provides dense within-trajectory evidence in the realized rollout group.The response-space notation describes the ideal Gibbs distribution, while the objective and profiled partition are evaluated on sampled complete trajectories.
  • Trajectory Balance: Trajectory balance fits the normalized target by matching trainable-policy log probabilities to the stopped energy and partition terms.At zero residual, the partition cancels in pairwise probability ratios, leaving the energy to control relative preference while log Z absorbs the common offset.
  • Subtrajectory Balance: The same balance principle can be applied between intermediate states through continuation partitions and subtrajectory residuals.The complete-response implementation is used in the experiments unless otherwise stated.

4 Why FlowBalance Supports Verifier-Grounded Self-Improvement

The analysis characterizes FlowBalance as a conservative, verifier-controlled distributional update that preserves within-group evidence. It also shows that outcome calibration reverses false-positive guidance on rejected responses rather than reinforcing it.

  • Scope of Guarantees: The target-level guarantees are conditional on the realized rollout group and treat verifier and guidance quantities as stopped.They describe the induced target and local fitting objective, not guaranteed convergence of an arbitrary finite neural-network update in one step.
  • Distributional Structure: The profiled partition removes one common energy offset while preserving all N − 1 independent within-group probability contrasts.Thus the rollout group retains relative evidence from every sampled response rather than collapsing to a single comparison.
  • Distributional Structure: Among group distributions reaching the target’s expected composite energy, FlowBalance is the unique minimum-reverse-KL displacement from the reference.The reference is restricted and renormalized to the realized rollout group for this characterization.
  • Verifier Control: Increasing the verifier coefficient ηA monotonically shifts the target toward higher-reward responses when guidance scores and the rollout group are fixed.The verifier therefore remains an explicit control knob even in the presence of dense self-guidance.
  • Outcome Calibration: For a verified success and a rejected response in the same mixed-outcome group, sign gating converts positive guidance on the rejected response into a probability-ratio correction favoring the verified response.This correction addresses the case where privileged hindsight assigns positive local likelihood to both correct reasoning and plausible failures.

5 Experiments

Experiments evaluate FlowBalance across mathematical reasoning accuracy, training dynamics, coefficient ablations, and correct-strategy diversity. Across Qwen3-4B and Qwen3-8B, it improves aggregate performance over comparison methods while accelerating and stabilizing training and preserving diverse successful strategies.

  • Main results: 64.26 is FlowBalance’s five-benchmark average on Qwen3-4B, improving over GRPO by 1.95 points and FlowRL by 1.04 points.On Qwen3-8B, FlowBalance reaches 67.61, improving over GRPO by 2.12 points and FlowRL by 1.76 points.
  • Training dynamics: 1.43× faster: FlowBalance reaches 0.5 AIME24 validation accuracy in about 100 steps versus roughly 143 for GRPO.It remains near peak accuracy over 400 steps, while GRPO degrades sharply after approximately step 180; FlowBalance also avoids direct OPSD’s rapid response-length collapse.
  • Ablation study: The ablations vary verifier grounding ηA over {5, 10, 15} and self-guidance strength βG over {1, 2, 3}, using one-dimensional sweeps around the default setting.The default run uses ηA = 15 and βG = 1, with training stability tracked alongside accuracy.
  • Correct-strategy diversity: 0.2194: FlowBalance’s correct-only Simpson strategy diversity exceeds 0.1017 for GRPO and 0.1456 for RLSD on AIME24.The diagnostic is LLM-judged and measures semantic strategy diversity among correct trajectories; the case study shows differing mathematical representations, including box embedding versus Cayley–Menger.

6 Related Work

Related work frames FlowBalance as combining verifier-grounded reinforcement learning, privileged self-guidance, self-improvement loops, and trajectory-balance distribution matching. Its distinctive focus is the normalized complete-response distribution learned from sparse outcomes and dense training-time evidence.

  • Reinforcement learning from verifiable outcomes: RLVR provides reliable outcome grounding through automatically checkable answers, but its supervision remains response-level.FlowBalance addresses how sparse outcomes and dense training-time evidence should define a normalized distribution over complete responses.
  • Privileged self-guidance and on-policy distillation: Privileged self-guidance supplies dense token-level information from a more informed view of student-generated trajectories, but can shorten reasoning, suppress uncertainty, or leak unavailable information.FlowBalance uses this family of training-time evidence while addressing the risk of false self-confidence.
  • Guided self-improvement and self-evolving systems: Self-evolving systems generate, select, or schedule new learning experiences, whereas FlowBalance studies the inner-loop policy update given prompts and on-policy solution trajectories.The paper does not experimentally combine FlowBalance with R-Few; R-Few is presented as a possible outer-loop task supplier.
  • Distribution matching and trajectory balance: Trajectory-balance methods learn distributions over complete trajectories rather than concentrating only on maximizers; FlowBalance combines this framework with verifier advantages and calibrated privileged-hindsight guidance.Its comparison to FlowRL isolates the added guidance within the same broad trajectory-balance family.

7 Conclusions and Discussions

FlowBalance frames self-improvement as normalized distribution learning over on-policy reasoning experience, using verifier-calibrated self-guidance and trajectory balance. The method shows theoretical distributional guarantees and strong mathematical-reasoning results, but its scope remains limited.

  • Conclusions and Discussions: FlowBalance combines privileged-hindsight self-guidance, verifier-controlled direction, and profiled trajectory balance to fit a normalized distribution over complete responses.The policy is updated only through the normalized trajectory-balance objective, without a separate token-level imitation loss.
  • Conclusions and Discussions: The theory establishes within-group contrast preservation, minimum reverse-KL displacement, monotonic verifier control of target reward, and anti-self-confirmation for rejected trajectories.
  • Conclusions and Discussions: FlowBalance achieves the strongest five-benchmark average among GRPO, OPSD, RLSD, FlowRL, and FlowBalance on Qwen3-4B and Qwen3-8B.It also reaches the AIME24 validation threshold in fewer updates, remains stable during extended training, avoids direct OPSD’s response-length collapse, and shows higher correct-only semantic strategy diversity.
  • Limitations and scope: Large-scale experiments focus on mathematical reasoning, leaving generalization to agentic, multimodal, and other long-horizon domains open.The diversity diagnostic uses one checkpoint and seed, and the method does not itself generate or curate new tasks.

A Detailed-Balance View of FlowBalance

The detailed-balance view interprets FlowBalance’s normalized target as an equilibrium distribution whose relative response preferences are determined by reference-policy mass and verifier-grounded self-guidance energy. Trajectory balance fits that distribution directly rather than constructing a Markov transition kernel.

  • A Detailed-Balance View of FlowBalance: Detailed balance interprets the target as an equilibrium distribution whose forward and reverse probability flows are equal.A reversible kernel satisfying detailed balance has the target as its stationary distribution.
  • A Detailed-Balance View of FlowBalance: FlowBalance defines a Gibbs target for each fixed prompt–context pair, with the reference policy providing baseline mass and the energy exponentially reweighting responses.The partition function is necessary for normalization, although it cancels from pairwise probability ratios.
  • A Detailed-Balance View of FlowBalance: Trajectory balance directly imposes a global identity for each complete response instead of constructing or simulating the Markov kernel.
  • A Detailed-Balance View of FlowBalance: When the trajectory-balance residual is zero for all responses, the learned policy equals the normalized target and shares its pairwise probability ratios.Detailed balance supplies the equilibrium interpretation, while trajectory balance supplies the learning constraint.
  • A Detailed-Balance View of FlowBalance: The analysis conditions on a fixed prompt–context pair and realized rollout group, with the reference policy restricted and renormalized to that group for KL calculations.

B.1 Distributional Properties

The distributional analysis characterizes profiled trajectory balance through its within-group contrasts and reference displacement. Profiling removes one common group offset while preserving relative preferences, and the resulting target is the unique minimum reverse-KL displacement under the stated energy constraint.

  • B.1 Distributional Properties: When the target is representable, the global profiled loss is zero.
  • B.1 Distributional Properties: Profiling the trajectory-balance intercept preserves all N −1 within-group contrast directions while removing exactly one common group-level offset.Zero profiled loss is equivalent to matching every within-group target log probability-ratio contrast.
  • B.1 Distributional Properties: The profiled intercept is obtained by minimizing a convex quadratic loss in one scalar partition parameter.At the optimum, all trajectory-balance residuals vanish exactly when the within-group contrasts match.
  • B.1 Distributional Properties: The FlowBalance target uniquely minimizes reverse-KL displacement from the reference distribution among group distributions satisfying the expected-energy condition.

B.2 Effects of Verifier Weighting and Sign Gating

Verifier weighting strengthens the target’s preference for verified outcomes, while sign gating converts positive self-guidance on rejected responses into a correction favoring verified responses. Controlled diagnostics show substantial gains when guidance is useful but also identify a reliability boundary beyond which stronger guidance can reduce verified success.

  • B.2 Effects of Verifier Weighting and Sign Gating: Increasing the verifier coefficient monotonically increases expected reward under the FlowBalance target when self-guidance scores and the rollout group are fixed.
  • B.2 Effects of Verifier Weighting and Sign Gating: Sign gating converts positive privileged support on a verifier-rejected response from self-reinforcement pressure into a probability-ratio correction favoring the verified response.
  • B.2 Effects of Verifier Weighting and Sign Gating: With (η, β, τ) = (0.75, 0.80, 1), FlowBalance reaches success mass 0.900 and robust-success mass 0.440, versus 0.818 success mass for reward-only shaping.Ungated self-guidance reaches success mass 0.832 while also reinforcing positive guidance on failures.
  • B.2 Effects of Verifier Weighting and Sign Gating: At G− = 0.5, exact target success probabilities are 0.894 for FlowBalance, 0.817 for reward-only shaping, and 0.807 for ungated shaping.FlowBalance dominates both baselines throughout the useful-but-imperfect quadrant described by the diagnostic.
  • B.2 Effects of Verifier Weighting and Sign Gating: Below a self-guidance reliability threshold, stronger guidance can reduce verified success, so sign gating does not make systematically wrong guidance safe.The reliability parameter in the map is a synthetic interpolation rather than an empirical calibration estimate.

C.1.4 Structural View: Conservative Change and All-Contrast Efficiency

FlowBalance is presented as a conservative, statistically efficient distributional update: it follows a minimum reverse-KL path while profiled group intercepts preserve within-group contrasts. The experiments use matched training and evaluation controls to compare update objectives fairly.

  • Conservative change: The exponential-tilt path traces the minimum reverse-KL frontier as sign-gated energy increases.A fixed-data target matched to the exact FlowBalance energy requires reverse KL 0.973 versus 0.273 for FlowBalance, a 3.6× larger displacement from the reference.
  • All-contrast efficiency: Profiling one nuisance intercept per rollout group preserves all N −1 contrast directions and yields the exact local Gaussian risk reduction.These properties explain the method’s statistical efficiency in using each rollout group.
  • Experimental controls: The comparison controls hold prompts, rollout group size, response-length cap, verifier, optimizer schedule, checkpoint cadence, and evaluation script fixed within each backbone.Privileged training information is restricted to a frozen scoring path during training.
  • Training signal construction: FlowBalance combines verifier rewards, group-relative advantages, and frozen-snapshot privileged-hindsight scores into a sign-gated trajectory energy optimized through profiled trajectory balance.Positive, negative, and outcome-indifferent groups receive respectively retained, reversed, and zero self-guidance.
  • Baseline comparison: The baselines differ in update objective: GRPO uses standard reward-policy optimization, OPSD uses forward-KL distillation, RLSD combines verifier and teacher signals, and FlowRL uses outcome-only trajectory balance.FlowBalance instead fits a normalized trajectory target.

D.2 Grounding and Guidance Ablations

The ablation and diversity analyses examine coefficient sensitivity and semantic variation under explicitly controlled evaluation protocols. The diversity result is intentionally limited to a single-seed AIME24 diagnostic because LLM-based clustering is partly subjective.

  • Coefficient ablations: Coefficient sweeps vary ηA or βG one dimension at a time while holding the other coefficient and all remaining settings fixed.Each sweep point uses the same five-benchmark average and completed five-seed runs.
  • Coefficient ablations: The expanded ablation tables report Qwen3-8B step-180 results across five benchmarks, with Pass@16 for AIME24 and Pass@1 for the other tasks.The reported average is the unweighted mean of the five benchmark means.
  • Semantic diversity diagnostic: The diversity study decodes 16 complete responses for each of 30 problems and three methods at one checkpoint and one seed, producing 1440 trajectories for judging.No response is truncated or heuristically compressed before evaluation.
  • Semantic diversity diagnostic: A two-stage judge extracts each trajectory’s attempted strategy and then clusters anonymized, randomly ordered summaries without source-algorithm or correctness-label information.Correctness labels are applied only after clustering, and diversity is computed on correct trajectories.
  • Semantic diversity diagnostic: Correct-only Simpson strategy diversity measures the probability that two randomly sampled correct trajectories use different semantic solution strategies.A problem enters the aggregate only when it has at least two correct trajectories.
  • Scope: The diversity conclusion is restricted to the AIME24 seed-0 diagnostic because LLM-based clustering remains partly subjective.It does not constitute a multi-seed stability estimate.
  • Semantic diversity diagnostic: FlowBalance’s additional correct trajectories include distinct mathematical backbones, such as envelope versus multiplicity criteria and two-dimensional versus three-dimensional geometric treatments.The case studies are intended to show meaningful strategy differences rather than merely longer or differently worded traces.
Loading 2609.03241v1…