PaperlayerView exact revision on arXiv

Source-linked AI summary

RLFR: Extending Reinforcement Learning for LLMs with Flow Environment

Jinghao Zhang, Naishan Zheng, Ruilin Li, Dongzhou Cheng, Zheming Liang, Feng Zhao, Jiaqi Wang

arXiv:2510.10201v1cs.LGcs.AIcs.CL

TL;DR

Binary verification can overlook promising reasoning exploration, while process rewards require costly annotation and logit-space auxiliary signals may be unreliable for optimization. RLFR uses velocity deviations in latent-space flow fields built from expert or rejection-sampled data as rewards. The paper reports stable training and consistent gains over baseline RLVR across language and multimodal reasoning benchmarks.

  • Problem

    Binary-verification RLVR can miss useful reasoning trajectories when responses end with incorrect answers, while PRMs require costly intermediate-step annotation.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    RLFR constructs latent flow fields from off-policy high-quality or on-policy rejection-sampling data and uses policy-latent velocity deviations as reward signals.

    Source: §1 · p. 2

  • Results

    RLFR reports stable training gains without degeneration and consistent progress over baseline RLVR across language and multimodal reasoning benchmarks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The results highlight latent space as an underexplored substrate for reward design and show that off-policy expert trajectories can contribute to reward signals.

    Source: §6 · p. 10

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a promising framework for improving reasoning abilities in Large Language Models (LLMs). However, policy optimized with binary verification prone to overlook potential valuable exploration in reasoning trajectory. In view of heavy annotation cost of golden Process Reward Models (PRMs), recent works attempt using auxiliary signals for reward shaping of process tokens, involving entropy and likelihood collected from logit space. In this work, we offer a novel perspective on shaping RLVR with flow rewards derived from latent space, and propose RLFR, where the flow fields of model latents are constructed from either off-policy high-quality data and on-policy rejection sampling data, and the velocity deviations of policy latents within it are quantified to serve as a reward signal. RLFR first demonstrates that a well-established flow field can be a sound environment for reward signal collection, highlighting the expressive latent space is much underexplored. Moreover, RLFR is able to compress any off-policy expert data as reference for constituting reward signals, and we show that the efficient context dependence compressed within the hidden states are utilized, rather than individual token-level denotation for context comprehending. Experiments on both language and multimodal reasoning benchmarks demonstrate the reliability of flow rewards, and suggesting a promising paradigm for reward shaping with auxiliary signals.

1 INTRODUCTION

Binary outcome verification can miss valuable reasoning exploration, while process rewards are costly to annotate and logit-derived confidence signals may be unreliable for optimization. RLFR shapes RLVR with latent-space flow rewards and reports stable gains across language and multimodal reasoning benchmarks.

1 INTRODUCTION

  • Binary verification can overlook promising reasoning trajectories when a final answer is incorrect, despite RLVR’s robustness against reward hacking.The introduction also notes that effective credit assignment over binary outcome rewards remains underexplored.Source: §1 · p. 2
  • PRMs face costly intermediate-step annotation and gaps between training corpora and online trajectories, while logit-space confidence signals may be exploited during optimization.The cited auxiliary signals include model likelihood and token entropy.Source: §1 · p. 2
  • RLFR builds latent flow fields from off-policy high-quality or on-policy rejection-sampling data and rewards policy latents according to their velocity deviations.Larger deviations are penalized and smaller deviations encouraged; fields are updated online with filtered rejection-sampling data.Source: §1 · p. 2
  • RLFR shows consistent progress over baseline RLVR methods across language and multimodal reasoning benchmarks, with stable training gains and no sign of degeneration.The experiments span Qwen and Llama models.Source: §1 · pp. 2–3

2 PRELIMINARIES

The preliminaries define binary-verification RLVR and introduce flow matching as a way to model latent distributions with a learned velocity field. They also motivate denser rewards by describing the limits of outcome rewards and the distinct signals visible in latent-space trajectories.

2.1 REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS

  • RLVR trains a language model to maximize expected binary-verifier reward over prompt-response pairs, but coarse outcome rewards can miss useful exploration in incorrect responses.The prompt dataset includes corresponding ground-truth answers.Source: §2.1 · p. 3
  • GRPO samples a response group per prompt, uses average reward as the advantage baseline, and retains PPO’s clipped surrogate objective.The resulting advantage is shared across tokens in each response.Source: §2.1 · p. 3
  • Reward shaping modifies environment rewards with proxy rewards or auxiliary signals such as response length and token entropy.The paper notes these signals can steer desired behavior more effectively.Source: §2.1 · p. 3

2.2 FLOW MATCHING

  • Flow matching learns a continuous-time velocity field that transports samples from a simple prior toward the target data distribution.It trains velocity prediction on linear interpolations between prior and data samples.Source: §2.2 · pp. 3–4
  • In 256 MATH rollouts, latent-space tail-token distributions show progressively expressive signals, unlike logit distributions and textual trajectory clouds.The figure describes the latent signals as arising through interaction with preceding tokens for context compression.Source: Figure 3 · p. 4

3 RLFR

RLFR uses flow fields over model latents to shape process-token rewards in RLVR. This section motivates latent signals, defines velocity-deviation rewards, and extends them through token-level advantages and online flow updates.

3.1 ANALYZING LATENT SPACE IN REASONING TRAJECTORY

  • Tail-token latents distinguish answer-correctness trajectories more clearly than head-token latents, while likelihood and textual clouds show no noticeable signal.Some incorrect-answer latents remain close to correct trajectories, motivating tolerance for minor deviations from high-quality latents.Source: §3.1 · p. 4

3.2 FLOW REWARD FROM VELOCITY DEVIATION

  • Velocity-prediction deviation under a reference flow trained on high-quality data provides a metric for evaluating whether token latents fit its reference distribution.The flow network is frozen for reward calculation after pretraining.Source: §3.2 · p. 5
  • Larger, less noisy timesteps are more favorable for flow-reward construction, and score-based drift directions are proposed for evaluating deviation.The authors describe score directions as more locally accurate than the global-consistent direction from velocity prediction.Source: §3.2 · p. 5
  • Negative velocity deviation forms an evidence lower bound on reference-flow log-likelihood, so the two measures are inversely correlated.The minimum velocity deviation for a latent corresponds to the maximal evidence lower bound on its log-likelihood.Source: §3.2 · p. 5

3.3 EXTENDING RLVR WITH FLOW REWARD

  • RLFR shapes each token’s advantage with accumulated, decayed flow rewards, a formulation the authors say is flexible across RL algorithms.The method does not require a specific advantage-estimation method.Source: §3.3 · p. 5
  • Substantial deviations above η are retained, sequence-normalized to [−1, 1], and evaluated with subsequent-token latents as context.The authors describe the context conditioning as establishing context dependence through an enlarged interactive space.Source: §3.3 · p. 6
  • Because policy latents shift during optimization, RLFR updates the flow using online rejection-sampled data; correctness is the most effective filtering metric reported.The frozen offline flow can otherwise introduce a distribution gap.Source: §3.3 · p. 6

4 EXPERIMENTS

Across language and multimodal reasoning benchmarks, RLFR improves over baseline RLVR and remains competitive with logit-space entropy shaping. Ablations and reward analysis indicate that stable flow rewards depend on reference-data choices, timestep debiasing, and context-sensitive latent representations.

4.1 EXPERIMENTAL SETUP

  • Flow pretraining uses 93k OpenR1 math problems for language and 115k MMPR multimodal problems, with reinforcement learning on MATH and MMK12.Language evaluation includes AIME24/25, AMC23, MATH500, and OlympiadBench; multimodal evaluation includes four math benchmarks and LogicVista and VisuLogic.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • RLFR beats basic RLVR by 1.5% average score on Qwen2.5-Math-7B and 5.3% average score on Llama3.1-8B, with consistent gains across language and multimodal benchmarks.It also outperforms entropy-based shaping on language benchmarks and shows steady generalization on out-of-domain multimodal logic benchmarks.Source: §4.2 · pp. 7–8, Table 1 · p. 7, Table 2 · p. 7

4.3 ABLATIONS

  • Offline flow initialization and online rejection-sampling updates each contribute steadily, while filtering substantial reward deviations is essential for stable training.Outcome correctness is the most effective of the tested metrics for filtering reference data.Source: §4.3 · p. 8, Table 4 · p. 8, Table 5 · p. 8
  • With timesteps T = {0.2, 0.4, 0.6, 0.8}, debiasing improves average score by 1.7% over equal averaging; larger, less noisy timesteps yield more reliable velocity predictions.Figure 5 and Table 6 report the timestep comparison.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Post-token conditioning outperforms identity-token and previous-token conditions for velocity prediction, supporting the use of a larger auxiliary space.The comparison is reported in Table 6.Source: §4.3 · p. 8, Table 6 · p. 8

4.4 ANALYSIS

  • Flow rewards favor tokens that execute the question over empty connection tokens, while high-entropy states produce larger latent velocity deviations because they represent many candidate tokens.Rewarded tokens initially relate to offline-start data, including sqrt and angle, and are progressively updated with the policy.Source: Figure 4 · p. 8, §4.4 · p. 9
  • Flow rewards use context dependence compressed in hidden states rather than token-level denotation alone; a token such as frac can receive either reward sign by context.The reward-token case study is presented in Table 3.Source: Table 3 · p. 8, §4.4 · p. 9

5 RELATED WORK

Prior work shapes binary-verification rewards mainly with logit-space signals or uses flow as the policy model. RLFR instead treats flow as an environment for collecting rewards from latent velocity deviations.

Reinforcement learning beyond binary verification

  • Logit-space likelihood and entropy signals have been used to shape rewards or identify tokens for different optimization, while latent space remains comparatively underexplored.The discussion also notes pass@k training as an orthogonal direction that tolerates incorrect answers with potentially valuable trajectories.Source: §5 · p. 9

Flow Matching in Reinforcement Learning

  • Flow-based work in visual generation and robotics uses flow as the policy model, unlike RLFR's use of flow as a reward-collection environment based on velocity deviations.RLFR does not reverse the flow process for policy optimization.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents RLFR as latent-space reward shaping for RLVR, using flow fields built from off-policy expert data or on-policy rejection sampling. It reports steady gains across language and multimodal benchmarks and identifies scaling the flow environment and latent signals for test-time scaling as future directions.

6 CONCLUSION

  • RLFR uses deviations of policy latents within flow fields, built from off-policy high-quality data or on-policy rejection-sampling data, as reward signals for RLVR.The conclusion characterizes latent space as expressive but underexplored, complementing methods focused on logit space.Source: §6 · pp. 9–10

A.1 SETTINGS

  • The appendix reports training on 8 H20 GPUs with AdamW and eight rollouts per prompt, using bbox prompts for language and tag prompts for multimodal training.The prompt template places the question between system instructions to reason step by step and an assistant response; the training-prompt table is identified as Table 7.Source: §A.1 · p. 15, Table 7 · p. 15

A.2 TRAINING LOGS

Training logs show RLFR steadily improves reasoning performance and accelerates policy optimization relative to RLVR, while maintaining slightly higher policy entropy at the training plateau. Response lengths increase and adjust steadily, with no observed degeneration.

A.2 TRAINING LOGS

  • RLFR training logs show steady reasoning gains and faster policy optimization, slightly higher plateau entropy than RLVR, and no response-length degeneration.Response length increases steadily before adjusting during training.Source: §A.2 · p. 15, Figure 6 · p. 15

B EXTENDED ANALYSIS OF LATENT SPACE

The analysis finds coherent latent signals for trajectory-quality identification across layer positions and uses intermediate hidden-state percentiles for reward collection. The final percentile is excluded because the last hidden states are heavily modulated by the language-model head.

B EXTENDED ANALYSIS OF LATENT SPACE

C.1 DERIVATION OF SCORE FUNCTION FOR VELOCITY

This derivation relates the score function to the velocity field, using a linear interpolation schedule to establish their equivalence.

C.1 DERIVATION OF SCORE FUNCTION FOR VELOCITY

  • The derivation assumes independent endpoints and notes that the noisy variable is Gaussian when conditioned on the data endpoint.Source: §C.1 · p. 16
  • Fisher’s identity expresses the marginal score as an expectation of conditional scores given the noisy variable.Source: §C.1 · p. 16
  • Under the linear interpolation schedule, the score function and velocity field are exactly equivalent.Source: §C.1 · p. 17

C.2 PROOF OF VARIATIONAL LOWER BOUND

The appendix connects flow-matching velocity deviations to likelihood under a target distribution, then includes a lottery conditional-probability case study.

C.2 PROOF OF VARIATIONAL LOWER BOUND

  • The proof connects velocity deviations measured by the flow-matching objective with likelihood under the target distribution, with the special case λ = 1 yielding the stated equivalence.Source: §C.2 · pp. 17–18
  • The proof derives the likelihood relation using the flow’s continuity equation and change-of-variables formula, with the final step invoking Stein’s identity.Source: §C.2 · pp. 17–18

D CASE STUDY

  • The case study presents a lottery conditional-probability question whose stated ground-truth value for m + n is 116.Source: D CASE STUDY · p. 18

Related papers

Loading 2510.10201v1…