PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Latent Reasoning via Reinforcement Learning

Zhenrui Yue, Bowen Jin, Huimin Zeng, Honglei Zhuang, Zhen Qin, Jinsung Yoon, Lanyu Shang, Jiawei Han, Dong Wang

arXiv:2505.18454v2cs.CL

TL;DR

Latent reasoning can use hidden states for internal computation, but existing approaches may depend on CoT traces and can conflict with LLM generation. HRPO uses reinforcement learning and a learnable gate to blend sampled tokens with hidden features, and the authors report gains across knowledge and reasoning benchmarks.

  • Problem

    Latent reasoning approaches often rely on CoT traces and can impair generation when hidden states are fed into subsequent decoding steps.

    Source: §1 · pp. 1–2

  • Method

    HRPO uses reinforcement learning and a learnable gate to progressively blend sampled token embeddings with prior hidden representations.

    Source: §1 · p. 2

  • Results

    HRPO outperforms SFT and RL baselines with consistent gains across knowledge and STEM benchmarks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    HRPO-trained models exhibit intriguing reasoning patterns, offering insights into RL-based continuous-space learning.

    Source: §1 · p. 3, §5 · p. 10

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have introduced latent reasoning as a promising alternative to autoregressive reasoning. By performing internal computation with hidden states from previous steps, latent reasoning benefit from more informative features rather than sampling a discrete chain-of-thought (CoT) path. Yet latent reasoning approaches are often incompatible with LLMs, as their continuous paradigm conflicts with the discrete nature of autoregressive generation. Moreover, these methods rely on CoT traces for training and thus fail to exploit the inherent reasoning patterns of LLMs. In this work, we explore latent reasoning by leveraging the intrinsic capabilities of LLMs via reinforcement learning (RL). To this end, we introduce hybrid reasoning policy optimization (HRPO), an RL-based hybrid latent reasoning approach that (1) integrates prior hidden states into sampled tokens with a learnable gating mechanism, and (2) initializes training with predominantly token embeddings while progressively incorporating more hidden features. This design maintains LLMs' generative capabilities and incentivizes hybrid reasoning using both discrete and continuous representations. In addition, the hybrid HRPO introduces stochasticity into latent reasoning via token sampling, thereby enabling RL-based optimization without requiring CoT trajectories. Extensive evaluations across diverse benchmarks show that HRPO outperforms prior methods in both knowledge- and reasoning-intensive tasks. Furthermore, HRPO-trained LLMs remain interpretable and exhibit intriguing behaviors like cross-lingual patterns and shorter completion lengths, highlighting the potential of our RL-based approach and offer insights for future work in latent reasoning.

1 Introduction

The introduction motivates latent reasoning as a way to use hidden-state features, but notes that current approaches often require CoT traces or costly training and can impair generation. HRPO combines sampled tokens and hidden representations through reinforcement learning, and is reported to outperform existing models across knowledge and reasoning benchmarks.

1 Introduction

  • Latent reasoning uses continuous hidden representations from prior steps, unlike discrete autoregressive chain-of-thought generation.Coconut feeds the last hidden state back as input embeddings for the next reasoning step [11].Source: §1 · p. 1
  • Existing latent reasoning methods often depend on extensive CoT traces or costly, tailored training, and may overlook pretrained LLMs’ reasoning capabilities.Directly feeding hidden states into the next decoding step can degrade generation quality, including through repetition or incoherence.Source: §1 · pp. 1–2
  • HRPO uses reinforcement learning to combine sampled token embeddings with prior hidden states through a learnable gate, without requiring CoT trajectories.The gate starts with token-dominant inputs and progressively incorporates hidden features during training.Source: §1 · p. 2
  • Token sampling makes HRPO rollouts stochastic, allowing policy updates from hybrid trajectories using an outcome-based reward.The hybrid rollout buffer stores tokens and latent representations for calculating log probabilities.Source: §1 · p. 2
  • The authors present HRPO as a training-efficient, scalable alternative that mitigates reliance on CoT annotations and expensive multi-stage training.Source: §1 · p. 2
  • HRPO outperforms existing models and latent reasoning baselines across multiple knowledge and reasoning benchmarks.The introduction reports consistent gains across diverse scenarios.Source: §1 · p. 2

2 Related Work

Related work covers latent representations for reasoning and language modeling, alongside methods that use extra computation or internalize explicit reasoning traces. It also situates HRPO among online reinforcement-learning approaches that optimize language-model policies.

2.1 Latent Reasoning

  • Prior latent-reasoning work analyzes transformer features, uses latent representations for in-context example selection, and adds latent variables or special tokens for internal computation [2] [47] [45] [52] [8] [20] [9] [29].Source: §2.1 · p. 3
  • Implicit-reasoning methods convert explicit token-level trajectories into internal reasoning; CODI uses self-distillation to align explicit and implicit tokens [6] [7] [35].The authors contrast HRPO’s reinforcement-learning approach with methods that require processed traces and extensive training.Source: §2.1 · p. 3

2.2 Reinforcement Learning

  • Online RL methods for language models include policy-gradient algorithms such as REINFORCE, actor-critic A2C, PPO [32], and REINFORCE-style RLOO [1], GRPO [33], and REINFORCE++ [16].Source: §2.2 · p. 3
  • The authors position their approach as online RL that progressively incorporates hidden states into LLM inputs to encourage hybrid latent reasoning.Source: §2.2 · p. 3

3 Methodology

Section 3 combines sampled-token embeddings with projected hidden features through a learnable gate, then trains the hybrid policy with reinforcement learning over hybrid rollouts.

3.1 Hybrid Reasoning with Gating

  • Hidden states are projected into the model’s embedding space through a probability-weighted interpolation of token embeddings, preserving differentiability and aligning inputs with the model’s native space.The interpolation uses output probabilities and is normalized to preserve the output vector’s scale and variance.Source: §3.1 · p. 4
  • A learnable gate blends sampled-token embeddings with projected hidden states, initially favoring tokens before progressively incorporating latent features.Hybrid reasoning applies during the reasoning phase; the final answer uses standard autoregressive decoding.Source: §3.1 · pp. 4–5

3.2 Hybrid Reasoning Policy Optimization (HRPO)

  • HRPO updates the policy from hybrid rollouts using outcome rewards on discrete answer tokens, within-group standardized advantages, and KL-regularized policy gradients.The rollout buffer supplies the policy-update loss.Source: Figure 2 · p. 4, §3.2 · p. 5
  • Each hybrid trajectory is used for only one gradient update because its hidden representations depend directly on the current policy parameters, keeping HRPO strictly on-policy.The authors state that reusing trajectories, even with importance sampling, violates on-policy constraints.Source: §3.2 · p. 5

4 Experiments

The experiments evaluate HRPO on knowledge and STEM benchmarks, then analyze its latent-reasoning behavior and sensitivity to initialization and temperature.

4.1 Evaluation on Knowledge Benchmarks

  • HRPO achieves the strongest exact-match results across the knowledge benchmarks with smaller Qwen models, rivaling larger 7B baselines.Qwen2.5-3B reaches 0.380 EM and exceeds the strongest 7B RAG baseline by 4.5%; the 1.5B model averages 0.337 and surpasses PPO by 3.0%.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Evaluation on STEM Benchmarks

  • Across five STEM benchmarks, HRPO delivers the strongest results with compact Qwen backbones and can match much larger LLMs.The 3B model averages 0.700 accuracy, matching the best 7B baseline on four datasets; the 1.5B model averages 0.617 and exceeds the 7B leader on MATH by 2.0%.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Analysis of HRPO

  • Hidden ratios rise throughout training, while completion lengths first increase and then decline, with the largest drops under HRPO.With rmin = 0.95, completion lengths decrease substantially, a behavior not observed for the other variants.Source: §4.3 · p. 8, Figure 4 · p. 8
  • Knowledge-task performance improves as rmin decreases, whereas STEM performance is higher at lower or higher rmin than in the intermediate range.For knowledge tasks, average performance peaks at rmin = 0.98 and most individual datasets peak at 0.95; STEM performance drops in the [0.98, 0.999] range.Source: Table 3 · p. 8, §4.3 · p. 9
  • On MMLU, τ = 0.3 and τ = 0.5 converge faster and reach higher reward plateaus than τ ≥ 0.7, while completion length varies noticeably across temperatures.Both smaller and larger temperatures shorten completions, whereas τ = 0.5 and τ = 0.7 maintain longer generations.Source: §4.3 · p. 9, Figure 5 · p. 9
  • HRPO produces readable reasoning trajectories without CoT supervision and sometimes mixes languages or gives compact responses to simple and factual queries.The paper associates shorter decoding for these queries with richer context encoded in hidden representations.Source: §4.3 · pp. 9–10, Figure 6 · p. 9

5 Conclusion

The conclusion presents HRPO as an RL framework combining token sampling and continuous latent representations, with gains across knowledge and STEM benchmarks. It also identifies computational overhead, on-policy training efficiency, and transparency as limitations.

5 Conclusion

  • HRPO combines discrete token sampling with continuous latent representations through a learnable gate that gradually incorporates hidden features.The framework is intended to encourage hybrid reasoning while retaining language-model generative capabilities.Source: §5 · p. 10
  • Evaluations on knowledge and STEM benchmarks show HRPO outperforming SFT and RL baselines with gains across diverse scenarios.The conclusion also reports stable hybrid latent reasoning and intriguing reasoning patterns.Source: §5 · p. 10
  • HRPO adds computation overhead, its on-policy design may reduce large-scale training efficiency, and its continuous representations can be less transparent.The authors propose simpler designs, off-policy extensions, and advanced latent reasoning techniques as future directions.Source: §5 · p. 10

A Implementation

The implementation describes gate initialization and training settings, prompt formatting, and evaluation procedures for knowledge and reasoning tasks.

A Implementation

  • The gate is initialized so exp(−c · softplus(Λ)) is drawn uniformly from [rmin, 0.999], with c fixed at 8.A larger rmin increases the initial proportion of sampled token embeddings; the authors tune temperature τ and report that HRPO is robust across a wide range.Source: A Implementation · p. 14
  • Training uses optimized kernel implementations and LoRA, with hyperparameters selected empirically for efficiency and performance.The authors report that the lightweight, optimized framework can run on a single GPU across diverse tasks.Source: A Implementation · p. 14
  • Prompts direct the model to reason internally before answering, with #### separating the reasoning span from the final answer.The delimiter is a single token; repeated occurrences receive zero reward to discourage early termination of hybrid reasoning.Source: A Implementation · pp. 14–15
  • Knowledge-task queries use top-three Wikipedia passages retrieved with E5 embeddings, while evaluation uses greedy decoding and exact match scores.The passage also specifies post-processing and scoring for mathematical and multiple-choice datasets.Source: A Implementation · p. 15

B Additional Results

HRPO outperforms prior latent-reasoning baselines on GSM8K and MATH, while additional analyses examine initialization and interpolation settings, training dynamics, and statistical significance. Smaller initial rmin generally helps, though its effects are less pronounced with the 3B backbone.

Comparison to Latent Reasoning Methods

  • HRPO achieves the best accuracy on GSM8K and MATH, gaining 9.42% and 23.63% respectively over the best latent-reasoning baseline, CODI.It also improves on distilled CoT from the larger QwQ model on both datasets.Source: B Additional Results · p. 16, Table 6 · p. 16

Sensitivity Analysis for Λ and τ

  • Smaller initial rmin generally outperforms larger settings, while performance drops slightly for intermediate values in [0.98, 0.999]; τ values from 0.5–0.7 offer a reliable efficiency–accuracy balance.Performance often peaks around that τ interval for both backbone models and remains stable with only minor fluctuations at the extremes.Source: B Additional Results · pp. 16–17, Table 7 · p. 17, Table 8 · p. 17

Additional Analysis for Λ Initialization

  • For Qwen 3B, a smaller initial rmin generally accelerates convergence and shortens completions, but these benefits are less pronounced than with Qwen 1.5B.The authors attribute the weaker effect to fewer update steps and trainable parameters; training remains stable and preserves generative capabilities across rmin values.Source: B Additional Results · pp. 17–18, Figure 13 · p. 19, Figure 14 · p. 19

Statistical Significance Analysis

  • HRPO gains over PPO and GRPO are statistically significant in four of five Qwen 3B comparisons, with an average gain of +1.4%.For Qwen 1.5B, the gains are significant in three of five tasks.Source: B Additional Results · p. 18, Table 9 · p. 18

C Qualitative Analysis

HRPO-generated correct reasoning includes coherent English, rare tokens, cross-lingual mixtures, and uncommon reasoning steps. The examples also reveal errors involving knowledge limits, formatting, repetition, and generation-length truncation.

Correct reasoning examples

  • Correct HRPO reasoning ranges from coherent English to rare-token use, English–Chinese mixtures, and uncommon tokens or atypical steps that still reach correct answers.The latter three patterns are reported as more prevalent in HRPO-trained models than in standard reasoning LLMs.Source: C Qualitative Analysis · p. 19, Figure 17 · p. 20, Figure 18 · p. 20

Mistaken reasoning examples

  • Observed errors include cross-lingual mistakes from limited knowledge, correct answers rejected for format violations, repetitive loops, and reasoning truncated by length limits.Formatting failures receive zero score; loops can continue until the maximum completion length.Source: C Qualitative Analysis · p. 19, Figure 22 · p. 21, Figure 23 · p. 22
  • The authors identify format rewards, longer conservative optimization, increased model parameters, and longer context or generation limits as possible improvement directions.These are proposed responses to the observed error patterns, not demonstrated fixes.Source: C Qualitative Analysis · p. 19
Loading 2505.18454v2…