PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaSR: Adaptive Streaming Reasoning with Hierarchical Relative Policy Optimization

Junlong Tong, Wenqi Xu, Yingqi Fan, Anhao Zhao, Xuan Lu, Yang Tan, Xiaoyu Shen

arXiv:2606.14694v2cs.CL

TL;DR

Dynamic streams require models to reason under partial observations, but existing streaming methods largely imitate costly fixed trajectories. AdaSR learns adaptive streaming and final reasoning with HRPO, achieving a better balance of accuracy, efficiency, and latency than supervised fine-tuning baselines.

  • Problem

    Streaming inputs require reasoning before the complete context arrives, while existing methods rely on costly fixed trajectories that do not learn adaptive reasoning decisions.

  • Method

    AdaSR uses reinforcement learning with HRPO to assign stage-aware advantages across streaming, deep-reasoning, and full-rollout tokens while rewarding format, accuracy, and adaptive thinking.

  • Results

    AdaSR achieves a better balance among reasoning accuracy, computational efficiency, and streaming latency than supervised fine-tuning baselines, with Qwen3-4B reducing total length by 6.9%–13.4% while increasing accuracy.

  • Takeaways & Limitations

    AdaSR indicates that adaptive computation can make streaming reasoning more flexible, efficient, and responsive for dynamically evolving inputs.

  • Takeaways & Limitations

    The study focuses on text streams with verifiable answers, leaving continuous audio, video, and open-ended interactive streams for future work.

Abstract

from arXiv · show

Large reasoning models typically follow a read-then-think paradigm: they observe the complete input, reason over a static context, and then produce the answer. Yet many real-world scenarios are inherently dynamic, such as audio and video stream, where information arrives as a continuous stream and models must reason, update, and respond under partial observations. Recent streaming reasoning methods allow models to think while reading, but they largely rely on supervised imitation of pre-constructed trajectories, which limits their flexibility. In this paper, we propose AdaSR, an adaptive streaming reasoning framework that enables models to reason during input streaming and perform final deliberation once the stream is complete, learning when to think, and how much computation to allocate across different stages. To optimize this hierarchical reasoning process, we introduce Hierarchical Relative Policy Optimization (HRPO), which decomposes policy optimization into streaming reasoning and deep reasoning phases, providing more fine-grained advantage assignment instead of uniformly distributing a single sequence-level advantage over all tokens. HRPO integrates format, accuracy, and adaptive thinking rewards to enforce valid reasoning protocols, preserve final task performance, and encourage latency-aware computation allocation. Experiments show that AdaSR achieves a better balance among reasoning accuracy, computational efficiency, and streaming latency compared with supervised fine-tuning baseline. We release our code at https://github.com/EIT-NLP/StreamingLLM/tree/main/AdaSR.

1 Introduction

AdaSR addresses the mismatch between static-context reasoning and dynamic streaming inputs by learning when to reason, when to skip, and how to allocate computation across streaming and final deliberation. Its HRPO objective assigns hierarchy-aware credit across reasoning stages, combining format, accuracy, and adaptive-thinking rewards.

  • Motivation: Large reasoning models typically use a read-then-think paradigm, beginning chain-of-thought only after observing a complete static context.This paradigm supports mathematical reasoning, code generation, and multi-step decision making.
  • Motivation: Dynamic environments instead provide continuous streams, requiring models to reason as speech, video, or observations unfold.Streaming inputs may reveal information incrementally rather than as complete contexts.
  • Motivation: Existing streaming chain-of-thought methods typically rely on costly supervised fine-tuning over carefully constructed trajectories with local reasoning annotations.The introduction identifies fine-grained supervision as costly and imitation learning as limiting flexibility.
  • AdaSR: AdaSR uses reinforcement learning to learn when to reason, when to skip, and how to balance streaming thinking with final deliberation over evolving inputs.The framework allocates computation adaptively during streaming and performs final deep thinking after the stream.
  • HRPO: HRPO replaces flat sequence-level credit with distinct advantages for streaming tokens, deep-reasoning tokens, and the full rollout.It preserves group-relative optimization while aligning credit assignment with the temporal structure of streaming reasoning.
  • HRPO: Format, accuracy, and adaptive thinking rewards jointly optimize valid reasoning protocols, final task performance, and the computation path.HRPO provides local-to-global, hierarchy-aware credit signals for structured streaming trajectories.

2 Preliminary

This section contrasts standard read-then-think reasoning with streaming thinking, which reasons under partial observations before final deliberation. It also introduces GRPO and identifies its uniform sequence-level credit assignment as too coarse for streaming reasoning.

  • Streaming Thinking: Standard reasoning observes the full context before generating reasoning segments associated with the complete input.The context is decomposed into sequential sentences, but reasoning follows full-context observation.
  • Streaming Thinking: Streaming thinking emits local thoughts after arriving sentences under partial context, optionally skipping irrelevant input, then performs final deep deliberation.Boundary and end-of-thought tokens separate local segments, with phase boundary ts marking the transition.
  • GRPO for Reasoning: GRPO compares rewards among grouped candidate outputs to form relative advantages, while retaining PPO-style clipping and KL regularization without a value model.The clipping threshold ε controls trust-region behavior, while β controls KL regularization strength.
  • GRPO for Reasoning: GRPO attaches the same sequence-level advantage to every token, treating outputs as flat trajectories and obscuring credit between streaming thoughts and final deliberation.This is suitable for batch reasoning but too coarse when local thoughts use partial observations and final deliberation uses complete context.

3 Methodology

AdaSR combines streaming rollout, adaptive rewards, and HRPO to allocate computation between incremental streaming reasoning and post-stream deep reasoning. HRPO assigns phase-specific and trajectory-level learning signals, while rewards enforce valid structure, final correctness, and latency-aware efficiency.

  • Framework Overview: AdaSR is an RL framework combining HRPO, adaptive rewards, and streaming rollout for adaptive computation allocation between streaming and deep reasoning.The streaming phase reasons under partial observations, while the deep phase integrates the full context after streaming.
  • Hierarchical Relative Policy Optimization: HRPO replaces GRPO’s uniform trajectory advantage with streaming-local, deep-local, and trajectory-global advantages for hierarchical temporal credit assignment.Local advantages shape phase-specific behavior, while the global advantage preserves answer-level correctness across the full rollout.
  • Adaptive Rewards: The format reward requires each streaming segment to end with <EOT> and valid thought or <skip> content, while deep reasoning must be non-empty and end with <EOR>.This reward enforces structural validity rather than reasoning quality.
  • Adaptive Rewards: The accuracy reward supplies a terminal signal by rewarding predictions that match the ground-truth answer after both streaming and deep reasoning.It anchors learning to final task correctness.
  • Adaptive Rewards: The adaptive thinking reward uses token length to discourage excessive computation while permitting additional reasoning when useful, with phase-aware latency discounts and success-conditioned efficiency.Length shaping applies only to correct, parseable trajectories, and deep reasoning receives full post-stream latency cost while streaming can overlap input arrival.
  • Streaming Rollout: Streaming rollout feeds input sentences sequentially, decodes reasoning until <EOT>, and uses independent position indices plus streaming attention masks during HRPO training.Collected rewards are normalized into component advantages, composed at temporal levels, assigned to token ranges, and optimized with the HRPO loss.

4 Experiments

AdaSR is evaluated with Qwen3 models across mathematical, context-based question-answering, and logical-reasoning tasks. The evaluation spans in-domain and out-of-domain datasets, while the training procedure uses HRPO with hierarchical token advantages.

  • Experimental Setup: AdaSR is evaluated with Qwen3 series models on mathematical reasoning, context-based question answering, and logical reasoning tasks.The setup covers GSM-Symbolic, MetaMathQA, PubMedQA, GSM-Infinite, and LogicNLI.
  • Experimental Setup: In-domain evaluation uses GSM-Symbolic, MetaMathQA, and PubMedQA, while out-of-domain evaluation uses GSM-Infinite and LogicNLI.The datasets contain sufficiently long problem states.
  • Training Procedure: AdaSR training computes reward components for each rollout, assigns hierarchical token advantages, and optimizes JHRPO using a time-grouped form.These operations are specified in Algorithm 1 with references to Eqs. 4, 5, and 10.

18 end

AdaSR-HRPO improves streaming-reasoning accuracy and efficiency by assigning credit across streaming and deep stages, while reallocating computation to reduce delayed deliberation and serving latency. Stage-level assignment and format- and length-aware rewards outperform coarser or finer alternatives across in-domain and out-of-domain evaluations.

  • Main Experiment: Compared with SFT-based StreamingThinker, HRPO improves accuracy on both Qwen3-1.7B and Qwen3-4B, including +22.7% on GSM-symbolic P2 and +20.1% on MetaMathQA.These gains indicate that supervised streaming trajectories alone are insufficient for robust reasoning under partial observations.
  • Latency-Aware Computation: On GSM-symbolic P2 with Qwen3-1.7B, HRPO reduces deep-stage reasoning from 1866.474 to 160.210 tokens while increasing accuracy from 0.424 to 0.788.AdaSR shifts computation into streaming reasoning and keeps final deliberation compact rather than postponing all reasoning until the input ends.
  • Influence of Hierarchical Advantage Assignment: Stage-level HRPO provides the best accuracy-efficiency trade-off, whereas HRPO-sentence lowers accuracy and HRPO-token achieves shorter total length but insufficient semantic reasoning credit.Token-level assignment remains useful for precise format control at the boundary token, but stage-level assignment better matches streaming and deep deliberation.
  • Adaptive Streaming Rewards: Adding format rewards improves accuracy under GRPO and HRPO, while adding length rewards to HRPO improves accuracy and reduces total length by 15.2% on P2 and 12.9% on P1.Format rewards stabilize parseable read-think trajectories; length-aware rewards suppress redundant computation, whereas flat GRPO length pressure can hurt accuracy.
  • Out-of-Domain Cases: On GSM-Infinite, HRPO increases accuracy from 0.479 to 0.546 while producing the shortest streaming and total lengths, and vLLM delivers a 4.3× AdaSR-HRPO latency speedup.AdaSR introduces only minor latency fluctuations versus StreamingThinker, particularly with vLLM, supporting deployment in realistic serving scenarios.

5 Conclusion

AdaSR enables adaptive streaming reasoning by deciding when to think or skip and allocating computation between intermediate streaming reasoning and final deep deliberation. The framework aims to support more flexible, efficient, and responsive reasoning for real-time, dynamically evolving inputs.

  • 5 Conclusion: AdaSR lets LLMs decide when to think, when to skip, and how to allocate computation between streaming reasoning and final deep deliberation as inputs unfold.This adaptive framework targets progressively arriving inputs.
  • 5 Conclusion: The work introduces Hierarchical Relative Policy Optimization to decompose advantage assignment across streaming and deep-reasoning stages.
  • 5 Conclusion: AdaSR is positioned as a step toward more flexible, efficient, and responsive reasoning models for real-time and dynamically evolving inputs.

Limitations · A Related Work

AdaSR is positioned as an initial RL-based framework for adaptive streaming reasoning, currently evaluated on text streams with verifiable answers. Related work spans streaming interaction, think-while-reading systems, and RLVR, while HRPO addresses hierarchical credit assignment and learns adaptive computation policies.

  • Limitations: AdaSR currently focuses on text streams with verifiable answers to isolate temporal credit assignment and evaluate HRPO cleanly.Extending it to audio, video, and open-ended interactive streams may require richer rewards, modality-specific rollout engines, and adaptive hierarchy scheduling.
  • A Related Work: Streaming LLM research moves inference beyond static full-context processing toward dynamic interaction among perception, reasoning, generation, and external actions.Applications include simultaneous translation with read/write policies such as wait-k.
  • A Related Work: Think-while-reading offers a streaming-reasoning abstraction in which models process segments sequentially, reason under partial observations, and deliberate after the stream ends.StreamingThinker implements sentence-level streaming CoT with streaming-specific attention, positional encoding, and KV-cache designs.
  • A Related Work: StreamingThinker demonstrates reasoning under streaming visibility constraints, but its intermediate reasoning behavior is largely shaped by constructed supervised traces.This limitation motivates learning computation policies directly rather than relying primarily on preconstructed trajectories.
  • A Related Work: AdaSR studies whether models can learn when to think, when to skip, and how to allocate computation between online thoughts and final deep reasoning.This distinguishes its objective from methods that mainly reproduce externally constructed reasoning behavior.
  • A Related Work: RLVR optimizes language-model policies with automatically checkable outcome rewards rather than learned reward models, advancing critic-free optimization, length-bias correction, stability, and multi-reward composition.The related work also considers multimodal grounding.
  • A Related Work: Existing RLVR methods largely optimize completed trajectories at sequence or token level without explicitly modeling streaming reasoning’s hierarchical temporal structure.HRPO assigns advantages across streaming, deep-reasoning, and global token ranges for phase-structured credit assignment under partial and evolving inputs.

B HRPO Analysis

The analysis shows that GRPO’s sequence-level credit assignment is mismatched with streaming reasoning because it gives streaming and deep-thinking tokens the same learning signal. HRPO instead decomposes local and global objectives by phase, enabling distinct streaming- and deep-reasoning credit assignment.

  • GRPO Limitation for Streaming Reasoning: GRPO assigns one group-normalized trajectory advantage to every token, despite streaming and deep-thinking tokens serving different roles.Streaming tokens should reflect partial-observation quality and timeliness, whereas deep tokens should reflect full-context integration.
  • GRPO Limitation for Streaming Reasoning: Flat credit assignment can reinforce an incorrect or unhelpful streaming segment when later deep deliberation rescues the final result.
  • Advantage Decomposition and Policy Gradient: HRPO groups local and global surrogate terms by token time step, assigning streaming tokens streaming-local and global signals, and deep tokens deep-local and global signals.
  • Advantage Decomposition and Policy Gradient: Before the boundary, each token uses streaming-local coefficient λ/|ts| plus global coefficient 1/|oi|; afterward, streaming-local credit becomes deep-local coefficient λ/(|oi|−|ts|).
  • Advantage Decomposition and Policy Gradient: HRPO’s clipped-surrogate policy gradient is active only on the selected unclipped PPO branch, with a KL-gradient term subtracted at every token.

C Reward Analysis for Streaming Reasoning

The reward analysis identifies a loophole in gated local length penalties: incorrect or malformed trajectories can be favored over correct but verbose ones. It also explains why HRPO applies weighting after normalization, preserving the effect of β under fixed gates.

  • Adaptive reward hacking: Gated local length penalties can favor incorrect or malformed trajectories over correct but verbose ones.Correct, format-valid samples incur penalties when too long, whereas samples failing the gate receive zero penalty.
  • Adaptive reward hacking: After group normalization, failed rollouts can receive positive local length advantages while a correct long rollout receives a negative advantage.This creates a conflict between the local length branch and the global correctness signal.
  • Adaptive reward hacking: Tuning the gated local-length coefficient cannot fix the loophole because normalization cancels the coefficient.The analysis recommends avoiding accuracy-gated local length scores and handling format validity through masking or filtering during advantage computation.
  • Hyperparameter Sensitivity Analysis: Reward-level weighting makes β ineffective under fixed gates, whereas applying β after component-wise normalization preserves it as an explicit multiplier.A zero-variance component still contributes no relative advantage; corresponding experiments are reported in Table 6.
  • Hyperparameter Weighting Experiments: Table 6 compares format, length, accuracy, and advantage-weighting choices across GRPO, HRPO, HRPO-Sentence, and HRPO-Token using Qwen3-1.7B.The reported study varies reward composition and the advantage weighting coefficient λ before selecting a configuration.

D Evaluation Details of AdaSR

AdaSR’s evaluation details include robustness testing across multiple random seeds and out-of-domain generalization experiments. The latter trains SFT-base on specified mathematical datasets and evaluates it on mathematical and logic-based benchmarks.

  • Multiseed Results with Standard Deviation: Multiple random seeds are used to assess result robustness, with mean performance and corresponding standard deviation reported.Detailed results appear in Tables 7 and 8.
  • Out-of-Domain Experiment Details: The SFT-base model is trained on the math-based datasets P1, P2, and MetaMathQA for the out-of-domain experiment.These datasets define the training setting described for Table 4.
  • Out-of-Domain Experiment Details: GSM-Infinite evaluates out-of-domain mathematical performance, while LogicNLI tests out-of-task generalization on a logic-based benchmark.Both benchmarks are used in Table 4’s out-of-domain evaluation setup.

E Training Details of AdaSR

AdaSR training combines streaming and deep reasoning through hierarchical reward assignment and HRPO optimization. Its implementation adapts rollout generation, batch processing, and causal masking to enforce partial-observation streaming semantics.

  • Training procedure: AdaSR samples streaming rationales followed by deep reasoning, decomposes rewards into streaming, deep, and global advantages, and optimizes them with HRPO.Advantages are assigned to corresponding token groups, allowing explicit control over each reasoning stage’s contribution.
  • Forward computation: AdaSR replaces standard full-context processing with packed logical sequences and sentence-level streaming masks that block future source segments while granting deep reasoning full source access.The implementation removes invalid padding, reconstructs logical source-response sequences, and aligns logits with response-token positions.
  • Training data flow: The training loop computes rewards before repacking rollouts into the StreamingThinker layout, preserving segment boundaries and shared streaming visibility for policy-gradient quantities.The repacked metadata includes _lengths, source_seg_len, target_seg_len, and target_seg_roles.
  • Rollout generation: Streaming rollout uses a round-based vLLM state machine that reveals source segments incrementally and decodes each reasoning segment until a streaming stop token such as <EOT>.Source and target roles receive independent position counters, while segmented roles construct block-diagonal visibility during prefill.

F Latency Analysis of Streaming Reasoning · G Case Study

The latency analysis contrasts read-then-think with streaming thinking, which performs intermediate reasoning during input reception to reduce post-input waiting. The case study shows that RL avoids an arithmetic error that SFT propagates into deep reasoning and an incorrect answer.

  • F Latency Analysis of Streaming Reasoning: Read-then-think defers all reasoning until the full input is observed.Its entire reasoning process therefore contributes to latency after the input stream ends.
  • F Latency Analysis of Streaming Reasoning: Streaming thinking performs intermediate reasoning during input reception.This contrasts with waiting for the complete input before beginning reasoning.
  • F Latency Analysis of Streaming Reasoning: Time delay measures wall-clock latency from final input-token arrival to first answer-token generation.The measure directly reflects user-perceived waiting time after the input stream ends.
  • G Case Study: The extracted math example compares supervised fine-tuning and reinforcement learning for the streaming paradigm.Figure 5 highlights their differing reasoning trajectories.
  • G Case Study: The RL model performs the calculation correctly in the third streaming-thinking segment.This is the step highlighted in the comparison example.
  • G Case Study: The SFT model makes an arithmetic error at the same third-segment step.That error propagates into deep thinking and ultimately produces an incorrect answer.
Loading 2606.14694v2…