When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoning

Jiaqi Wei; Xuehang Guo; Pengfei Yu; Xiang Zhang; Wanli Ouyang; Siqi Sun; Qingyun Wang; Chenyu You

Source-linked AI summary

When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoning

Jiaqi Wei, Xuehang Guo, Pengfei Yu, Xiang Zhang, Wanli Ouyang, Siqi Sun, Qingyun Wang, Chenyu You

arXiv:2605.03314v2May 5, 2026cs.CL

TL;DR

Single-stream reasoning forces models to choose between delaying useful disclosure and making premature commitments. SxS Interleaved Reasoning learns when to think privately and when to disclose supported partial progress, improving accuracy–content-latency trade-offs across architectures, scales, and domains.

Problem
At fixed compute speed, it remains unclear what task-relevant content models should disclose while reasoning, rather than emitting unsupported filler or delaying useful progress.
Method
SxS makes disclosure timing a learnable think-versus-speak decision, trained with entailment-aligned interleaving, SFT, and RL in standard autoregressive decoding.
Results
SxS improves accuracy–latency Pareto trade-offs across MoE and dense architectures, two model scales, and AIME25 and GPQA-Diamond.
Takeaways & Limitations
SxS turns response streaming into a learned commitment policy that supports earlier visible progress while constraining disclosures to the available reasoning.
Takeaways & Limitations
When answer order mismatches reasoning order, interleaving can collapse toward standard reasoning–response structure, and this work adds no constraints to prevent it.

Abstract

from arXiv · show

In single-stream autoregressive interfaces, the same tokens both update the model state and constitute an irreversible public commitment. This coupling creates a silence tax: additional deliberation postpones the first task-relevant content, while naive early streaming risks premature commitments that bias subsequent generations. We introduce Side-by-Side (SxS) Interleaved Reasoning, which makes disclosure timing a controllable decision within standard autoregressive generation. SxS interleaves partial disclosures with continued private reasoning in the same context, but releases content only when it is supported by the reasoning so far. To learn such pacing without incentivizing filler, we construct entailment-aligned interleaved trajectories by matching answer prefixes to supporting reasoning prefixes, then train with SFT to acquire the dual-action semantics and RL to recover reasoning performance under the new format. Across two Qwen3 architectures/scales (MoE Qwen3-30B-A3B, dense Qwen3-4B) and both in-domain (AIME25) and out-of-domain (GPQA-Diamond) benchmarks, SxS improves accuracy--content-latency Pareto trade-offs under token-level proxies such as inter-update waiting.

1. Introduction

SxS treats disclosure timing as a learnable think-versus-speak control problem within standard single-stream autoregressive decoding. It uses entailment-aligned supervision and two-stage training to enable supported early disclosure and improve accuracy–content-latency trade-offs without architectural changes.

Motivation: Single-stream autoregressive decoding couples internal-state updates with irreversible public commitments, creating a tension between deliberation and timely task-relevant disclosure.Additional reasoning can improve substantive claims but delays visible content, while early streaming risks premature commitments that constrain subsequent generation.
Motivation: CoT exposes this tension through long visible preambles, while system-level acceleration does not determine what supported content to disclose during ongoing reasoning.The paper focuses on justified disclosure rather than low-information filler that merely improves measured latency.
Method: SxS lets the model choose between think and speak actions in one tagged token stream, making visibility controllable without a second model, separate hidden state, or specialized inference machinery.Think tokens are non-disclosed deliberation, whereas speak tokens provide user-facing disclosure within the same autoregressive context.
Training: Entailment-aligned trajectories match answer prefixes to reasoning prefixes that entail them, while SFT teaches dual-action semantics and RL recovers reasoning performance under the new format.This design aims to prevent premature commitments and superficial early output by ensuring earlier disclosures are supported by reasoning so far.
Evaluation: Across MoE and dense Qwen3 models, two scales, and AIME25 and GPQA-Diamond, SxS improves accuracy–latency trade-offs using token-level content-latency proxies without architectural changes.Reported proxies include when supported user-visible progress first appears and inter-update waiting.

2. Problem Formulation

The formulation treats disclosure timing as a learnable commitment decision in a dual-channel autoregressive process, separating private reasoning from irreversible public answers. It optimizes task accuracy against content-based latency while allowing early supported disclosures followed by continued reasoning.

2.1. Generation under Coupled State and Commitment: Standard single-stream decoding synchronizes recurrent state evolution with token-by-token public disclosure, so disclosed prefixes constrain all later generation.Longer commitments make the decoding-feasible continuation set more restrictive, capturing the cost of premature commitment.
2.2. Dual-Channel Autoregressive Generation: SxS introduces channel actions R and A, representing private reasoning and public answer emission within one autoregressive trajectory.The same model predicts lightweight channel tags, avoiding a separate policy network.
2.2. Dual-Channel Autoregressive Generation: Private tokens append only to the reasoning stream, whereas answer tokens append to both streams, making public disclosure monotone and irreversible.The standard interface is recovered when every channel action is A.
2.3. Anytime Commitment as Policy Learning: The channel policy πϕ(ck | Σk−1) learns when to disclose by trading off deliberation and commitment.The first public emission time alone can reward low-information filler, so the formulation supplements it with content-based responsiveness and filler monitoring.
2.3. Anytime Commitment as Policy Learning: Content latency g(τ) is measured as the onset of the first speak block containing task-relevant content, excluding generic acknowledgements.The evaluation instantiation uses a candidate answer token as an example of substantive content.
PROBLEM FORMULATION: The objective balances final-answer task loss with latency penalty, enabling supported partial progress early, continued private reasoning, and later answer refinement.Task quality is supplied by an outcome-based correctness signal, while latency uses token-level content-latency statistics.

3. Method

The method converts standard reasoning–answer triples into entailment-aligned interleaved trajectories, then uses SFT and GRPO-based RL to learn when to think and speak. Deterministic segmentation, monotone alignment, optional trailing reasoning, and reward filtering support stable training while preserving correctness.

Trajectory construction: SFT transforms each input–reasoning–answer triple into alternating private reasoning blocks and user-visible answer blocks, with an optional trailing reasoning block.The sequence has the form S = (x, r(1), a(1), …, r(N), a(N), [r(N+1)]).
Entailment-based alignment: Each reasoning prefix is aligned to the largest entailed answer prefix, and only newly unlocked answer increments are emitted when the boundary increases.Monotonicity prevents noisy entailment checks from retracting progress, while the terminal safeguard emits the full answer.
Segmentation: The procedure deterministically segments reasoning and answers on normalized double-newline delimiters, avoiding learned segmentation because it adds overhead and cascading errors.Whitespace normalization makes SPLIT reproducible across corpora and formatting artifacts.
Trailing reasoning: When the full answer is supported before reasoning ends, the remaining reasoning suffix can be preserved as trailing reasoning, often retaining post-solution self-checks.Adjacent reasoning blocks are merged when no answer content is unlocked to avoid overly fragmented trajectories.
Limitations: Mismatched reasoning–answer order can cause early boundary jumps and collapse interleaving toward a near-standard reasoning–response structure; no additional constraints are imposed.The paper leaves penalties on rapid boundary growth and answer reordering for future work.
Reinforcement learning: RL follows SFT with outcome-only GRPO, group filtering, and optional granularity shaping to restore accuracy and improve accuracy–latency trade-offs.Outcome-only reward typically improves accuracy faster, whereas shaping increases granularity but slows accuracy recovery.

4. Experiments

Experiments evaluate SxS across Qwen3-30B-A3B and Qwen3-4B on in-domain and out-of-domain reasoning benchmarks using token-level content-latency proxies. Results show that SxS improves the accuracy–latency trade-off, though SFT can initially reduce accuracy and RL is needed for recovery.

Experimental setup: Experiments use post-trained MoE Qwen3-30B-A3B and dense Qwen3-4B models, evaluating AIME25, GPQA-Diamond, LiveCodeBench, and KOR-Bench.AIME25 uses k = 16 generations, GPQA-Diamond k = 3, LCB reports pass@1, and KOR-Bench reports overall accuracy.
Metrics: The study measures responsiveness with ARI, ABO, and AIRW, where lower values indicate earlier visible content, earlier block onset, and shorter inter-update gaps.Metrics are computed over full sequences containing both think and speak tokens.
Breaking the Silence Tax: 8,519 tokens: Qwen3-4B SxS RL Final reduces AIRW from 21,316 tokens under Standard CoT, shortening gaps between visible updates.Interleaving introduces a small total-token overhead from tagging and switching.
The Alignment Tax and RL Recovery: 50.8%: Qwen3-30B-A3B accuracy after SFT illustrates the immediate dip caused by learning interleaving before adapting to remain correct under the new format.Outcome-based RL subsequently recovers accuracy while largely preserving interleaved behavior rather than reverting to one-shot CoT.
OOD Robustness and Catastrophic Forgetting: 49.3%: SxS retains much higher Qwen3-4B GPQA-Diamond accuracy than Standard CoT after math-focused RL, which falls from 55.9% Base accuracy to 19.0%.The comparison is presented as an empirical observation consistent with reduced catastrophic forgetting under domain-skewed post-training.
Additional benchmark results: 39.62 vs. 39.34: Qwen3-4B SxS RL Final slightly improves LCB accuracy over Standard CoT while lowering AIRW from 12,579 to 9,631.On Qwen3-30B-A3B, accuracy is nearly identical at 54.60 vs. 54.79, while AIRW decreases from 10,401 to 9,270; KOR-Bench shows the same broader pattern.

5. Conclusion

The conclusion presents Side-by-Side (SxS) reasoning as a controllable disclosure-timing framework that mitigates the silence tax by interleaving private reasoning with supported user-visible progress.

Conclusion: SxS makes disclosure timing controllable in standard autoregressive decoding while mitigating single-stream chain-of-thought’s silence tax.The framework interleaves private reasoning with user-visible disclosures, allowing partial progress to appear earlier when supported by the available reasoning prefix.
Conclusion: SxS combines entailment-aligned interleaved supervision with a two-stage SFT+RL training approach.

A. Hyperparameter Configuration & Notation Reference

This section consolidates the paper’s key experimental hyperparameters across training and evaluation stages and provides a reference for its mathematical notation. The notation covers sequences, model and channel-policy variables, trajectories, alignment, and reinforcement-learning components.

Hyperparameter Configuration: Table 3 summarizes key hyperparameters for dataset construction, supervised fine-tuning, reinforcement learning, evaluation, and infrastructure.The configuration is organized by training stage and includes model architecture specifications and training parameters.
Mathematical Notation: Table 4 lists symbols for input and output sequences, model parameters, trajectory representations, policy and generation distributions, alignment procedures, and reinforcement learning.The reference is intended to facilitate comprehension of the paper’s mathematical formulations.
Mathematical Notation: The notation defines x as the input prompt, y1:T as a generated token sequence, V as the vocabulary, θ as model parameters, and ϕ as channel-policy parameters.It also introduces ht for recurrent hidden state and Γt for the committed public transcript.
Mathematical Notation: Trajectory notation distinguishes channel actions ck, emitted tokens zk, complete trajectories τ, private reasoning subsequences r1:IR(K), and public answer subsequences a1:IA(K).R and A denote reasoning and answer channel actions, respectively, while K is the total trajectory length.
Mathematical Notation: Additional rollout notation covers parsed answers, correctness labels, rewards, reasoning-length limits, structure scores, group mean rewards, advantages, correctness margins, and normalization statistics.These symbols include ai, gi, Ri, ℓi, Si, ¯R, Ai, ϵ, µx, and σx.

B. SFT Data Generation Details · B.1. Entailment Detection Prompt

The SFT data-generation pipeline uses GPT-OSS-120B as a Response Coverage Decider to determine which response blocks are supported by the available reasoning. Its prompt enforces conservative no-new-derivation entailment and returns only the number of consecutive addable blocks in JSON.

B.1. Entailment Detection Prompt: GPT-OSS-120B serves as the Response Coverage Decider in the SFT data-generation function.The model is used to implement Algorithm 1, line 10.
B.1. Entailment Detection Prompt: The segmentation prompt prevents hallucinated entailment by requiring coverage decisions from the provided reasoning prefix alone.It asks how many pending solution blocks can move to the covered state using only the available reasoning prefix.
B.1. Entailment Detection Prompt: The decider evaluates processed thoughts, covered responses, the current thought, and remaining response blocks.These fields define the prompt inputs used to determine additional coverage.
B.1. Entailment Detection Prompt: A response block is addable only when its content is established by processed and current thoughts without additional reasoning.This is the prompt’s explicit no-new-derivation entailment rule.
B.1. Entailment Detection Prompt: Allowed zero-derivation operations include paraphrasing, reformatting, definitional substitution, notation normalization, background knowledge, and dropping dead ends.These transformations preserve already established content rather than introducing new inference.
B.1. Entailment Detection Prompt: New inference, fact combination, caveat removal, computation, and implicit theorem application are not addable under the conservative rule.The decider must stop at the first non-addable block and must not skip it.
B.1. Entailment Detection Prompt: The output is JSON-only with the schema { "num_blocks": k }, where k counts addable blocks from the start of the remaining sequence.The prompt forbids extra keys and defines k = 0 as adding none and k = 1 as adding block 0 only.
B.1. Entailment Detection Prompt: In general, the decider adds blocks 0..k-1, with 0 <= k <= M for M remaining blocks.The count therefore represents a contiguous prefix of the remaining response blocks.

B.2. Engineering Optimizations · C. Incentive Design via Quadratic Programming

The engineering design replaces sequential entailment checks with asynchronous parallelization, monotonicity repair, and cancellation to reduce latency and computation. The incentive design uses auxiliary reward shaping to favor fine-grained interleaving while preserving correctness-based advantage separation.

B.2. Engineering Optimizations: Sequentially querying hundreds of reasoning steps creates a significant latency bottleneck, so asynchronous optimistic parallelization reduces wall-clock time within concurrency limits.
B.2. Engineering Optimizations: Concurrent entailment checks launch independent prompts for every cumulative reasoning prefix instead of waiting for each boundary sequentially.
B.2. Engineering Optimizations: Post-processing enforces theoretically monotonic entailment because independent LLM calls can produce noisy, non-monotonic coverage counts.The finalized boundaries repair local failures where longer contexts fail to preserve previously recognized entailment.
B.2. Engineering Optimizations: When a reasoning step achieves complete coverage, pending later checks are cancelled and assigned complete coverage, reducing API costs for easy samples.
C. Incentive Design via Quadratic Programming: Auxiliary reward shaping assigns each batch sample a scalar reward to encourage frequent interleaving without compromising reinforcement-learning signal correctness.
C. Incentive Design via Quadratic Programming: The preference objective favors shorter maximum reasoning block lengths, corresponding to higher interleaving granularity.
C. Incentive Design via Quadratic Programming: The correctness constraint requires correct rollouts to exceed average reward and incorrect rollouts to fall below it, producing positive and negative GRPO advantages respectively.

C.1. Data Preprocessing · C.2. Optimization Formulation · C.3. Solvability and Edge Cases

The method converts reasoning-block structure into correctness-aware target scores, then assigns rewards through a constrained quadratic program. The convex formulation prioritizes correctness while preserving structure alignment and handles degenerate batches by discarding them during GRPO training.

C.1. Data Preprocessing: For each response, the method computes a raw structure penalty from the maximum length of any single continuous reasoning block.This penalty is defined per sample before normalization.
C.1. Data Preprocessing: Incorrect samples receive a worst-case penalty equal to the batch’s maximum observed length, preventing formatting incentives for incorrect answers.The resulting lengths are normalized into structure scores and inverted because shorter blocks are preferred.
C.1. Data Preprocessing: Structure-score normalization uses the mean and standard deviation of penalties among valid correct samples in the batch.The normalization statistics are computed only over samples with g(y_i) = 1.
C.2. Optimization Formulation: The reward assignment is formulated as a quadratic programming problem seeking a final reward vector close to the structure scores in L2 distance.This optimization couples structural alignment with correctness constraints.
C.2. Optimization Formulation: Correct samples must receive rewards significantly above the batch mean, while incorrect samples must remain below it.The constraints are imposed on the optimized rewards relative to their batch mean.
C.2. Optimization Formulation: ε = 0.5 sets the experimental margin, ensuring advantages maintain the correct sign relative to the baseline and prioritize correctness over structure.The advantage relation is approximately A_i ≈ R_i − V(s).
C.3. Solvability and Edge Cases: The optimization is convex and efficiently solvable with standard solvers, but batches containing only correct or only incorrect samples lack an optimal solution and are discarded.This edge-case handling occurs during GRPO training.

D. Additional analysis on LCB and KOR-Bench. · E. Related Work

The paper supplements its latency analysis with additional LCB and KOR-Bench results, while situating SxS among work on reasoning, interleaving, controllable disclosure, and efficiency. It emphasizes that SxS learns fine-grained pacing within standard autoregressive decoding using support-based construction and an accuracy–content-latency objective.

E. Related Work: Prior reasoning methods improve multi-step LLM reasoning by eliciting intermediate computation through CoT, self-consistency, scratchpads, and structured decompositions.The passage situates these techniques within single-stream reasoning and the cost of deliberation.
E. Related Work: Related approaches add temporal structure through reasoning-action alternation, explicit branching or search, observation separation, and interleaving reasoning with partial answers.These examples include ReAct, Tree-of-Thoughts, ReWOO, and recent training for responsiveness.
D. Additional analysis on LCB and KOR-Bench.: Additional latency results are reported for LCB, with latency measured in token indices and lower values preferred.The reported metrics include ARI, ABO, and AIRW.
D. Additional analysis on LCB and KOR-Bench.: Additional latency results are also reported for KOR-Bench using token-index latency measurements.The passage identifies ARI among the latency metrics.
E. Related Work: Another line of work targets long-CoT inefficiency by controlling reasoning length or inducing concise rationales.The passage presents this direction as complementary to SxS.
E. Related Work: SxS instead learns fine-grained pacing within standard autoregressive decoding without extra modules.Its method uses a support-based construction to control disclosure timing.
E. Related Work: SxS explicitly optimizes the accuracy–content-latency trade-off to control when intermediate progress is disclosed.This objective is stated as part of the approach’s distinction from efficiency-focused work.

F. Limitations

The study’s limitations are primarily practical rather than conceptual. They concern the cost and noise of entailment alignment and the simplicity of current pacing objectives.

The study’s limitations are largely practical rather than conceptual.
Entailment-aligned supervision can be expensive at scale and may unlock disclosure segments too early or too late.The method requires only an approximate prefix entailment oracle and does not fundamentally depend on a 120B-class model.
Current pacing incentives use simple structural proxies and outcome-based correctness rewards rather than richer user-utility objectives.Proposed extensions include verifiable intermediate results, uncertainty-aware commitments, and task-specific notions of substantive content.