PaperlayerView exact revision on arXiv

Source-linked AI summary

RetroThinker: Enabling Retrospective Thinking in Speech LLMs

Yi-Jen Shih, Puyuan Peng, Abdelrahman Mohamed, David Harwath

arXiv:2609.11864v1eess.AScs.AIcs.CL

TL;DR

Streaming SpeechLLMs face complex-reasoning gaps and tight interaction-latency constraints, motivating exploration of on-the-fly reasoning revision. RetroThinker adds retrospective correction to streaming Moshi and, on spoken GSM8K, achieves an 11 percentage-point accuracy gain at nearly the same latency as Standard Reasoning.

  • Problem

    Streaming SpeechLLMs face weaker complex reasoning than comparable text-based models and tight near-real-time latency constraints, motivating exploration of dynamic reasoning revision.

    Source: §I · pp. 1–2

  • Method

    RetroThinker equips streaming SpeechLLMs with forward-only retrospective CoT verification and correction, combined with Early Reasoning and LengthDPO.

    Source: §I · p. 2, §VI · p. 6

  • Results

    An 11 percentage-point absolute accuracy improvement at nearly the same latency compared to Standard Reasoning was achieved by EarlyReasoning + Retro + DPO with the 95% QC threshold.

    Source: C. Main Results · p. 4

  • Takeaways & Limitations

    Early Reasoning and LengthDPO mitigate retrospective reasoning’s latency overhead in the evaluated streaming speech setting.

    Source: §VI · p. 6

    Evaluation is limited to TTS GSM8K, which does not capture natural dialogue’s acoustic variability, interruptions, disfluencies, or task diversity.

    Source: §VII · p. 6

Abstract

from arXiv · show

Speech large language models (SpeechLLMs) offer reduced latency and retain paralinguistic nuances that are typically lost in cascaded automatic speech recognition (ASR) and text-based LM architectures. However, they continue to lag behind text-only LLMs on complex reasoning tasks, while real-time spoken interaction imposes strict latency constraints. Although prior works employ Chain-of-Thought (CoT) and concurrent reasoning to enhance reasoning capabilities without inducing prohibitive delays, an inherent accuracy-latency trade-off persists. In this paper, we investigate whether a streaming SpeechLLM can dynamically revise its reasoning traces on the fly. We introduce RetroThinker, a multi-stage post-training framework that equips the Moshi model to self-verify and forward-correct CoT steps during inference. RetroThinker combines supervised fine-tuning (SFT) on curated retrospective thinking data with length-based direct preference optimization (DPO) to optimize retrospective during early reasoning (i.e., reasoning concurrently while the user speaks). Evaluated on the GSM8K benchmark, RetroThinker significantly improves the accuracy-latency trade-off over non-retrospective baselines, achieving an 11% absolute accuracy gain at a comparable latency.

I. INTRODUCTION

The introduction proposes retrospective thinking to let SpeechLLMs identify and correct reasoning errors during real-time interaction, addressing the challenge of improving reasoning under strict latency constraints.

I. INTRODUCTION

  • SpeechLLMs [4]–[6] directly consume and generate speech tokens, avoiding cascaded ASR–LLM–TTS bottlenecks [1]–[3] while offering lower latency and stronger paralinguistic processing.Speech preserves speaker identity, prosody, and emotional nuance that transcription typically loses.Source: §I · p. 1
  • SpeechLLMs often underperform similar-scale text LLMs on complex reasoning, while CoT and concurrent reasoning reduce but do not eliminate the accuracy-latency trade-off,,, [10]–[12], [13].CoT improves accuracy through step-by-step reasoning but adds latency; concurrent reasoning can begin while the user speaks or while speech is generated.Source: §I · p. 1
  • 11 percentage points higher accuracy at comparable latency: RetroThinker outperforms a comparable non-retrospective baseline on TTS-spoken GSM8K.The evaluation is described as a controlled multi-step reasoning setting.Source: §I · p. 2
  • RetroThinker enables streaming SpeechLLMs to verify and correct hidden CoT traces forward-only, without rolling back generated speech.The approach is motivated by the possibility of dynamically identifying and rectifying errors in reasoning traces during inference.Source: §I · p. 2
  • The Moshi-based post-training recipe combines rule-based initialization, sample-based alignment to model errors, and length-based DPO with Early Reasoning.Moshi’s multistream architecture allows CoT in the text stream without disrupting synchronized audio streams; the paper also analyzes how the techniques interact.Source: §I · p. 2

II. BACKGROUND · A. Moshi · B. CoT Finetuning and Early Reasoning

Moshi processes synchronized speech and text streams with a multi-stream architecture, while prior work adds CoT finetuning and early reasoning to trade accuracy against latency. The section establishes the model and training mechanisms that RetroThinker builds on.

A. Moshi

  • Moshi [4] is a full-duplex SpeechLLM that concurrently processes User Audio, System Text, and System Audio streams.Its inner monologue provides word-level, time-aligned transcripts synchronized with system speech.Source: A. Moshi · p. 2
  • Mimi discretizes speech into audio tokens every 80 ms using 8 codebooks for Moshi’s heterogeneous input streams.Source: A. Moshi · p. 2
  • Moshi’s Temporal Transformer predicts the next system-text token, then its Depth Transformer predicts 8 system-audio tokens.The Temporal Transformer is initially text-only pretrained; the complete model is then pretrained on large-scale audio and fine-tuned on multi-stream spoken conversations.Source: A. Moshi · p. 2

B. CoT Finetuning and Early Reasoning

  • Prior work trains Moshi to generate a CoT trace after the user’s question and uses “thinkingwhile-listening” to start reasoning before the user finishes speaking.Source: B. CoT Finetuning and Early Reasoning · p. 2
  • Prior work uses Question Completeness, scored from 0 to 1, and threshold θ to curate when reasoning starts; inference instead uses a special token to initiate early reasoning.Tuning θ enables explicit control over the accuracy-latency trade-off.Source: B. CoT Finetuning and Early Reasoning · p. 2
  • LengthDPO reduces latency in Early Reasoning by preferring accurate, concise samples over incorrect ones; Standard Reasoning starts CoT after the full query, whereas Early Reasoning overlaps it with speech.Source: B. CoT Finetuning and Early Reasoning · p. 2

C. Retrospective Sampling · III. RETROTHINKER

RetroThinker adapts retrospective correction to streaming SpeechLLMs through multi-stage training and forward-only inference, while REVERSE uses retrospective sampling with rollback for image captioning. Its training progresses from rule-based corrections to corrections based on model failures and length-based preference optimization.

C. Retrospective Sampling

  • REVERSE trains VLMs to detect caption errors using inserted erroneous segments, self-validation tokens, and masked loss on injected errors.Captions are segmented; synthesized error segments are inserted before corresponding ground-truth segments, with a self-validation token appended after each segment.Source: C. Retrospective Sampling · p. 2
  • When REVERSE detects an incorrect segment, it rolls generation back and regenerates that segment to preserve factual consistency.Source: C. Retrospective Sampling · pp. 2–3

III. RETROTHINKER

  • RetroThinker modifies the reasoning trace while preserving other streams, supporting standard and early reasoning and combination with concurrent-reasoning methods [10]–[12].The framework is built on the architecture in.Source: §III · p. 3
  • Stage 1 trains Moshi on perturbed GSM8K CoT steps followed by negative verification markers, corrected steps, and positive markers, masking loss on erroneous tokens.Unlike REVERSE, the model continues generation after an error, learning to revise and proceed; loss is retained on verification markers and corrected continuation.Source: §III · p. 3
  • Stage 2 aligns corrections with authentic model failures by replacing masked model-generated errors with judge-provided corrected continuations.An LLM judge evaluates each generated step given the question and preceding reasoning context, then supplies a corrected continuation when a step is incorrect.Source: §III · p. 3
  • Stage 3 applies LengthDPO to prefer correct, concise outputs over incorrect ones, compressing reasoning to reduce latency while adapting early reasoning to incomplete queries.DPO addresses distribution mismatch because Stage 2 revisions are judged without the constraint that the user query may still be incomplete.Source: §III · p. 3
  • At inference, RetroThinker uses forward-only self-correction: it preserves incorrect and corrected reasoning in the stream and generates corrections as subsequent text.This avoids the temporal rollback that could disrupt streaming speech interaction and matches the fine-tuning setup.Source: §III · p. 3

IV. EXPERIMENTS · A. Training and Inference Details

The experiments use GSM8K [18] as a controlled spoken-delivery test of multi-step math reasoning, evaluating user-facing audio with a shared synthesis, decoding, transcription, and judging pipeline.

A. Training and Inference Details

  • GSM8K [18] provides multi-step reasoning and objective answers; synthesized questions and answers test spoken math reasoning, not natural open-domain dialogue.The authors explain that broader speech benchmarks such as VoiceBench [19] mix reasoning with factual and short-form tasks that do not require retrospective CoT correction.Source: A. Training and Inference Details · p. 3
  • LoRA, rather than full fine-tuning, adapts Moshi; SFT learning rates span 2e-5 to 2e-6, with checkpoints selected by minimum validation loss.The setup uses GSM8K [18] for both controlled training and evaluation.Source: A. Training and Inference Details · p. 3
  • DPO learning rates span 1e-7 to 1e-6 with β = 0.1, alongside positive-example NLL loss weighted by 0.1 and length-normalized probabilities.The policy distribution uses only the text stream, and final checkpoints are selected by highest reward-based validation accuracy.Source: A. Training and Inference Details · p. 3
  • At inference, user audio streams into Moshi and its response waveform is captured; all settings share the same synthesis, decoding, transcription, and judging pipeline.The evaluation pipeline follows.Source: A. Training and Inference Details · p. 3
  • Latency is the number of seconds from the end of the user’s speech to the onset of Moshi’s audio response, measured with VAD on the output waveform.Source: A. Training and Inference Details · p. 3
  • Generated waveforms are transcribed with Whisper, then Qwen3-235B-Instruct [24] judges transcript logical correctness; evaluation uses audio rather than Moshi’s internal text stream.The authors evaluate the waveform because it is the user-facing output.Source: A. Training and Inference Details · p. 4

B. Baselines and Experiment Settings · C. Main Results

Across standard and early-reasoning settings, combining RetroThinker with EarlyReasoning and LengthDPO yields the strongest reported accuracy-latency trade-off.

B. Baselines and Experiment Settings

C. Main Results

  • RetroThinker consistently improves accuracy across evaluated settings, but before LengthDPO these gains add approximately two seconds of latency on average.Source: C. Main Results · p. 4
  • At the 95% QC threshold, EarlyReasoning + Retro + DPO improves absolute accuracy by 11 percentage points at nearly the same latency as Standard Reasoning and reduces latency about 30% versus StandardReasoning + Retro.Source: Figure 3 · p. 4, C. Main Results · p. 4
  • DPO largely closes EarlyReasoning’s trade-off gap with StandardReasoning, and EarlyReasoning + RetroThinker + LengthDPO achieves the best trade-off.Source: C. Main Results · p. 4
  • Removing RetroThinker from EarlyReasoning + DPO, as in, causes a substantial accuracy drop; RetroThinker helps produce the most favorable accuracy-latency Pareto frontier.Source: C. Main Results · p. 5
  • With RetroThinker, QC thresholds still determine reasoning onset, with higher thresholds delaying it, but variable CoT lengths weaken QC’s precise control over latency.Source: C. Main Results · p. 5

V. DETAILED ANALYSIS · A. Understanding the Accuracy Boost of RetroThinker

The analysis frames RetroThinker’s accuracy gains through two kinds of revisions: correcting earlier logical errors and integrating information from the ongoing user query. It also reports CoT length and retrospective-thinking ratio before and after LengthDPO in the EarlyReasoning + Retro setting.

A. Understanding the Accuracy Boost of RetroThinker

B. Rule-based vs. Sample-based RetroThinker · C. Identifying the Accuracy Bottleneck

Stage 2 sample-based training improves accuracy where rule-based Stage 1 does not, while an oracle-verifier test identifies error revision as the primary accuracy bottleneck. The findings distinguish the need for authentic model-generated errors from the remaining challenge of correcting them.

B. Rule-based vs. Sample-based RetroThinker

  • A 9% absolute accuracy gain at θ = 0.85 comes from Stage 2 sample-based training, unlike rule-based Stage 1, where accuracy remains stagnant.Stage 2 uses authentic model-generated errors.Source: B. Rule-based vs. Sample-based… · p. 5
  • Stage 1 establishes the structural format of retrospective tokens but fails to impart strong self-correction abilities.Its accuracy remains stagnant.Source: B. Rule-based vs. Sample-based… · p. 5
  • Authentic model-generated errors in Stage 2 are necessary to bridge the gap between synthetic perturbations and actual inference errors.Source: B. Rule-based vs. Sample-based… · p. 5

C. Identifying the Accuracy Bottleneck

  • The oracle-verifier test replaces sampled internal verification tokens with an external LLM’s judgment of the preceding reasoning.It is applied to the most accurate setting, Standard Reasoning + Retro, to establish an upper bound.Source: C. Identifying the Accuracy Bottleneck · p. 5
  • Accuracy rises from 35% to 42% with the oracle verifier, a 7% gain in Standard Reasoning + Retro.Source: C. Identifying the Accuracy Bottleneck · p. 5
  • The remaining 58% error rate indicates that error revision, rather than internal verification alone, is the primary bottleneck.The result suggests some errors stem from internal verification failures, but most remain after oracle verification.Source: C. Identifying the Accuracy Bottleneck · p. 5

D. Impact of LengthDPO on Retrospective Thinking · E. Analysis by Reasoning Complexity

LengthDPO is associated with shorter CoTs and fewer revisions, while complexity analysis shows the largest reliable accuracy gains on moderately difficult problems, alongside higher latency on harder ones.

D. Impact of LengthDPO on Retrospective Thinking

  • Average CoT length and the Retrospective Thinking Ratio decrease across all QC thresholds, with the ratio declining more at higher θ.The Retrospective Thinking Ratio is the proportion of test instances with at least one CoT revision.Source: D. Impact of LengthDPO on… · p. 5

E. Analysis by Reasoning Complexity

  • At 3 steps, EarlyReasoning w/ RetroThinking & DPO improves accuracy over EarlyReasoning [7] from 23.1% to 37.9% (+14.8 points), and at 4 steps from 12.8% to 25.2% (+12.4 points).The reported accuracy gains at both step counts have non-overlapping error bars.Source: Figure 4 · p. 6, E. Analysis by Reasoning Complexity · p. 6
  • Latency rises monotonically with reasoning steps for both systems, and RetroThinking with DPO adds negligible time at 2–3 steps, roughly +1.2 s at 4 steps, and +1.4 s at 6+ steps.The authors attribute the widening overhead to additional retrospective reasoning on harder problems, where accuracy gains are concentrated.Source: Figure 4 · p. 6, E. Analysis by Reasoning Complexity · p. 6

D. Impact of LengthDPO on Retrospective Thinking

  • Lower θ leaves more limited initial information and causes more reasoning errors, whereas higher θ supports stronger initial hypotheses and fewer revisions.Together with the observed accuracy gains, the authors interpret DPO finetuning as making the model more selective: it retains essential retrospective steps, reducing latency without sacrificing accuracy.Source: D. Impact of LengthDPO on… · p. 6

E. Analysis by Reasoning Complexity

  • The complexity analysis groups GSM8K questions by intermediate reference-solution steps, covers 2–6+ steps, merges the sparse ≥6-step tail, and excludes noisy single-step annotations.Figure 4 compares accuracy and response latency for EarlyReasoning and EarlyReasoning w/ RetroThinking & DPO.Source: E. Analysis by Reasoning Complexity · p. 6
  • Accuracy declines as complexity grows for both systems, but RetroThinking with DPO improves over the baseline in every bin, with smaller, uncertain gains at 5 and 6+ steps.Performance approaches zero at 6+ steps; the largest and most reliable gains are in the mid-complexity regime, so the method mainly recovers moderately difficult problems rather than the hardest long-horizon questions.Source: E. Analysis by Reasoning Complexity · p. 6

VI. CONCLUSION

RetroThinker equips streaming SpeechLLMs with forward-only retrospective reasoning, enabling self-verification and corrective CoT continuations without trace rollback. On spoken GSM8K, Early Reasoning and LengthDPO mitigate the latency overhead of accuracy-improving retrospective training.

VI. CONCLUSION

  • RetroThinker enables streaming SpeechLLMs such as Moshi to self-verify CoT steps and generate corrective continuations without trace rollback.The framework adds forward-only retrospective reasoning through post-training.Source: §VI · p. 6
  • Early Reasoning and LengthDPO mitigate the latency overhead that accompanies retrospective training’s accuracy improvements.The experiments were conducted on the spoken GSM8K benchmark.Source: §VI · p. 6
  • Ablation studies validate the three-stage pipeline and substantiate the paper’s core hypotheses.The analysis of current failure modes points to future research directions for retrospective, low-latency reasoning in spoken language models.Source: §VI · p. 6

VII. LIMITATIONS · VIII. AI-GENERATED CONTENT DISCLOSURE

The limitations constrain the findings to streaming speech math reasoning and qualify accuracy and latency comparisons. The authors also disclose limited use of generative AI for manuscript preparation.

VII. LIMITATIONS

  • The evaluation uses only a TTS version of GSM8K, so its findings support streaming speech math reasoning rather than broad conversational generalization.GSM8K provides step-level rationales and objective answers but does not capture natural dialogue’s acoustic variability, interruptions, disfluencies, or task diversity.Source: §VII · p. 6
  • Accuracy relies on Whisper transcription and an LLM judge, so ASR or judge errors may affect absolute scores.Because all systems use the same pipeline, relative comparisons are more reliable than absolute accuracy scores.Source: §VII · p. 6
  • Retrospective corrections add noticeable delay before DPO, and DPO does not entirely eliminate reasoning’s latency cost.The authors report that RetroThinker significantly improves performance, but the latency cost remains.Source: §VII · p. 6
  • After DPO, the QC threshold controls latency less precisely than in non-retrospective early reasoning.Source: §VII · p. 6
  • Resolving highly complex, multi-step problems remains an open challenge; future work will test natural and noisy speech benchmarks and explicitly control retrospective corrections.The proposed control targets the number and length of corrections.Source: §VII · p. 6

VIII. AI-GENERATED CONTENT DISCLOSURE

  • Generative AI tools were used only for figure formatting and manuscript language editing and refinement; the authors take responsibility for the scientific content, originality, and final work.Source: §VIII · p. 7
Loading 2609.11864v1…