Source-linked AI summary
RetroThinker: Enabling Retrospective Thinking in Speech LLMs
Yi-Jen Shih, Puyuan Peng, Abdelrahman Mohamed, David Harwath
TL;DR
SpeechLLMs face a gap in complex reasoning while real-time speech interaction limits latency, motivating dynamic revision of streaming reasoning traces. RetroThinker equips Moshi with retrospective self-verification and forward correction, combined with early reasoning and length-based DPO; on TTS-spoken GSM8K, it improves accuracy by 11 percentage points at comparable latency. Its evidence is limited to spoken math reasoning and does not establish broad conversational generalization.
Problem
SpeechLLMs underperform text-based LLMs on complex reasoning while real-time spoken interaction imposes strict latency constraints, leaving an accuracy-latency trade-off.
Method
RetroThinker post-trains Moshi with retrospective thinking data and length-based DPO so it can self-verify and forward-correct CoT during early reasoning.
Results
11 percentage points of absolute accuracy improvement are achieved at nearly the same latency as Standard Reasoning on TTS-spoken GSM8K.
Takeaways & Limitations
Combining retrospective thinking with EarlyReasoning and LengthDPO produces the most favorable reported accuracy-latency Pareto frontier.
Takeaways & Limitations
Evaluation is limited to TTS-spoken GSM8K and therefore supports streaming speech math reasoning rather than broad conversational generalization.
Abstract
from arXiv · showhide
Speech large language models (SpeechLLMs) offer reduced latency and retain paralinguistic nuances that are typically lost in cascaded automatic speech recognition (ASR) and text-based LM architectures. However, they continue to lag behind text-only LLMs on complex reasoning tasks, while real-time spoken interaction imposes strict latency constraints. Although prior works employ Chain-of-Thought (CoT) and concurrent reasoning to enhance reasoning capabilities without inducing prohibitive delays, an inherent accuracy-latency trade-off persists. In this paper, we investigate whether a streaming SpeechLLM can dynamically revise its reasoning traces on the fly. We introduce RetroThinker, a multi-stage post-training framework that equips the Moshi model to self-verify and forward-correct CoT steps during inference. RetroThinker combines supervised fine-tuning (SFT) on curated retrospective thinking data with length-based direct preference optimization (DPO) to optimize retrospective during early reasoning (i.e., reasoning concurrently while the user speaks). Evaluated on the GSM8K benchmark, RetroThinker significantly improves the accuracy-latency trade-off over non-retrospective baselines, achieving an 11% absolute accuracy gain at a comparable latency.
I. INTRODUCTION
SpeechLLMs preserve speech information and can reduce latency, but they lag text LLMs on complex reasoning under stricter real-time constraints. RetroThinker addresses the resulting accuracy-latency trade-off by combining retrospective correction with concurrent reasoning.
- SpeechLLMs reduce latency and preserve paralinguistic signals compared with cascaded ASR–LLM–TTS systems.
- Complex reasoning remains difficult for SpeechLLMs, while spoken interaction imposes tighter latency constraints than text-based interaction.
- Concurrent Chain-of-Thought reasoning improves capability and latency but leaves an accuracy-latency Pareto trade-off.
- RetroThinker introduces retrospective thinking so SpeechLLMs can identify and dynamically correct errors in their reasoning traces.
- RetroThinker combines EarlyReasoning, retrospective correction, and DPO, outperforming a comparable non-retrospective baseline by 11 percentage points at comparable latency on TTS-spoken GSM8K.
A. Moshi
Moshi is a full-duplex SpeechLLM that processes user audio, internal text reasoning, and system audio concurrently. Prior work extends it with early Chain-of-Thought reasoning, while REVERSE provides a retrospective self-verification and correction precedent.
- A. Moshi: Moshi concurrently processes User Audio, System Text, and System Audio streams during each forward pass.Mimi discretizes speech into audio tokens, while the inner-monologue stream carries time-aligned text reasoning.
- A. Moshi: The Temporal Transformer predicts system-text tokens, and the Depth Transformer uses them to predict eight system-audio tokens.The Temporal Transformer receives all three streams at each timestep and was initially pretrained on text alone.
- A. Moshi: Early reasoning relaxes audio-text alignment so Moshi generates a CoT trace before speaking, including while the user is still talking.This thinking-while-listening framework targets the latency introduced by CoT generation.
- A. Moshi: REVERSE trains retrospective sampling by inserting synthesized erroneous caption segments and using self-verification to identify incorrect segments.Identified errors can trigger rollback and regeneration to preserve factual consistency.
III. RETROTHINKER
RetroThinker adds retrospective thinking to SpeechLLMs by modifying reasoning traces while preserving the other streams. Its staged training supports both standard and early reasoning and uses preference optimization to favor correct, concise traces and address early-reasoning mismatch.
- III. RETROTHINKER: RetroThinker modifies the reasoning trace while preserving the remaining streams, making it compatible with standard and early reasoning.The framework can also be combined with other concurrent-reasoning methods.
- III. RETROTHINKER: LengthDPO constructs preference pairs that favor correct, concise reasoning traces over incorrect ones.The selected outputs come from the Stage 2 model.
- III. RETROTHINKER: DPO reduces early-reasoning distribution mismatch because the Stage 2 judge does not model queries that are still being received.Early-reasoning DPO adapts retrospective behavior to the partially received-query setting.
A. Training and Inference Details
The training and evaluation setup uses LoRA-adapted Moshi models and GSM8K-derived speech data, with a three-stage retrospective pipeline. Accuracy is measured from transcribed generated audio and judged for logical correctness.
- A. Training and Inference Details: GSM8K is selected because its multi-step ground-truth reasoning and objective final answers support controlled training and evaluation.Questions and answers are synthesized into audio for speech-domain adaptation.
- A. Training and Inference Details: LoRA is used instead of full fine-tuning to reduce computational cost when adapting Moshi.The setup adapts the text-based GSM8K dataset to speech by synthesizing questions and answers into audio.
- A. Training and Inference Details: RetroThinker training comprises rule-based synthetic-error injection, external-LLM judging of authentic errors, and LengthDPO refinement.The final stage addresses distribution mismatch and condenses unnecessarily long retrospective traces.
- A. Training and Inference Details: The evaluation transcribes generated waveforms with Whisper and uses an LLM judge to assess logical correctness.Qwen3-235B-Instruct is used as the judge across experiments, and evaluation targets the user-facing waveform rather than internal text.
B. Baselines and Experiment Settings
The experiments compare Standard and Early Reasoning across QC thresholds, with and without RetroThinker and with LengthDPO. The strongest setting combines EarlyReasoning, RetroThinker, and DPO to improve accuracy at comparable latency.
- Experiment Settings: Early Reasoning experiments vary QC thresholds from 65% to 95%, while comparisons include Standard Reasoning and settings with or without RetroThinker.LengthDPO is applied to early-reasoning models to examine reasoning-trace effects on latency.
- Main Results: RetroThinker consistently improves accuracy, although it initially adds approximately two seconds of latency before LengthDPO.The gray arrows in Figure 3 indicate the performance shift after adding RetroThinker.
- Main Results: LengthDPO largely alleviates EarlyReasoning’s mismatch with StandardReasoning, producing comparable accuracy-latency trade-offs.The strongest trade-off combines EarlyReasoning with RetroThinker and LengthDPO.
- Main Results: Removing RetroThinker from EarlyReasoning + DPO causes a substantial accuracy drop, indicating that retrospective thinking is critical to the best frontier.The baselines are settings without RetroThinker.
V. DETAILED ANALYSIS
The detailed analysis examines retrospective behavior and its effect on reasoning traces. Table III compares CoT length and retrospective-thinking ratios before and after LengthDPO in the EarlyReasoning + Retro setting.
- A. Understanding the Accuracy Boost of RetroThinker: Retrospective revisions include correcting previous logical errors and integrating new information from the ongoing user query.These categories describe how retrospective thinking dynamically improves accuracy.
- A. Understanding the Accuracy Boost of RetroThinker: LengthDPO reduces both average CoT length and the Retrospective Thinking Ratio across all QC thresholds.The ratio reduction is more pronounced at higher QC thresholds.
B. Rule-based vs. Sample-based RetroThinker
The ablation separates rule-based and sample-based retrospective SFT and tests verification independently from correction. Sample-based training improves accuracy, while an oracle verifier reveals correction as the larger bottleneck.
- B. Rule-based vs. Sample-based RetroThinker: A 9% absolute accuracy gain at θ = 0.85 comes from Stage 2 sample-based SFT, whereas Stage 1 rule-based SFT leaves accuracy stagnant.Sample-based training uses authentic model-generated errors rather than only synthetic perturbations.
- C. Identifying the Accuracy Bottleneck: An oracle verifier raises accuracy from 35% to 42%, showing that verification failures contribute errors but revision remains the primary bottleneck.The remaining 58% error rate is attributed to the error revision process.
D. Impact of LengthDPO on Retrospective Thinking
LengthDPO makes RetroThinker more selective, reducing retrospective activity and latency while preserving accuracy gains. By reasoning complexity, the method improves accuracy most reliably on moderately difficult problems but adds latency as complexity increases.
- D. Impact of LengthDPO on Retrospective Thinking: Average CoT length and the Retrospective Thinking Ratio decrease consistently across all QC thresholds after LengthDPO.The ratio reduction is more pronounced at higher θ.
- D. Impact of LengthDPO on Retrospective Thinking: Higher QC thresholds require fewer revisions because more complete input enables more robust initial hypotheses.The reported accuracy gains suggest that DPO retains essential retrospective steps without sacrificing accuracy.
- E. Analysis by Reasoning Complexity: At 3 steps, accuracy rises from 23.1% to 37.9% (+14.8 points), while at 4 steps it rises from 12.8% to 25.2% (+12.4 points).Both improvements have non-overlapping error bars.
- E. Analysis by Reasoning Complexity: RetroThinking with DPO improves accuracy across every complexity bin, with the largest and most reliable gains at mid-complexity.At 5 and 6+ steps, gains are smaller and within the margin of error.
- E. Analysis by Reasoning Complexity: Latency increases monotonically with reasoning steps, while RetroThinking with DPO adds roughly +1.2 s at 4 steps and +1.4 s at 6+ steps.The overhead is negligible at 2–3 steps and widens with complexity.
VI. CONCLUSION
RetroThinker equips streaming SpeechLLMs with forward-only retrospective reasoning. On spoken GSM8K, Moshi can self-verify and correct CoT continuations without trace rollback, while Early Reasoning and LengthDPO mitigate the resulting latency overhead.
- VI. CONCLUSION: RetroThinker equips streaming SpeechLLMs with forward-only retrospective reasoning capabilities.The framework enables self-verification and corrective continuations without requiring trace rollback.
- VI. CONCLUSION: On spoken GSM8K, Moshi can be fine-tuned to self-verify CoT steps and generate corrective continuations without trace rollback.
- VI. CONCLUSION: Early Reasoning and LengthDPO mitigate the latency overhead introduced by retrospective reasoning.The conclusion states that retrospective training improves accuracy at the cost of increased latency.
VII. LIMITATIONS
The evaluation is limited to TTS-spoken GSM8K and an accuracy pipeline using Whisper transcription and an LLM judge. These constraints limit broad conversational interpretation, and reasoning latency and highly complex problem solving remain unresolved challenges.
- VII. LIMITATIONS: Evaluation is limited to TTS-spoken GSM8K, which does not capture natural dialogue variability, interruptions, disfluencies, or task diversity.The authors therefore scope the evidence to streaming speech math reasoning rather than broad conversational generalization.
- VII. LIMITATIONS: Accuracy depends on Whisper transcription followed by an LLM judge, so ASR or judge errors can affect absolute scores.Because all systems use the same pipeline, relative comparisons are more reliable than absolute accuracy.
- VII. LIMITATIONS: RetroThinker improves performance but does not eliminate reasoning latency costs or fully solve highly complex, multi-step problems.The authors identify natural and noisy speech evaluation and explicit control of retrospective corrections as future directions.