PaperlayerView exact revision on arXiv

Source-linked AI summary

AURAL: Adaptive Latent Reasoning with Joint Chunk for Speech Language Models

Yuxiang Wang, Kunyu Feng, Yuancheng Wang, Zihang Liu, Shengbo Cai, Qinke Ni, Wan Lin, Tao Feng, Yingda shen, Ming-Hao Hsu, Zhixian Zhao, Liqiang Zhang, Teddy Sun, Steve Yves, Zhizheng Wu

arXiv:2610.01560v1cs.CLcs.LGcs.SD

TL;DR

Speech language models need to reason well while responding quickly, but explicit CoT adds latency and existing latent approaches can trail CoT with fixed budgets. AURAL trains distributional latent chunks and uses reinforcement learning to adapt reasoning effort; across two backbones it is comparable to CoT-RL, with an 11.8× first-token speedup on Qwen2.5-Omni.

  • Problem

    Speech interaction calls for both intelligence and responsiveness, while CoT adds serial latency and existing latent methods can trail CoT with single-path supervision and non-adaptive budgets.

    Source: §1 · pp. 1–2

  • Method

    AURAL models multiple plausible latent continuations, jointly predicts future-state chunks, and uses concise-answer rewards in AURAL-RL to adapt reasoning effort.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    Across two backbones AURAL-RL is comparable to CoT-RL; on Qwen2.5-Omni it cuts time to first answer token from 1.22 s to 0.10 s (11.8×).

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    AURAL brings deep reasoning closer to the immediacy required for spoken interaction, with harder questions eliciting more latent reasoning steps.

    Source: §1 · p. 2, §6 · p. 10

    The traces exclude inferences about stable personality, illness, and social status, and some source licenses permit research use only.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Model intelligence and fast response jointly shape the quality of interaction with speech language models, yet remain difficult to achieve together. Explicit chain-of-thought (CoT) improves reasoning and audio understanding, but generating intermediate reasoning tokens delays responses. Describing fine-grained acoustic cues further lengthens CoT and increases latency. Latent reasoning can reduce this overhead, yet existing methods often trail CoT and remain limited by single-path supervision and reasoning budgets that do not adapt to problem difficulty. We introduce AURAL, which models a distribution over multiple plausible reasoning continuations in latent space and jointly predicts chunks of future states to reduce sequential forward passes and reasoning latency. To provide initial supervision for latent reasoning, we construct AuralReason-683K: 683K bilingual speech utterances (about 1,000 hours) with concise CoT for emotion recognition, empathetic dialogue, and general reasoning. AURAL-RL then explores beyond these traces, rewarding concise reasoning that yields high-quality answers and adapting reasoning effort to each problem. Across two backbones, AURAL-RL achieves performance comparable to CoT-RL, with larger gains over the respective supervised checkpoints on most metrics. Analysis further shows that harder questions elicit more latent reasoning steps. On Qwen2.5-Omni, it reduces time to the first answer token by 11.8x, from 1.22 to 0.10 s, versus 0.05 s for direct answering.

1 INTRODUCTION

AURAL targets the joint challenge of reasoning well and responding quickly in speech interactions. It replaces token-by-token reasoning at inference with distributional latent chunks and adaptive reinforcement learning, achieving CoT-RL-comparable performance with lower latency.

Motivation

  • Speech interaction requires both intelligence to understand users and response speed that avoids long pauses.The introduction frames both as necessary to closing the gap between current SLMs and responsive, perceptive interaction.Source: §1 · p. 1
  • CoT can improve reasoning and audio understanding, but serial token generation adds latency, especially when detailed acoustic cues require more words.The introduction notes that text labels can flatten mixed vocal cues, while longer descriptions further delay answers.Source: §1 · pp. 1–2
  • Existing latent methods can trail CoT because their supervision favors one continuation and their preset reasoning budgets do not adapt to problem difficulty.The paper also notes that reliable speech CoT traces remain scarce for initial latent supervision.Source: §1 · p. 2

Contributions

  • Across two backbones, AURAL-RL is comparable to CoT-RL; on Qwen2.5-Omni, time to first answer token falls from 1.22 s to 0.10 s (11.8×).It achieves larger gains over the respective supervised checkpoints on most metrics.Source: §1 · p. 2
  • AURAL models plausible continuations in latent space, jointly predicts future-state chunks, and uses AURAL-RL to adapt reasoning effort; AuralReason-683K supplies bilingual CoT traces.The corpus contains 683K utterances, about 1,000 hours, spanning emotion recognition, empathetic dialogue, and general reasoning.Source: §1 · p. 2

2 RELATED WORK

Prior latent-reasoning approaches trade off supervision, sequential computation, and control over reasoning depth. AURAL is motivated by the need to preserve multiple valid continuations while allocating computation according to problem difficulty.

Latent reasoning

  • Prior methods use varied strategies, including recursively fed states, endpoint self-distillation, teacher-forced targets, parallel refinement, or targets from a separate encoder.The cited discussion also describes pooled CoT embeddings as a simpler but coarser supervision design.Source: §2 · p. 2

Latent reasoning in SLMs

  • Some latent approaches remain constrained by serial generation or refinement, preset depth, or supervisory representations fixed in advance.The related-work discussion contrasts these limitations with methods that learn stochastic trajectories based on answer quality and computation cost.Source: §2 · pp. 2–3

Motivation

  • AURAL’s design is motivated by preserving multiple valid continuations and using more computation only when a problem requires it.The related-work discussion identifies single-target supervision and non-adaptive budgets as gaps; Figure 1 depicts a distribution over joint future-state chunks and adaptive RL depth.Source: §2 · p. 2, Figure 1 · p. 3

3 METHOD

AURAL retains CoT as training supervision but performs intermediate reasoning in continuous latent chunks. Its supervised stage models alternative continuations and trains chunk relationships; its reinforcement-learning stage adapts reasoning depth using rollout outcomes.

3.1 AURAL-SFT: LEARNING LATENT REASONING WITH JOINT CHUNK

  • AURAL-SFT pools consecutive CoT embeddings into ordered latent targets, enabling explicit targets at each latent position and teacher-forced training.The targets compress groups of CoT tokens according to compression factor c.Source: §3.1 · p. 3
  • A Gaussian mixture represents multiple plausible next-state modes rather than treating one recorded CoT’s pooled state as the unique continuation.Its mixture components model distinct modes, with weights expressing the relative plausibility of continuations.Source: §3.1 · p. 4
  • Jointly predicting m future states models within-chunk relations and reduces the sequential backbone passes for a trace of length L from L to ⌈L/m⌉.Shared learned slots exchange information through token mixing and are refined through channel mixing.Source: §3.1 · p. 4
  • Chunkwise semantic alignment keeps latent states readable to the language head, while latent scheduled sampling trains on predicted chunks to address the mismatch with inference.The language head must emit EOL to stop latent reasoning and begin response generation; scheduled sampling uses a second pass over predicted chunks.Source: §3.1 · pp. 4–5

3.2 AURAL-RL: ADAPTIVE REASONING WITH REINFORCEMENT LEARNING

  • AURAL-RL scores complete rollouts and rewards concise reasoning only when it produces high-quality answers, allowing reasoning depth to vary by problem.The length bonus is largest for short traces and is multiplied by answer quality, so incorrect answers cannot earn a shortcut reward by stopping early.Source: §3.2 · p. 5

4 DATA CONSTRUCTION

AuralReason-683K provides bilingual speech CoT supervision across voice-sensitive interaction and general reasoning. Its construction combines selected speech sources with concise, task-specific reasoning traces and verified outputs.

Corpus overview

  • AuralReason-683K contains 683K bilingual utterances, about 1,000 hours, for emotion recognition, empathetic dialogue, and general reasoning.The corpus addresses the scarcity of public speech corpora with reasoning that uses vocal information rather than only transcripts.Source: §4 · p. 5

Source construction

  • The corpus combines affective dialogue, human emotion recordings, screened public speech, and synthesized spoken problems in mathematics, knowledge, comprehension, and instruction following.For HumanSpeech-1M, 75K of 1.69M candidates were retained after screening for cases where vocal delivery materially changes the appropriate response.Source: §4 · pp. 5–6

Trace construction

  • CoTs are shortened to retain necessary inferences, explain how vocal cues affect dialogue replies, and stop before the final answer.The construction pipeline verifies answers and repairs CoTs and responses for the main training set; the final corpus has 377K Chinese and 306K English utterances.Source: §4 · p. 6

5 EXPERIMENTS

The experiments test answer quality, transfer across backbones, and response efficiency on speech and broader reasoning tasks. Across the evaluated settings, AURAL-RL improves its supervised checkpoints, while analysis examines compression, latent alternatives, and adaptive reasoning depth.

5.2 MAIN RESULTS

  • AURAL-RL improves all eleven scores on both Qwen2.5-Omni and Kimi-Audio and remains competitive with CoT-RL.On Qwen2.5-Omni, it leads both EchoMind metrics and both MMSU evaluations.Source: §5.2 · pp. 6–7, Table 1 · p. 7

5.3 COMPRESSION FACTOR AND CHUNK SIZE

  • AURAL-SFT accuracy is lower at c = 2 than at c = 4, 6, or 8, while larger chunk sizes generally raise average accuracy.AURAL-RL improves every tested combination, with its largest gains at c = 2; the other compression factors show similar smaller gains.Source: §5.3 · pp. 7–8, Figure 2 · p. 8
  • At c = 4, m = 10, AURAL-RL reduces serial reasoning passes from 64.25 to 1.79 and achieves an 11.8× time-to-first-token speedup over CoT-RL.At c = 8, m = 10, reducing latent reasoning to one pass improves mean time to first token by only 0.016 s over the default; c = 4, m = 10 is the default.Source: §5.3 · p. 8, Figure 2 · p. 8

5.4 ABLATION STUDY

  • Removing the quality-gated conciseness bonus raises EchoMind MCQ by 0.18 points but lowers MMSU by 0.80, while latent depth increases by 19.5%.The EOL hit rate remains 100.00%; the authors conclude the bonus suppresses redundant latent states without a consistent accuracy loss.Source: §5.4 · p. 9
  • A single Gaussian loses 2.64 EchoMind MCQ points, while setting r = 0 costs 2.53 points; increasing K beyond four brings no consistent gain.The authors conclude that four components suffice and that coupling positions within a chunk matters beyond per-coordinate uncertainty.Source: §5.4 · p. 9

5.5 ANALYSIS

  • Shortening GeneralSpeech CoTs reduces mean trace length from 317.5 to 65.3 tokens and raises GPQA accuracy from 29.85% to 35.35%.The paired examples retain the same question and answer, making CoT length the only supervision difference.Source: §5.5 · p. 9
  • Four GMM components produce two meaningfully different correct paths for nearly four in ten questions, and their majority voting beats one-component sampling on all three benchmarks.Individual accuracy changes little, so the authors report that the difference cannot be explained by extra noise.Source: §5.5 · p. 9, Table 3 · p. 9
  • Mean latent depth rises with question difficulty on both GPQA and VoiceBench BBH, indicating that AURAL-RL allocates more depth to harder questions rather than using a fixed budget.On GPQA, mean depth rises from 13.27 to 15.07 across difficulty levels 3–5; on BBH, it rises from 10.68 to 13.47 across levels 1–3.Source: §5.5 · pp. 9–10

6 CONCLUSION

AURAL replaces token-by-token CoT at inference with continuous latent chunks, represents multiple plausible continuations, and uses RL to adapt reasoning depth. Across two SLM backbones, AURAL-RL remains competitive with CoT while achieving an 11.8× speedup in mean time to first token.

6 CONCLUSION

  • Across two SLM backbones, AURAL-RL remains competitive with CoT and achieves an 11.8× speedup in mean time to first token.The conclusion reports this comparison without specifying a per-backbone breakdown.Source: §6 · p. 10
  • AURAL replaces token-by-token CoT at inference with continuous latent chunks, models multiple plausible continuations with a GMM, and uses RL to adapt reasoning depth.Source: §6 · p. 10
  • The work constructs AuralReason-683K, a bilingual corpus of 683K speech utterances.Source: §6 · p. 10

AI USE STATEMENT

The authors used an LLM as a writing aid for linguistic quality and clarity, restricting its role to copy-editing. They attribute all scientific contributions to the human authors.

AI USE STATEMENT

  • An LLM was used as a writing aid to improve the manuscript’s linguistic quality and clarity.Source: AI USE STATEMENT · p. 10
  • The LLM’s assistance was confined to copy-editing, including grammar and spelling corrections, rephrasing, and concision.Source: AI USE STATEMENT · p. 10
  • The authors state that all scientific contributions, including ideas, experimental design, analysis, and conclusions, are their original work.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement limits how speech-derived impressions and source audio are handled. The appendix specifies AURAL’s latent head, joint chunk parameterization, and sampling and feedback procedures.

ETHICS STATEMENT

  • The corpus traces describe how voices sound, restrict perceived age, gender, or accent to response-relevant acoustic impressions, and exclude stable personality, illness, and social-status inferences.Crisis, medical, and self-harm content receives dedicated safety handling; teacher-generated traces may carry biases.Source: ETHICS STATEMENT · p. 10
  • Source licenses and the lack of speaker consent constrain release, with permitted subsets and source identifiers used where waveform redistribution is restricted.Source: ETHICS STATEMENT · p. 10

A AURAL HEAD AND TRAINING DETAILS

A.1 JOINT CHUNK PARAMETERIZATION

  • Learned relative-slot embeddings distinguish future positions, while token and channel mixers communicate across slots.The token-mixing output and final channel projection are zero initialized, so the residual mixer initially preserves slot features.Source: §A.1 · p. 15
  • Chunk-level mixture weights select one component for the entire chunk, whose slotwise parameters include means, scales, and loading vectors.Global directions are shared across anchors, components, and slots.Source: §A.1 · p. 15

A.2 LIKELIHOOD, SAMPLING, AND FEEDBACK

  • Off-diagonal covariance blocks couple stochastic deviations across future positions, while token mixing couples the deterministic parameters defining those deviations.For partial final chunks, the likelihood uses the exact Gaussian marginal over valid slots rather than zero padding; normalization by valid coordinates makes full and partial chunks comparable.Source: §A.1 · p. 15, §A.2 · pp. 15–16

A.3 AURAL-SFT AND AURAL-RL IMPLEMENTATION

AURAL-SFT uses sampled latent chunks in a second training pass, while AURAL-RL optimizes rollouts containing latent, stopping, and answer actions. The implementation scores actions under tempered distributions and uses a dimension-normalized latent surrogate score.

A.3 AURAL-SFT AND AURAL-RL IMPLEMENTATION

  • Latent scheduled sampling first computes gold hidden states, then trains the backbone and language head on sampled chunks in a second pass; drift across chunks is left to AURAL-RL rollouts.Source: §A.3 · p. 16
  • The sampled component index uses stop-gradient, but continuous latent samples remain differentiable so second-pass language loss updates selected distribution parameters and the backbone.Supervised mixture NLL supplies the direct signal for the categorical mixture and all components.Source: §A.3 · p. 16
  • Each RL rollout has latent-chunk, stop, and answer action classes; the sampled component index is internal to the sampler rather than a separate action.Source: §A.3 · pp. 16–17
  • The latent chunk score divides its log density by valid dimensionality, making it a length-comparable surrogate multiplier rather than a strict importance ratio.Source: §A.3 · p. 17
  • When temperature differs from 1, behavior and current-policy scores use the actual tempered distributions; stop actions use Bernoulli EOL probabilities and answer actions use categorical token probabilities.Source: §A.3 · p. 17
  • With one gradient step per rollout batch, the policy ratios equal one up to numerical differences and clipping is inactive, reducing the update to a group-relative policy gradient.Training uses no learned critic, reference policy, or KL regularizer.Source: §A.3 · p. 17

B.1 TRAINING SETUP

The setup specifies supervised and reinforcement-learning training, evaluation subsets and decoding, and protocols for timing reasoning and examining depth against question difficulty. It reports increasing mean latent depth across the sampled difficulty levels on GPQA and VoiceBench BBH.

B.1 TRAINING SETUP

  • AURAL-SFT adds one latent-SFT epoch after a shared CoT initialization; AURAL-RL starts from matching SFT checkpoints and rewards concise correct reasoning.The RL set contains 3,883 prompts, with eight rollouts per prompt; CoT-RL uses the same objective and reward.Source: §B.1 · pp. 17–18

B.2 EVALUATION AND DECODING

  • The evaluation suite excludes EchoMind ASR and open-ended MMAU-Pro questions, and halts latent reasoning at 100 states while reporting mean accepted states and EOL hit rate.EchoMind OpenQ uses a 1–5 scale; other metrics are specified in the evaluation table.Source: §B.2 · p. 19, Table 7 · p. 19, Table 8 · p. 19

B.3 REASONING TIME MEASUREMENT

  • Reasoning time is measured on the same 256 examples across configurations, from after prompt prefill until CoT closure or latent EOL/cap.The reported interval excludes preprocessing, prefill, visible answer generation, scoring, and judge calls; speedup is the reference mean divided by the matched AURAL-RL mean.Source: §B.3 · p. 19

B.4 LATENT LENGTH AND QUESTION DIFFICULTY

  • Mean latent depth rises with rated difficulty on both sampled tasks: GPQA increases from 13.27 to 15.07 states and BBH from 10.68 to 13.47 across reported levels.Levels with too few examples are omitted; difficulty ratings use questions and unlabeled choices without model answers, correctness, or latent trajectories.Source: §B.4 · pp. 19–20

C COMPRESSION AND CHUNK SIZE ACROSS METRICS

Across the additional metrics, AURAL-RL improves every evaluated configuration, while compression and chunk size shape scores. The latency analysis traces reasoning savings through answer onset and response generation, finding substantially earlier visible answers than CoT-RL.

C COMPRESSION AND CHUNK SIZE ACROSS METRICS

  • AURAL-RL improves all ten additional metrics at each of 20 configurations; scores rise on average with chunk size, while c = 2 trails other compression factors.The lead among c = 4, 6, and 8 varies by metric and chunk size; Figure 3 pairs AURAL-SFT scores with gains over matched SFT checkpoints.Source: C COMPRESSION AND CHUNK SIZE ACROSS… · pp. 20–21, Figure 3 · p. 20

D RESPONSE LATENCY

  • TTFT ends at the first visible answer token, and full response latency includes the remaining answer body; both exclude CPU preprocessing and input utterance time.TTFT includes prefill, reasoning, and answer onset.Source: §D.1 · p. 21, D RESPONSE LATENCY · p. 21

D.1 MEASUREMENT PROTOCOL

  • The paired latency comparison uses the same 256 examples and hardware settings, and all measured examples finish before their generation budgets.Stage times are synchronized and speedups are ratios of mean latencies; the second-node results closely match the primary-node speedups.Source: §D.1 · p. 21, Figure 4 · p. 22

D.2 FROM REASONING SAVINGS TO FASTER RESPONSES

  • At c = 4, m = 10, AURAL-RL reaches the first visible answer token in 0.103 seconds versus 1.218 seconds for CoT-RL, an 11.8× TTFT speedup.Its full response speedup is 4.7×; the direct-answer reference reaches answer onset in 0.050 seconds.Source: §D.2 · p. 22

D.3 LATENCY DISTRIBUTIONS AND TASK DIFFERENCES

The latency analysis shows that reasoning savings translate into earlier answers, but the benefit varies with response length and compression settings. The branching protocol then tests whether mixture components yield distinct, valid continuations and describes how those differences are measured.

D.3 LATENCY DISTRIBUTIONS AND TASK DIFFERENCES

  • TTFT speedup ranges from 8.2× to 23.0× across benchmarks, while full-response speedup ranges from 2.4× to 10.8×; longer answers reduce the full-response gain.Full-response speedup is about 2.4× on EchoMind OpenQ and VoiceBench, where AURAL-RL responses are slightly longer than CoT-RL responses.Source: §D.3 · p. 22, Table 9 · p. 23

D.4 LATENCY ACROSS COMPRESSION AND CHUNK SIZES

  • Across compression and chunk settings, TTFT speedup ranges from 6.9× to 13.9× and full-response speedup from 1.3× to 5.3×, despite reasoning speedups reaching 55.6×.At c = 4, m = 10, reasoning speedup is 30.7×, compared with the grid maximum of 55.6×; prefill and answer decoding remain fixed costs.Source: §D.4 · p. 23, Figure 5 · p. 23

E.1 BRANCHING PROTOCOL

  • Branching fixes the prompt and accepted latent prefix, then intervenes at the state with highest mixture entropy across three answer-checkable tasks.The 2,001-prompt set includes GPQA, VoiceBench OpenBookQA, and VoiceBench BBH; four operators generate 16 rollouts per prompt.Source: §E.1 · p. 24
  • The four operators contrast component-forced branches, within-component resampling, a moment-matched collapsed Gaussian, and a separately trained single-Gaussian head.The collapsed Gaussian preserves the mixture's first two moments while removing its separate modes.Source: §E.1 · p. 24

E.2 MEASURING DISTINCT AND VALID PATHS

  • Branch comparisons measure answer change, branch and majority-vote accuracy, latent and language-projection distance, token overlap, depth difference, and distinct-correct rate.Distinctness is defined relative to the median projection distance under within-component resampling; a prompt counts when it has at least two correct paths beyond that threshold.Source: §E.2 · pp. 24–25

E.3 INTERVENTION RESULTS

The intervention results associate separate mixture modes and higher mixture entropy with more distinct valid reasoning continuations and stronger majority voting. The corpus sections describe AuralReason-683K’s sources, filtering, concise CoT construction, templates, and checks on whether dialogue plans shape responses.

E.3 INTERVENTION RESULTS

  • Mode intervention produces greater branch separation than within-component resampling, with latent distance rising from 0.087 to 0.312 and projection distance from 0.064 to 0.283.Mode branches reuse the same noise draw, so differences at the branching point come from component parameters.Source: §E.3 · p. 25, Table 10 · p. 25
  • Collapsing the mixture lowers distinct-correct rate to 16.09%, versus 38.88% for mode intervention; mode-intervention voting also adds 4.30, 3.41, and 3.75 points on GPQA, OpenBookQA, and BBH.DCR rises from 22.79% in the bottom entropy tercile to 54.72% in the top tercile.Source: §E.3 · p. 26, Table 11 · p. 26, Table 12 · p. 26

F.1 CORPUS COMPOSITION

  • AuralReason-683K combines affective dialogue, human speech, and general reasoning, with each example containing a waveform, concise CoT, and response.Table 13 summarizes the bilingual mixture after filtering and CoT construction.Source: §F.1 · p. 26, Table 13 · p. 27

F.2 SOURCES AND FILTERING

  • The source branches undergo distinct filtering: spoken-question normalization and CoT verification for general tasks, label-agreement screening for emotion clips, and speech and dialogue screening for human recordings.The human-speech dialogue branch narrows 90K candidates to approximately 75K utterances and 83 hours; the emotion branch retains 17,861 of 33,242 clips.Source: §F.2 · pp. 27–29

F.3 COT CONSTRUCTION

  • CoTs retain necessary evidence and reasoning steps while removing repetition and post-conclusion reasoning; generated traces are checked for alignment, useful intermediate steps, and answer separation.Dialogue traces connect audible evidence to conversational need and response plan; emotion traces compare acoustic evidence across candidate classes.Source: §F.3 · pp. 29–30, Table 18 · p. 29

F.4 PROMPT TEMPLATES

  • The templates specify high-recall and high-precision utterance filtering, dialogue and emotion reasoning states, speech-question normalization, and separate comparison and verified-training trace versions.The comparison version preserves the original answer, while the verified version checks and repairs it when needed.Source: §F.4 · pp. 30–31

F.5 PLAN EXECUTION

  • Dialogue CoT-response pairs are judged fully executed in 91.3% of cases, compared with 6.0% for randomly paired controls.The authors report that the judge distinguishes matched from mismatched reasoning and that the plans are specific enough not to fit arbitrary responses.Source: §F.5 · p. 31, Table 20 · p. 31

F.6 ILLUSTRATIVE EXAMPLES

  • The examples illustrate acoustic evidence linked to dialogue responses and emotion labels, alongside shortened general-reasoning traces.The cited examples report CoT reductions from 529 to 62 tokens and from 517 to 86 tokens; another rewrite resolves reversed inequalities while retaining the answer interval.Source: Table 21 · p. 32, Table 22 · p. 33, Table 23 · p. 34
Loading 2610.01560v1…