PaperlayerView exact revision on arXiv

Source-linked AI summary

SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models

Cheng-Han Chiang, Xiaofei Wang, Linjie Li, Chung-Ching Lin, Kevin Lin, Shujie Liu, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, Lijuan Wang

arXiv:2510.06917v2cs.CLeess.AS

TL;DR

Current LLMs and SLMs wait for the user to finish before thinking or acting, limiting real-time spoken interaction. SHANKS chunks streaming speech and generates unspoken reasoning while listening, enabling earlier interruptions and tool calls; it achieves 37.1% higher interruption validity and completes 56.9% of API calls before the user finishes.

  • Problem

    Current LLMs and SLMs begin thinking and calling tools only after the user’s input is complete, adding latency and limiting interaction during spoken turns.

  • Method

    SHANKS streams fixed-duration speech chunks and alternates them with unspoken reasoning based on previous speech and reasoning, enabling interruptions and tool calls during the user’s turn.

  • Results

    SHANKS achieves 37.1% more valid interruptions than a no-thinking baseline and successfully completes 56.9% of API calls while the user is still speaking.

  • Takeaways & Limitations

    SHANKS demonstrates thinking-while-listening for more timely spoken interaction across math tutoring and tool-augmented dialogue scenarios.

  • Takeaways & Limitations

    SHANKS requires sufficiently long, sequentially processable speech, and its listening-time reasoning significantly increases inference compute cost.

Abstract

from arXiv · show

Current large language models (LLMs) and spoken language models (SLMs) begin thinking and taking actions only after the user has finished their turn. This prevents the model from interacting during the user's turn and can lead to high response latency while it waits to think. Consequently, thinking after receiving the full input is not suitable for speech-to-speech interaction, where real-time, low-latency exchange is important. We address this by noting that humans naturally "think while listening." In this paper, we propose SHANKS, a general inference framework that enables SLMs to generate unspoken chain-of-thought reasoning while listening to the user input. SHANKS streams the input speech in fixed-duration chunks and, as soon as a chunk is received, generates unspoken reasoning based on all previous speech and reasoning, while the user continues speaking. SHANKS uses this unspoken reasoning to decide whether to interrupt the user and to make tool calls to complete the task. We demonstrate that SHANKS enhances real-time user-SLM interaction in two scenarios: (1) when the user is presenting a step-by-step solution to a math problem, SHANKS can listen, reason, and interrupt when the user makes a mistake, achieving 37.1% higher interruption accuracy than a baseline that interrupts without thinking; and (2) in a tool-augmented dialogue, SHANKS can complete 56.9% of the tool calls before the user finishes their turn. Overall, SHANKS moves toward models that keep thinking throughout the conversation, not only after a turn ends. Animated illustrations of Shanks can be found at https://d223302.github.io/SHANKS/

1 INTRODUCTION

SHANKS enables spoken language models to think while listening, addressing the latency and interaction limits of waiting until the user finishes. It demonstrates more accurate interruption during step-by-step math solutions and completes tool calls before the user’s turn ends.

  • Motivation: SHANKS targets a limitation of current reasoning models that begin thinking only after receiving the complete user input.Thinking while listening is motivated by human communication, where it supports timely reactions and earlier answer preparation.
  • Approach: SHANKS processes speech in fixed-size chunks and generates unspoken reasoning from previous speech and reasoning while the user continues speaking.Its reasoning can determine whether to interrupt the user or make tool calls.
  • Scenario 1: Math-solution interruption: 37.1% more valid interruptions were achieved than with a baseline that interrupts without thinking.SHANKS also interrupted 71% more when users made mistakes.
  • Scenario 2: Tool-augmented dialogue: 56.9% of API calls were successfully completed while the user was still speaking, reducing final-response latency.The task-oriented dialogue scenario used Booking.com APIs for travel planning.

2 METHOD: SIMULTANEOUS HEARING AND THINKING WITH CHUNKED INPUT SPEECH

SHANKS alternates streamed speech chunks with internal thinking chunks, conditioning each new reasoning segment on prior speech and reasoning. Its fixed-duration streaming design supports progressive reasoning but introduces a lag and requires sufficiently long, sequentially processable speech.

  • Inference: SHANKS alternates fixed-duration speech chunks with unspoken thinking chunks conditioned on all previous speech and thinking chunks.Only the final response is spoken; intermediate thinking remains internal.
  • Inference: At each chunk boundary, SHANKS appends the new speech to its context and generates the next thinking chunk while the user speaks the following chunk.For example, R1 is generated while the user is speaking S2, and R2 conditions on S1, R1, and S2.
  • Limitations: At least tchunk seconds of lag separates the user’s speech from SHANKS’s thinking because reasoning begins only after each fixed-duration chunk arrives.The chunk duration must also be large enough for the model to produce meaningful thinking chunks.
  • Limitations: SHANKS requires speech that is long enough for meaningful reasoning and structured so its information can be processed sequentially.Short or non-sequential speech therefore falls outside the method’s stated operating conditions.
  • Training: Training alternates speech and thinking blocks, with variants for interruption tokens and API-call responses whose response loss is masked.This trains the model to make API calls when an incomplete query already contains sufficient information.

3 TASK INTRODUCTION

The paper evaluates SHANKS in two real-time interaction scenarios: interrupting users during step-by-step math solutions and making API calls during spoken travel requests. It describes the task construction, training targets, streaming inference, and evaluation metrics for both scenarios.

  • 3.1 SCENARIO 1: INTERRUPTING USER TURN: SHANKS is evaluated on user speech that contains a math question followed by a correct or incorrect step-by-step solution.The evaluation set is self-constructed from GSM8K questions and includes synthesized speech for both solution types.
  • 3.1 SCENARIO 1: INTERRUPTING USER TURN: The interruption task requires the SLM to interrupt erroneous reasoning while avoiding interruption during correct solutions.The intended use is timely feedback during a student’s problem-solving process.
  • 3.1 SCENARIO 1: INTERRUPTING USER TURN: Training thinking chunks track known information, calculate intermediate variables, and append [INTERRUPT] when an error is detected.The final response either acknowledges a correct solution or explains the error when interrupting.
  • 3.1 SCENARIO 1: INTERRUPTING USER TURN: During inference, an [INTERRUPT] token triggers spoken output while the user continues speaking, after which future speech chunks are withheld.Speech is streamed in fixed-duration chunks and the model alternates between receiving speech and generating thinking.
  • 3.1 SCENARIO 1: INTERRUPTING USER TURN: The evaluation uses interrupt ratio and valid interrupt ratio to measure interruption frequency and whether interruptions correctly identify unclear or mistaken speech.The judge model assesses validity from the speech prefix available at interruption time and the model’s response.
  • 3.2 TOOL-AUGMENTED DIALOGUE: In the tool-augmented scenario, SHANKS determines which Booking.com API calls can be made from partial speech and corresponding thinking chunks.The task is designed to reduce delay by allowing calls before the user finishes speaking.

4 COMPARED METHODS

The comparison includes SHANKS variants, scenario-specific baselines, and implementations spanning end-to-end and cascade spoken language models. The baselines isolate thinking during listening from interruption or tool calling performed only after the user finishes.

  • 4 COMPARED METHODS: The compared methods include two SHANKS variants and scenario-specific baseline models, with training details provided separately.The comparisons cover the tasks introduced in Sections 3.1 and 3.2.
  • 4 COMPARED METHODS: The end-to-end SHANKS model fine-tunes Qwen2.5-Omni so its thinker generates unspoken reasoning while its talker produces only response speech.Originally, Qwen-omni sends every thinker token to the talker for speech synthesis; fine-tuning separates thinking from spoken responses.
  • 4 COMPARED METHODS: SHANKS alternates fixed-duration speech chunks with unspoken thinking chunks conditioned on all preceding speech and thinking.The thinking budget is limited by tchunk × ntps, and API-response tokens are excluded from that limit.
  • 4 COMPARED METHODS: The cascade SHANKS variant combines Whisper-large-v3 with Qwen-2.5-7B-Instruct, allowing a text-only LLM to reason over partial transcriptions.The two models share a base model, while the text-only model is fine-tuned on a larger reasoning dataset.
  • 4 COMPARED METHODS: The No-thinking interruption baseline predicts [NO INTERRUPT] or [INTERRUPT] from chunked speech without generating substantive thinking.This baseline represents SHANKS whose thinking chunks contain only the interruption decision token.
  • 4 COMPARED METHODS: The Call-after-listen baseline waits for the complete user query before iteratively making all required API calls.It represents the post-turn behavior of existing tool-augmented models.

5 EXPERIMENT RESULTS

SHANKS improves real-time interaction by reasoning during listening, enabling more accurate interruptions and early API calls. Combining early SHANKS calls with call-after-listening preserves quality while reducing post-turn generation.

  • 5.1 RESULTS FOR SCENARIO 1: INTERRUPTING USER TURN: SHANKS interrupts more often on incorrect solutions, with about 2 out of 3 interruptions on the wrong subset judged valid.Its interruption ratio is 54.2% higher on the wrong subset than on the correct subset.
  • 5.1 RESULTS FOR SCENARIO 1: INTERRUPTING USER TURN: SHANKS usually interrupts after an error, with 5.08 seconds average interruption latency on the wrong subset.The latency distribution indicates that most interruptions occur later than the first error.
  • 5.1 RESULTS FOR SCENARIO 1: INTERRUPTING USER TURN: The Figure 3 example shows SHANKS computing the correct answer while listening and interrupting when the user falsely calculates 25 petunias.The model begins calculating intermediate variables before the user finishes stating the question.
  • 5.1 RESULTS FOR SCENARIO 1: INTERRUPTING USER TURN: The no-thinking baseline performs much worse than SHANKS on interruption ratio and valid interruption ratio.This comparison supports reasoning before interruption as part of the interruption design.
  • 5.1 RESULTS FOR SCENARIO 1: INTERRUPTING USER TURN: A stronger Qwen-2.5-7B-Instruct backbone improves interruption performance, while changing tchunk from 4 to 3 or 5 seconds does not significantly change core performance.The 3-second setting has the smallest interruption latency and the 5-second setting the largest.
  • 5.2 RESULTS FOR SCENARIO 2: MAKING TOOL CALLS WHEN LISTENING: 56.9% of API calls are completed while the user is still speaking, and combining SHANKS with call-after-listening reduces post-turn generation by 62.3%.The combined method generates 117 tokens after the user finishes, versus 313 for call-after-listen.

6 RELATED WORKS

Related work applies thinking-before-responding to text and audio-aware models, while SHANKS focuses on speech-in-speech-out interaction. Concurrent methods instead emphasize thinking while speaking or issuing retrieval queries during listening without explicit silent chain-of-thought.

  • 6 RELATED WORKS: SHANKS targets speech-in-speech-out SLMs, unlike audio-aware language models that take audio input but output only text.
  • 6 RELATED WORKS: STITCH generates unspoken thinking while the SLM is speaking, whereas SHANKS generates it while listening to the user.
  • 6 RELATED WORKS: Stream RAG issues retrieval and tool queries during listening but does not introduce an explicit silent chain-of-thought process.

7 CONCLUSION, LIMITATIONS, AND FUTURE WORK

SHANKS enables spoken language models to reason while listening, supporting interruption and API calls before the user finishes. The method is promising but requires sufficiently long, sequential speech and incurs chunking and compute costs.

  • Conclusion: SHANKS progressively reasons over chunked user speech, enabling interruption during step-by-step math solving and API calls while the user is still speaking.The framework is evaluated in math tutoring and tool-augmented dialogue scenarios.
  • Limitations: SHANKS is most applicable when speech is long enough for meaningful reasoning and its information can be processed sequentially.These structural requirements are illustrated by the two studied scenarios.
  • Limitations: Fixed-size chunking makes SHANKS lag behind the user by tchunk seconds, motivating more sophisticated chunking methods.
  • Limitations: Incomplete user goals can make listening-time thoughts redundant, while SHANKS significantly increases inference compute cost.
  • Future Work: The authors position thinking while listening, together with the demonstrated scenarios and results, as a potentially fruitful research direction.

B DETAILS IN TRAINING

The training pipeline uses transcriptions rather than user audio when GPT-4o generates training data, with Whisper-large-v3 providing chunk transcriptions and word timestamps.

  • Training Data: GPT-4o receives transcriptions of speech chunks instead of audio to reduce API cost and calling time.Whisper-large-v3 supplies each chunk’s transcription and word timestamps.

B.1 FINE-TUNING FOR INTERRUPTION

Interruption fine-tuning uses sampled and filtered math data, GPT-4o-generated training content, and model-specific parameter-training strategies.

  • Data Preparation: The interruption training data samples 5K Tulu-3-SFT-Math-Grade examples and excludes audio longer than 80 seconds, leaving slightly less than 5K samples.
  • Optimization: All three models use two training epochs, an effective batch size of 64, a 1.0e−4 learning rate, cosine scheduling, and a 0.1 warm-up ratio.The shared setup covers SHANKS-E2E, SHANKS-Cascade, and the no-thinking model.
  • Data Generation: GPT-4o generates most training data, including prompts for reasoning chunks, interruption decisions, and uninterrupted responses.
  • Tool-Call Data: The tool-call training data contains 500 samples, with separate prompts for deciding API-call timing and generating final responses.
  • Fine-Tuning: SHANKS-E2E and think-after-listen use LoRA with selected talker components, whereas SHANKS-Cascade fine-tunes all parameters.The smaller cascade dataset is trained for 10 epochs.

C.1 EVALUATION DETAILS FOR INTERRUPTION

Interruption evaluation uses timestamped speech and GPT-4o judgments to identify the first user error and assess whether interruptions are valid.

  • Interruption Timing: Whisper-large provides word timestamps, while GPT-4o estimates the first error time from the question, answer, transcription, and alignment.
  • Interruption Validity: GPT-4o judges whether each interruption is valid using a dedicated evaluation prompt.

C.2 EVALUATION DETAILS FOR TOOL CALL

The tool-call evaluation constrains each thinking chunk to 320 model-generated tokens, while long API-call arguments can make calls incomplete and unsuccessful. The appendix documents prompts and templates for generating, judging, and timing interruptions and API calls.

  • Long API-call arguments can exceed the 320-token thinking-chunk limit, causing incomplete and unsuccessful API calls.This limitation arises during evaluation when API-call augments contain very long tokens.
  • A proposed workaround places previous API-call responses in a speculative-decoding draft reservoir when long arguments come from earlier tool results.The paper presents speculative decoding as a way to speed inference in this case, but does not specifically handle the failure mode directly.
  • The appendix provides prompts for generating interruption-task thinking chunks, interrupting corrections, and final responses.
  • The evaluation materials include templates for detecting the first interruption-task error and checking the earliest callable API time.
  • A separate prompt is used to judge whether an interruption is reasonable.
Loading 2510.06917v2…