PaperlayerView exact revision on arXiv

Source-linked AI summary

Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage

Siddhant Arora, Haidar Khan, Kai Sun, Xin Luna Dong, Sajal Choudhary, Seungwhan Moon, Xinyuan Zhang, Adithya Sagar, Surya Teja Appini, Kaushik Patnaik, Sanat Sharma, Shinji Watanabe, Anuj Kumar, Ahmed Aly, Yue Liu, Florian Metze, Zhaojiang Lin

arXiv:2510.02044v1cs.CLcs.SDeess.AS

TL;DR

End-to-end spoken dialogue systems can produce ungrounded or outdated responses, while integrating external tools can disrupt conversational flow through added latency. This paper introduces Streaming RAG, which issues tool queries during user speech, and reports improved factual accuracy with reduced tool latency.

  • Problem

    End-to-end spoken dialogue systems rely on static training knowledge that can limit factual grounding and currency, while integrating external tools adds latency that can disrupt conversational flow.

    Source: §1 · pp. 1–2

  • Method

    Streaming RAG issues tool queries in parallel with user speech, and its model-triggered approach post-trains the model to determine query timing from the evolving utterance.

    Source: §1 · p. 2

  • Results

    Over 200% relative improvement in accuracy (from 11.1% to 34.2% absolute) versus the no-tool baseline and a 20% reduction in tool result generation latency are reported for Model-Triggered Streaming RAG.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Tool-integrated speech-in speech-out dialogue can more than double factual question-answering accuracy compared with closed-book models while reducing tool usage latency.

    Source: §6 · p. 10

    Synthetic-audio latency calculations exclude endpoint-detection latency, which production systems require; including it would further amplify the observed latency savings.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

End-to-end speech-in speech-out dialogue systems are emerging as a powerful alternative to traditional ASR-LLM-TTS pipelines, generating more natural, expressive responses with significantly lower latency. However, these systems remain prone to hallucinations due to limited factual grounding. While text-based dialogue systems address this challenge by integrating tools such as web search and knowledge graph APIs, we introduce the first approach to extend tool use directly into speech-in speech-out systems. A key challenge is that tool integration substantially increases response latency, disrupting conversational flow. To mitigate this, we propose Streaming Retrieval-Augmented Generation (Streaming RAG), a novel framework that reduces user-perceived latency by predicting tool queries in parallel with user speech, even before the user finishes speaking. Specifically, we develop a post-training pipeline that teaches the model when to issue tool calls during ongoing speech and how to generate spoken summaries that fuse audio queries with retrieved text results, thereby improving both accuracy and responsiveness. To evaluate our approach, we construct AudioCRAG, a benchmark created by converting queries from the publicly available CRAG dataset into speech form. Experimental results demonstrate that our streaming RAG approach increases QA accuracy by up to 200% relative (from 11.1% to 34.2% absolute) and further enhances user experience by reducing tool use latency by 20%. Importantly, our streaming RAG approach is modality-agnostic and can be applied equally to typed input, paving the way for more agentic, real-time AI assistants.

1 Introduction

The paper addresses limited factual grounding in end-to-end spoken dialogue systems by integrating external tools, while tackling the latency that can disrupt conversational flow. It proposes Streaming RAG, which can issue tool queries during user speech.

1 Introduction

  • End-to-end spoken dialogue systems rely on static training knowledge that can leave responses ungrounded or out of date, especially for action-oriented and knowledge-seeking tasks.External tools can improve factual accuracy, but invoking them can add latency and awkward silences.Source: §1 · pp. 1–2
  • Streaming RAG launches tool queries in parallel with user speech, sometimes before the utterance ends; fixed-interval and model-triggered approaches are introduced.Fixed-interval queries run at regular intervals, while model-triggered streaming post-trains the model to choose query timing based on the evolving utterance.Source: §1 · p. 2
  • Over 200% relative improvement in accuracy (from 11.1% to 34.2% absolute) versus the no-tool baseline, alongside a 20% reduction in tool result generation latency, is reported for Model-Triggered Streaming RAG.The paper also reports that tool integration alone increased accuracy by up to 140% relative (from 11.1% to 26.3% absolute) while increasing first-token response time by 2.3x.Source: §1 · p. 2
  • AudioCRAG is a benchmark made by recording spoken queries from CRAG to evaluate tool usage in speech-in speech-out systems.The paper says it will open-source training code and the AudioCRAG-Human benchmark.Source: §1 · p. 2

2 Related studies

Prior tool-use benchmarks largely focus on text-based outputs, while end-to-end spoken dialogue systems have not been trained or evaluated for external tool use. Earlier speech retrieval work and cascaded tool interfaces leave key capabilities and latency questions open.

2.1 Benchmarks for Tool Usage

  • CRAG provides 4,409 question-answer pairs and mock APIs for web and knowledge-graph search, but tool-use benchmarks remain largely limited to text-based outputs.Other benchmarks extend evaluation to multimodal input and longer contexts without fully addressing speech-in speech-out challenges.Source: §2.1 · p. 3

2.2 E2E Spoken Dialogue Systems

  • End-to-end spoken dialogue systems had not been trained or evaluated for tool use; speech-to-text retrieval did not establish access to knowledge graphs or other APIs.The cited retrieval experiments restricted search to 10 paragraphs, unlike benchmarks requiring search across thousands of web pages.Source: §2.2 · p. 3
  • Comprehensive empirical study of tools in end-to-end speech-in speech-out systems and systematic analysis of user-perceived latency remained largely unexplored.Prior work had integrated tools into spoken interactions using cascaded pipelines.Source: §2.2 · p. 3

3 Methodology

The method integrates external tools through query and response generation, then streams tool queries alongside incoming speech to reduce user-perceived latency. It compares fixed-interval and model-triggered strategies, with post-training that teaches query timing and supports recovery from mistaken intermediate queries.

3.1 Tool Integration for Speech-in Speech-Out LLMs

  • Tool integration separates query generation from response generation, conditioning the spoken answer on both the original audio question and retrieved results.The formulation is intended to preserve speech-in speech-out advantages, including reduced error propagation and access to non-phonemic information such as prosody and speaker intent.Source: §3.1 · pp. 3–4

3.2 Streaming RAG

  • Streaming RAG issues tool queries in parallel as audio arrives, using trigger, thread, and reflector choices to manage when queries run and how results are used.The framework is designed to reduce user-perceived latency; its two approaches use fixed-interval or model-based triggering.Source: §3.2 · p. 4

3.2.1 Fixed-Interval Streaming RAG

  • Fixed-interval streaming predicts a query after each audio block, runs parallel tool calls, then uses a reflector to select the earliest intermediate call matching the final query’s results.The reflector checks whether the top five web documents match and whether KG results are identical; this approach needs no additional post-training, but parallel calls add computational overhead and the reflector can limit latency savings.Source: §3.2.1 · pp. 4–5

3.2.2 Model-Triggered Streaming RAG

  • Model-triggered streaming conditions each block’s decision on accumulated speech and the most recent query, choosing either NO_QUERY or a new tool query.It runs one tool-call thread at a time, removes the external reflector, and uses the latest results for the spoken response.Source: §3.2.2 · p. 5
  • The training pipeline generates pseudo-ground-truth queries for partial speech and labels a block NO_QUERY when its query is sufficiently similar to the previous one.For KG queries, sameness means an exact query match; for web queries, the top five retrieved documents must remain unchanged.Source: §3.2.2 · pp. 5–6
  • Multi-task fine-tuning trains streaming tool-query generation and response generation to improve query timing decisions and speech intelligibility.Source: §3.2.2 · p. 6
  • Negative-sample training addresses errors in intermediate queries, and the authors report that it maintains accuracy while achieving latency savings.Training sometimes substitutes an incorrect previous query and falls back to the pseudo-ground-truth query as the label.Source: §3.2.2 · p. 6

4 Experiment Setup

The evaluation uses two spoken-query benchmarks derived from CRAG and tests three speech-in, speech-out models under tool-augmented and non-tool-augmented conditions. It also includes ablations on post-training and a self-cascade setup for selected architectures.

4.1 Evaluation Benchmarks

  • AudioCRAG comprises 1,862 TTS-generated spoken queries in its Synthetic variant and 618 human-recorded spoken queries in its Human variant.The evaluation follows CRAG’s setup with web and KG tools and uses a random subset of 16,000 TriviaQA questions for post-training.Source: §4.1 · p. 7

4.2 Evaluated SOTA Speech-in Speech-out Models

  • The evaluation compares Qwen-OMNI, Kimi-Audio, and OpusLM under tool-augmented and non-tool-augmented conditions.Source: §4.2 · p. 7
  • Ablations compare streaming-RAG post-training with standard post-training and test whether RAG references during response generation affect generated speech quality.The self-cascade uses query generation, text response generation conditioned on the question and tool results, then speech generation; it applies only to thinker-talker or Chain-of-Thought-style architectures because the speech stage teacher-forces the text output.Source: §4.2 · p. 7

5 Results

Tool access improves factual QA but adds latency; Model-Triggered Streaming RAG predicts queries during speech, improving accuracy and reducing latency. The results also show a persistent gap between text and speech output and gains from post-training and negative sampling.

5.1 Impact of Tool Integration and Streaming RAG on SOTA Models

  • Closed-book accuracy is below 20% for all three models; Open Book substantially improves accuracy for Qwen2.5-7B and Kimi Audio, with added retrieval latency.The comparison covers AudioCRAG-Synthetic and AudioCRAG-Human.Source: §5.1 · p. 7
  • 34.2% accuracy for Qwen2.5-7B with Model-Triggered Streaming RAG is comparable to 32.1% for LLAMA-3 8B Instruct in the original CRAG paper; improvements also generalize to human-spoken queries.On the synthetic benchmark, Qwen2.5-7B and OpusLM reduce first-token latency by 20.7% and 14.8%, respectively, with greater savings on the human benchmark.Source: §5.1 · p. 8

5.2 Analysis: Modality Gap Between Speech and Text Output

  • Across models and settings, speech-output accuracy remains below text-output accuracy; Model-Triggered Streaming RAG improves speech output beyond self-cascade while maintaining comparable text performance.The comparison spans Closed Book, Open Book, and Model-Triggered Streaming RAG on AudioCRAG-Synthetic.Source: Table 2 · p. 8, §5.2 · p. 9

5.3 Analysis of Latency Bottlenecks in Tool-Integrated Speech Dialogue

  • 2.3x higher first-token latency in Open Book reflects external web-page delays; Model-Triggered Streaming RAG reduces P50 first-token latency by 9.8% and tool-use latency by 20.7%.Table 3 breaks out query generation, result generation, and speech synthesis, with median and 90th-percentile timings.Source: Table 3 · p. 8, §5.3 · p. 9

5.4 Ablation Study

  • Fixed-Interval Streaming RAG reduces Open Book Qwen-OMNI tool-use latency by 10.7% without post-training, while Model-Triggered Streaming RAG improves average savings and benefits more queries using one parallel thread.Fixed-Interval requires multiple parallel threads; Model-Triggered is described as plug-and-play with existing speech-in speech-out models.Source: §5.4 · p. 10
  • Post-training query generation preserves comparable performance, while negative sampling improves robustness to intermediate prediction errors and supports high accuracy with latency reductions.Without negative sampling, streaming query generation can reduce final text-output accuracy because of final-query errors.Source: §5.4 · p. 10, Table 4 · p. 10, Table 5 · p. 10

6 Conclusion and Discussion

The paper integrates external tools directly into end-to-end speech-in, speech-out dialogue systems and uses Model-Triggered Streaming RAG to decide when to query. On AudioCRAG, tool integration more than doubles factual QA accuracy versus closed-book models, while streaming reduces tool-use latency by 20%.

6 Conclusion and Discussion

  • On AudioCRAG, tool integration more than doubles factual QA accuracy versus closed-book models, while Streaming RAG reduces tool-use latency by 20%.The paper introduces a Model-Triggered Streaming RAG pipeline and says it will release training code and the AudioCRAG Human benchmark.Source: §6 · p. 10

A Benchmarking Text Dialogue Systems for Tool Usage

Text-based tool-use benchmarks assess factual QA, task completion, or web-based actions, but do not fully address speech-in, speech-out systems.

A Benchmarking Text Dialogue Systems for Tool Usage

  • CRAG contains 4,409 question-answer pairs and mock web and knowledge-graph APIs for retrieval tasks.Its evaluation uses two LLM judges and highlights hallucinations in RAG and the role of knowledge graphs and search ranking in factual accuracy.Source: A Benchmarking Text Dialogue Systems… · p. 14
  • SimpleQA tests factual accuracy with 4,326 adversarially collected questions across diverse topics and single-answer grading.It is designed for short, fact-seeking questions.Source: A Benchmarking Text Dialogue Systems… · p. 14
  • WebArena evaluates dialogue agents on web-based tasks using fully functional simulated websites, extending beyond question answering.The passage characterizes the benchmarks as largely limited to text-based interactions.Source: A Benchmarking Text Dialogue Systems… · p. 14

B Multimodal Benchmarks for Tool Usage

Multimodal tool benchmarks cover multi-step tasks, visual QA, and video understanding, yet the cited benchmarks do not evaluate speech-in, speech-out scenarios.

B Multimodal Benchmarks for Tool Usage

C Audio CRAG Benchmark

AudioCRAG adapts CRAG queries into synthetic and human-recorded speech, and the tool setup combines web search with a knowledge graph. The benchmark includes query adaptation and audio-quality filtering, with response-quality measures also reported.

C Audio CRAG Benchmark

  • 569 of the CRAG queries were rewritten before speech conversion when dates or brackets could impair spoken naturalness or clarity.The rewritten queries were combined with the remaining originals to form AudioCRAG.Source: C Audio CRAG Benchmark · p. 14
  • 1,862 filtered synthetic queries and 618 human-recorded queries make up the two AudioCRAG benchmarks.Synthetic utterances with non-zero Whisper word error rate or UTMOS scores below 3.5 were excluded.Source: C Audio CRAG Benchmark · pp. 14–15

D Tool Usage Setup

  • Web retrieval ranks documents from the 100,000-document CRAG corpus, while knowledge-graph queries use a simulated API.A BGE-based reranker retrieves and reranks web-document chunks for each query.Source: D Tool Usage Setup · p. 15
  • Table 6 covers accurate, hallucinated, and missing response rates and overall truthfulness scores on AudioCRAG-Synthetic.It compares text and speech outputs for Qwen2.5-7B, OpusLM, and Kimi Audio in the Open Book setting.Source: Table 6 · p. 15

E Evaluation Setting

The evaluation uses model-based scoring of answer accuracy and also reports response-type rates and truthfulness for the Open Book setting.

E Evaluation Setting

  • Answers receive scores of 1, -1, or 0 for accurate, incorrect, or missing responses, respectively, using a Llama 4-maverick evaluator.Speech outputs are transcribed with Whisper before their transcriptions are evaluated.Source: E Evaluation Setting · p. 15
  • Average accuracy is the primary reported measure; Open Book results also include accurate, hallucinated, and missing response rates and truthfulness.Qwen-OMNI was less likely to generate hallucinated responses than OpusLM and Kimi Audio.Source: E Evaluation Setting · p. 15

F Experiment Setup of SOTA speech-in speech-out models

The experiments cover three speech-in-speech-out models and both speech-in-text-out and speech-in-speech-out evaluation. Tool-query timing differs between the fixed-interval and model-triggered settings, and Kimi Audio has a reference-length constraint.

F Experiment Setup of SOTA speech-in speech-out models

G Post Training Data Preparation

The post-training data pipeline derives incremental query examples from spoken TriviaQA questions and uses task-specific prompts for answer generation and tool-query extraction. Streaming prompts compare new and prior tool queries to avoid redundant calls.

G Post Training Data Preparation

  • A random 16,000-question TriviaQA subset supplies conversational-style audio training examples, partial transcripts, and pseudo-ground-truth incremental queries.Word-level timestamps produce partial transcripts at 500 ms intervals; retrieved documents are ranked to the top 50 for each query.Source: G Post Training Data Preparation · p. 16

H.1 Prompt in Closed Book Setting.

  • The closed-book prompt requests a brief answer, uses “invalid question” for false premises, and “I don’t know” when the model is uncertain.It also provides the query time in Pacific Time.Source: §H.1 · p. 16

H.2 Prompt in Open Book / Streaming RAG Setting.

  • The Open Book / Streaming RAG answer prompt supplies references that may or may not help answer the audio question, alongside query time.Source: §H.2 · p. 17

H.3 KG Query extraction in Open Book Setting.

  • Open Book KG extraction prompts classify the query domain and extract domain-specific structured fields into flat JSON.The prompts specify fields for finance, sports, music, movies, and encyclopedia or other queries.Source: §H.3 · p. 18

H.4 KG Query extraction in Streaming RAG Setting.

  • Streaming KG extraction compares the new structured query with the previous result and returns NO_QUERY when they are identical.Source: §H.4 · p. 19

H.5 Web Query extraction in Open Book Setting.

  • The Open Book web prompt rewrites audio questions into concise retrieval queries centered on the topic, entities, and relevant aspects.It instructs the model to preserve crucial details and not invent new ones.Source: §H.5 · pp. 19–20

H.6 Web Query extraction in Streaming RAG Setting.

  • The Streaming RAG web prompt compares a rewritten query with the previous one and returns NO_QUERY when both convey the same intent and would retrieve similar information.Source: §H.6 · pp. 20–21

I LLM as judge prompt

The judge prompt assigns binary scores using correctness, completeness, uncertainty, and consistency criteria, then requires a JSON response with an explanation and score.

I LLM as judge prompt

  • The rubric gives 1 for an exact match or concise correct summary, and 0 for uncertainty, nonanswers, contradictions, or incorrect answers.For numeric answers, it requires an almost exact match; set-valued answers must contain exactly the ground-truth items.Source: I LLM as judge prompt · p. 21
  • The required response is a JSON object containing an explanation and a score of 1 or 0.Source: I LLM as judge prompt · p. 21
  • Examples score incomplete dividend dates and an extra franchise item as incorrect.The dividend prediction omits two dates, while the movie prediction adds The Animatrix beyond the ground-truth list.Source: I LLM as judge prompt · p. 22
  • The lake-depth prediction scored 1 because 618.23 feet exactly matches the ground truth of 618 ft after rounding, while the stock prediction of $2.16 billion scored 0 against $2,237,578,268.Source: I LLM as judge prompt · p. 22
Loading 2510.02044v1…