Source-linked AI summary
DuplexSpeechBench-Document Grounding: Benchmarking Document Grounding and Hallucinations in Voice Agents
Puneet Mathur, Nedim Lipka, Zeyu Jin, Dinesh Manocha
TL;DR
Document grounding in voice agents remains underexplored despite the need to answer faithfully under real-time speech constraints. DSB-DG tests grounding across context length, multi-turn dialogue, and context re-injection, finding architecture-specific failures and frequent unsupported generation. The results underscore contextual grounding as an unresolved challenge for reliable full-duplex voice agents.
Problem
Document grounding in full-duplex and real-time voice agents remains largely unexplored, despite the need to preserve factual support under latency and turn-taking constraints.
Source: §1 · p. 1Method
DSB-DG evaluates document-conditioned voice agents with three protocols for context-length effects, multi-turn fact retention, and periodic context re-injection.
Source: §1 · p. 2Results
Takeaways & Limitations
Contextual grounding remains an unresolved challenge for reliable full-duplex voice agents, and conversational responsiveness alone is insufficient for reliable document-grounded interaction.
Source: §6 · p. 6The results do not establish that evaluated systems are suitable for autonomous high-stakes use without additional safeguards and human oversight.
Source: §8 · p. 7
Abstract
from arXiv · showhide
Voice agents enable low-latency, natural interaction, yet their ability to faithfully ground responses in external documents remains underexplored. We introduce DuplexSpeechBench-Document Grounding (DSB-DG), a benchmark for evaluating document grounding in voice agents across five professional domains. DSB-DG targets three failure modes: Context Saturation, which measures grounding under increasing document length; Grounding Decay, which measures retention of document facts across multi-turn dialogue; and Proactive Grounding, which evaluates whether context re-injection mitigates conversational drift. The benchmark contains 1,636 adversarially verified QA pairs from 50 documents covering five professional domains, and supports fully automatic evaluation of grounding accuracy, hallucination, and response latency. Across systems spanning cascaded, proprietary full-duplex and real-time, and open-weight speech2speech architectures, we find substantial differences in effective grounding capacity. While cascaded pipeline (ASR-LLM-TTS) achieves the highest grounding accuracy, Gemini-Live and GPT-Realtime closely trail behind. Open-weight systems exhibit distinct failure modes, most notably an abrupt context-capacity collapse and multi-turn grounding decay. More broadly, grounding fidelity degrades with context and conversational load, and failures frequently manifest as unsupported generations rather than abstention. We show that contextual grounding as a key unresolved challenge for reliable full-duplex voice agents.
1 Introduction
Real-time voice agents need to answer from task-specific documents, but document grounding under speech and conversational constraints remains largely unexplored. DSB-DG evaluates grounding across context length, multi-turn dialogue, and context re-injection; results show architecture-specific failures and frequent unsupported generation.
1 Introduction
- Document-grounded speech must preserve factual support under latency and turn-taking constraints, yet grounding in full-duplex and real-time systems remains largely unexplored.Existing voice-agent benchmarks emphasize conversational behavior, while long-context benchmarks do not evaluate real-time spoken interaction.Source: §1 · pp. 1–2
- DSB-DG evaluates three grounding failure modes using 1,636 adversarially verified QA pairs from 50 documents across five professional domains.Its protocols test increasing document length, fact retention across dialogue, and whether context re-injection restores grounding.Source: §1 · p. 2
- The cascaded system has the strongest overall grounding, while Gemini-Live, GPT-Realtime, and MiniCPM-o approach its single-turn accuracy during real-time speech generation.SALMONN shows an abrupt context-capacity cliff, Ultravox-32B severe multi-turn grounding decay, and re-injection benefits vary by architecture.Source: §1 · p. 2
- Across systems, grounding failures under contextual load frequently appear as unsupported generation rather than abstention.The authors identify this as a reliability gap between conversational fluency and factual grounding.Source: §1 · p. 2
2 Methodology
DSB-DG uses three controlled protocols to assess document grounding under increasing context length, extended dialogue, and periodic context re-injection. Its design isolates context load, conversational progression, and refresh strategy as distinct conditions.
2 Methodology
- The benchmark’s three protocols assess grounding as documents grow, fact retention across extended dialogue, and whether periodic re-injection mitigates conversational drift.Responses are automatically scored using grounding, hallucination, retention, and latency metrics.Source: Figure 2 · p. 3, §2 · p. 3
2.1 Context Saturation
- Context Saturation holds the question and supporting evidence fixed across five prefix-consistent tiers of approximately 500 to 8,000 tokens, isolating contextual load.Each spoken question is asked under every eligible tier without conversational history.Source: §2.1 · p. 3
2.2 Grounding Decay
- Grounding Decay repeats the same factual probes after domain-relevant distractor dialogue, and GDS measures the change in accuracy from early to late probes.Negative GDS indicates grounding loss; near-zero decay is interpreted jointly with initial accuracy.Source: §2.2 · p. 3
2.3 Proactive Grounding
- Proactive Grounding compares no refresh, one mid-conversation refresh, and multiple context refreshes while keeping conversations and probes fixed.The comparison measures re-injection benefit relative to no refresh; positive values indicate improved grounding and negative values degradation.Source: §2.3 · pp. 3–4
2 Methodology
- The benchmark comprises 1,636 verified question–answer pairs across five professional domains, represented at five context tiers.It also includes controlled multi-turn conversations.Source: Table 2 · p. 4
3 Benchmark Design
The benchmark combines verified document questions, controlled spoken input, and automatic response scoring. Its design covers varied question types and separates grounded from unanswerable cases.
3 Benchmark Design
- The benchmark contains 1,636 QA pairs, including 1,341 grounded and 295 unanswerable questions, generated with restricted document context and three-stage adversarial verification.Questions cover entity and numerical retrieval, procedural reasoning, clause interpretation, and multi-hop comprehension.Source: §3 · p. 4
- Queries are synthesized as 16 kHz mono speech with a fixed TTS voice to standardize speaker, prosody, and recording conditions across systems.This isolates document grounding under standardized acoustic input.Source: §3 · p. 4
- An LLM judge scores Whisper-transcribed responses against the source context, question, and reference answer to assess document support.The judge receives the response transcript as well as the context and reference materials.Source: §3 · p. 4
4 Experiments
The experiments compare spoken-dialogue systems using grounding, hallucination, retention, and latency measures. The supplied results indicate markedly different context-scaling behavior across architectures.
4 Experiments
- The evaluation covers seven spoken-dialogue systems spanning full-duplex, cascaded, proprietary real-time, and open-weight speech architectures.The systems include Gemini-Live, Cascaded, GPT-Realtime, MiniCPM-o-4.5, SALMONN, Ultravox-v0.6-Qwen3-32B, and Moshi.Source: §4 · p. 4
- Across context tiers, Cascaded, Gemini-Live, and GPT-Realtime stay robust, while SALMONN collapses at T4, MiniCPM-o drops at T5, and Moshi never grounds.Table 3 reports Grounding Accuracy across token-length tiers.Source: Table 3 · p. 5
- Hallucination Rate rises with context-capacity failures: SALMONN’s T3-to-T4 collapse coincides with rising hallucination, and MiniCPM-o shows a similar failure at T5.Figure 3 identifies these failures as unsupported generation.Source: Figure 3 · p. 5
- Quality-Adjusted Latency combines median First-Token Emission Delay with T3 grounding accuracy to penalize low latency at the expense of grounding.The experiments also report median, mean, and p95 FTED latencies.Source: §4 · p. 5
5 Results
Grounding quality, context tolerance, dialogue retention, and refresh effects vary substantially across architectures. Strong grounding does not by itself ensure low latency, and context failures often appear as unsupported generation.
Context Saturation Analysis
- SALMONN’s GA drops from 70.3% at T3 to 37.2% at T4 (−33.1%), while Gemini-Live ranges from 85.9% at T1 to 90.2% at T4 and retains 86.9% at T5.UV-32B declines from 85.5% at T3 to 75.4% at T5; MiniCPM-o falls from 88.0% at T3 to 74.0% at T5, and Moshi remains at or below 6.8%.Source: §5 · p. 5
Capacity limits amplify hallucination
- At T5, hallucination rate rises to 46.5% for SALMONN, 25.0% for MiniCPM-o, and 22.7% for UV-32B as grounding degrades.SALMONN’s hallucination rate increases from 30.1% at T3 to 39.8% at T4; Gemini-Live stays between 10.5% and 13.7% across tiers.Source: §5 · p. 5
Document domain
- Clinical protocols and FDA 510(k) filings are the most challenging domains, while leases and municipal minutes reach 93.3%–98.6% GA across five systems.SALMONN records 57.3% GA on FDA filings and 62.5% on clinical protocols, but 86.2% on leases.Source: §5 · p. 5, Table 4 · p. 5
Quality–Latency Trade-off
- Cascaded achieves 90.3% grounding accuracy (GA) with 1469 ms p50 FTED and QAL 143; GPT-Realtime reaches 87.7% GA, 1214 ms p50 FTED, and QAL 150.Moshi is fastest at 229 ms p50 but has 3.2% GA; the reported quality–latency comparison emphasizes evaluating both dimensions.Source: Table 7 · p. 6, §5 · p. 6
Grounding Decay Analysis
- UV-32B has the largest reported dialogue decay, falling from 80.0% to 75.0% GA (GDS −5.0%), while Cascaded remains at 86.7% (GDS 0.0%).Gemini-Live has the strongest retention; Moshi’s −0.4% GDS is uninformative because initial grounding is only 1.7%.Source: Table 5 · p. 5, §5 · p. 6
Proactive Grounding Analysis
- Multiple context refreshes raise Cascaded GA from 87.8% to 94.9% (+7.1%), while one refresh raises GPT-Realtime from 79.6% to 94.2% (+14.6%).Repeated refreshes reduce GA for UV-32B (−2.2%), MiniCPM-o (−1.1%), and SALMONN (−4.4%); Gemini-Live is largely insensitive.Source: §5 · p. 6, Table 6 · p. 6
6 Conclusion
The paper introduces DSB-DG to evaluate document grounding in voice agents and identifies distinct failures under contextual and conversational load. Its findings underscore that responsiveness alone is insufficient for reliable document-grounded interaction.
6 Conclusion
- DSB-DG evaluates document grounding in voice agents and identifies context-capacity collapse, multi-turn grounding decay, and increased hallucination as grounding deteriorates.The authors present it as a reproducible benchmark for developing agents that jointly optimize factual grounding and conversational latency.Source: §6 · p. 6
7 Limitations
The benchmark’s scope is controlled spoken interaction across five domains and contexts up to approximately 8k tokens; its measures and comparisons have additional limits.
7 Limitations
- The evaluation covers five professional domains and contexts up to approximately 8k tokens, leaving longer documents, other domains, and less structured sources outside scope.Questions use standardized synthetic speech, so the benchmark does not measure robustness to speaker diversity, accents, noise, spontaneous speech, or natural disfluencies.Source: §7 · p. 7
- Different text-conditioning interfaces mean results characterize complete deployed systems, not a controlled comparison of language-model backbones alone.Systems are evaluated using their closest available context interface, which the authors note is not architecturally equivalent.Source: §7 · p. 7
- Grounding quality relies primarily on an LLM judge, which may retain errors for partially correct, implicitly supported, or acoustically ambiguous responses.The benchmark also does not directly evaluate naturalness, emotion, speaker similarity, interruption handling, safety, or task-completion quality.Source: §7 · p. 7
8 Ethics Statement
The paper describes privacy-conscious benchmark construction and evaluation procedures, alongside related benchmarks and appendices detailing data, verification, and system setup. Its ethical statement cautions against treating benchmark results as evidence of suitability for autonomous high-stakes use.
8 Ethics Statement
- DSB-DG uses no private conversations, personally identifying information, or person-specific factual queries; user speech is synthetically generated with a fixed TTS voice.The benchmark targets document content rather than personal profiling and avoids collecting or releasing human voice recordings.Source: §8 · p. 7
- The authors warn that unsupported outputs may occur and that results do not establish suitability for autonomous healthcare, finance, legal, or regulatory use without safeguards and human oversight.The benchmark measures failure modes rather than endorsing deployment in high-stakes settings.Source: §8 · p. 7
A Related Work
- DSB-DG complements conversational-dynamics, spoken-QA, and text long-context benchmarks by testing document fidelity in real-time, generative, multi-turn voice interaction.Its protocols distinguish failures associated with document length, conversational forgetting, and context re-injection.Source: A Related Work · p. 9
B.1 Document Sources and Selection
- The corpus contains 50 documents, ten per domain, covering clinical protocols, earnings calls, FDA 510(k) filings, commercial leases, and municipal minutes from public or synthetic sources.Documents are parsed into text without OCR, summarization, or reordering, then arranged into prefix-consistent tiers of approximately 500, 1k, 2k, 4k, and 8k tokens.Source: §B.1 · pp. 9–10, §B.2 · p. 10, §B.3 · p. 10
B.4 QA Generation
- GPT-OSS-20B generates document-grounded and unanswerable QA candidates across retrieval, procedural, interpretive, and multi-hop question types.Three verification stages check answerability and exact evidence matching, remove ambiguous candidates, and test cross-tier consistency; the final set has 1,636 pairs, including 1,341 grounded and 295 unanswerable.Source: §B.4 · p. 10, §B.5 · pp. 10–11
C.1 Evaluation Schema
- Responses are transcribed with Whisper-large-v3 and evaluated by GPT-OSS-120B using the document, question, reference answer, and transcript.Architecture-specific context interfaces include system-level text conditioning, conversation items, or KV-cache prefill for Moshi.Source: §C.1 · p. 11, §C.2 · p. 11, §D.3 · p. 12
D.4 Context Injection by Architecture
The evaluation specifies architecture-specific document injection and standardized audio, then tests context length, multi-turn retention, and context refresh under controlled protocols.
D.4 Context Injection by Architecture
- MiniCPM-o, SALMONN, and UV-32B receive document context through system messages or prompts, with UV-32B reasoning blocks stripped before scoring.The shared spoken-assistant prompt instructs systems to answer only from the reference document and say when an answer is unavailable.Source: §D.4 · p. 13, §D.5 · p. 13
- Moshi receives document tokens through silent-frame KV-cache pre-filling, while GPT-Realtime receives a system conversation item and Gemini-Live receives the document in session setup.Cascaded is evaluated as a three-stage Whisper–GPT-4o–TTS system, not used to generate the dataset.Source: §D.4 · pp. 12–13
D.5 Context-Injection Prompts
- Proactive-Grounding conditions all begin with the document; they differ in whether and how often it is re-injected during conversation.The refresh protocol holds scripts and probes fixed and changes only the refresh schedule.Source: §D.5 · p. 14, §E.4 · p. 15
D.6 Audio Generation and Standardization
- All systems receive identical synthesized 16 kHz mono PCM audio where supported, with standardized leading and trailing silence.Model adapters may add further end-of-turn silence.Source: §D.6 · p. 14
D.7 Experimental Runs and Reproducibility
- Each instance is run three times per system, with median performance reported and failed API or inference calls retried using exponential backoff.Open-weight evaluations use a specified A100 server environment, while API experiments use hosted OpenAI and Gemini APIs.Source: §D.7 · p. 14
E.1 Context Saturation
- Context Saturation evaluates each eligible QA pair’s identical spoken question at every applicable tier in content-preserving nested document samples, with conversation state reset between independent single-turn runs.Each question is evaluated from its minimum eligible tier through T5.Source: §E.1 · p. 14
E.2 Grounding Decay
- Grounding Decay uses 17-turn probe–distractor–repeat conversations, with identical early A and late A′ questions to measure changes across dialogue.Intermediate B probes target different document facts; distractors are domain-related and avoid revealing later probe answers, but are validated through generation-time constraints rather than a separate post-hoc validator.Source: §E.2 · p. 15, §E.3 · p. 15
E.4 Proactive Grounding
- Proactive Grounding reuses the Grounding Decay scripts and probes, re-injecting the T3 document before specified probe groups without otherwise altering conversation state.The complete benchmark includes the scripts; the representative conversation is abbreviated.Source: §E.4 · p. 15, §E.3 · p. 15
F Additional Results
The additional results characterize hallucination type, refusal behavior, and sensitivity to question difficulty and evidence position. Errors are often document contradictions, while difficulty effects are stronger and more consistent than evidence-position effects at short contexts.
Hallucinations
- Across models, grounding errors are dominated by intrinsic contradictions of the document rather than extrinsic unsupported additions, and false-confidence rates track hallucination rates closely.For Gemini-Live, intrinsic hallucinations are 10.0% versus 2.6% extrinsic hallucinations.Source: F Additional Results · p. 15
Abstention on Unanswerable Questions
- SALMONN answers roughly 38% of unanswerable questions and has a 31.4% false-confidence rate, while Moshi almost never abstains (NRR 12.5%, FCR 85.7%).The strong systems’ refusal rates range from GPT-Realtime at 86.6% to Cascaded at 84.6%, with Gemini-Live at 85.6%.Source: F Additional Results · p. 15
Hallucinations
- MiniCPM-o’s hallucination rate nearly doubles at T5 (12.3→25.0), while Gemini-Live’s stays between 10.5–13.7% across tiers.SALMONN’s hallucination rate rises steadily with context; Moshi’s high rate accompanies fluent but ungrounded speech.Source: Table 18 · p. 16
Sensitivity to Question Difficulty and Evidence Position
- Question difficulty degrades nearly every model monotonically; GPT-Realtime falls from 89.7 to 68.1 and SALMONN from 66.4 to 44.0 between easy and hard questions.Gemini-Live peaks on medium questions (easy 87.6, medium 89.1, hard 82.6).Source: F Additional Results · p. 16
- Evidence position has a smaller, less consistent effect than question difficulty at short contexts, with most models nearly position-invariant and Gemini-Live improving from 83.4 early to 90.4 late.GPT-Realtime dips on mid/late evidence.Source: F Additional Results · p. 16
G Qualitative Analysis
The examples illustrate distinct grounding failures and successes: context-capacity collapse, unstable or decaying retrieval, and unsupported additions alongside otherwise correct answers. They also show that a fluent response can fabricate an answer when evidence is absent.
G Qualitative Analysis
- Across models, grounding errors are overwhelmingly intrinsic document misreadings rather than extrinsic inventions; Moshi is degenerate, with GA of 3.6%.The hallucination decomposition is pooled over tiers T1–T5 on grounded or answered turns.Source: Table 19 · p. 16
Grounded retrieval at long context — Cascaded
- Cascaded retrieves the exact $100 late fee at the longest context tier, illustrating retrieval under contextual load.The example uses a lease document and a question about rent paid more than three days after the due date.Source: G Qualitative Analysis · p. 16
Extrinsic hallucination on a correct core answer — UV-32B
- UV-32B gives the correct meeting-minutes outcome but adds “unanimously,” an unsupported detail absent from the document.The example illustrates an extrinsic hallucination layered onto a correct core answer.Source: G Qualitative Analysis · p. 17
G Qualitative Analysis
- A response claiming 15 study arms for an unanswerable question shows that even a strong system can fabricate a specific answer instead of abstaining.The supplied document did not contain the requested fact.Source: G Qualitative Analysis · p. 17
Context-capacity cliff — SALMONN
- SALMONN answers correctly at T3 but produces degenerate speech at T4, matching its Context Saturation accuracy drop from 70.3% to 37.2%.The example concerns a question about Southern Company’s lower 2025 net income compared with 2024.Source: G Qualitative Analysis · p. 17
Measurable multi-turn decay — UV-32B
- The document defines an unreversed 2-point decline as a decline of at least 2 points not reversed to a 1-point decline or better at last observation; the repeated answer substitutes recovery within 1 point of the original level.Source: G Qualitative Analysis · pp. 17–18
Stable retention — Cascaded
- Cascaded repeats the same grounded diagnostic-purpose answer after intervening dialogue, consistent with aggregate GDS of 0.0 pp.The example compares identical early and repeated probes.Source: G Qualitative Analysis · p. 18
Turn-to-turn instability rather than monotonic decay — SALMONN
- SALMONN misses the early NCT-identifier probe but answers the repeated probe correctly, so its positive GDS reflects turn-level instability rather than improving memory.The example cautions that a positive change score can occur alongside weaker absolute grounding.Source: G Qualitative Analysis · p. 18
H.1 FTED Measurement
FTED measures the interval from the end of user audio to the model’s first emitted audio unit. Latency grows with context for some open-weight systems, while others remain nearly context-invariant.
H.1 FTED Measurement
- FTED runs from the end of the user audio segment to the first emitted model audio unit, including network transit and audio encoding/decoding.It excludes queuing and connection-establishment overhead; the report gives p50, mean, and p95 across turns.Source: §H.1 · p. 18
- Open-weight latency measurements use NVIDIA A100-SXM4-80GB GPUs with CUDA 12.x and each model’s native PyTorch/transformers inference stack.Models and servers are kept warm, turns run sequentially, and response ASR is excluded from FTED.Source: §H.1 · p. 18
- Latency grows with context for MiniCPM-o, SALMONN, and UV-32B, while Moshi and GPT-Realtime are essentially context-invariant.SALMONN’s high T4 p95 reflects large-context stalls; cascaded mean and p95 are inflated by occasional LLM/API stalls.Source: Table 23 · p. 19
- Gemini-Live has a high but nearly context-invariant latency floor (p50 6.4–7.0 s), while UV-32B is not real-time capable.Source: Table 23 · p. 19