Source-linked AI summary
Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency
Guan-Ting Lin, Chen Chen, Zhehuai Chen, Hung-yi Lee
TL;DR
Real-time voice agents must perform multi-step tool use while handling natural speech and maintaining conversational responsiveness. FDB-v3 evaluates this problem with human recordings, annotated disfluencies, deterministic chained API scenarios, and six model configurations. GPT-Realtime leads overall accuracy and interruption avoidance, whereas speed and turn-taking trade off across systems; self-correction remains a shared weakness.
Problem
Existing evaluations do not adequately test real-time tool execution with natural spoken disfluencies and mid-utterance intent changes.
Method
FDB-v3 combines real human speech, five disfluency categories, deterministic multi-step API scenarios, and evaluation of six voice-agent configurations.
Results
GPT-Realtime leads overall tool-use accuracy and interruption avoidance, while Gemini Live 3.1 is fastest and the Cascaded baseline has the highest latency; self-correction remains difficult.
Takeaways & Limitations
Future voice agents must balance rapid tool execution with the flexibility to update actions when users change intent mid-utterance.
Takeaways & Limitations
Cloud-model latency measurements include nondeterministic network overhead and varying server-side loads, while local mock APIs omit real network anomalies.
Abstract
from arXiv · showhide
We introduce Full-Duplex-Bench-v3 (FDB-v3), a benchmark for evaluating spoken language models under naturalistic speech conditions and multi-step tool use. Unlike prior work, our dataset consists entirely of real human audio annotated for five disfluency categories, paired with scenarios requiring chained API calls across four task domains. We evaluate six model configurations -- GPT-Realtime, Gemini Live 2.5, Gemini Live 3.1, Grok, Ultravox v0.7, and a traditional Cascaded pipeline (Whisper$\rightarrow$GPT-4o$\rightarrow$TTS) -- across accuracy, latency, and turn-taking dimensions. GPT-Realtime leads on Pass@1 (0.600) and interruption avoidance (13.5\%); Gemini Live 3.1 achieves the fastest latency (4.25~s) but the lowest turn-take rate (78.0\%); and the Cascaded baseline, despite a perfect turn-take rate, incurs the highest latency (10.12~s). Across all systems, self-correction handling and multi-step reasoning under hard scenarios remain the most consistent failure modes.
1 Introduction
Voice agents need to combine practical tool use with low-latency, natural conversation, but existing evaluations do not adequately test real speech, disfluencies, and multi-step state updates. FDB-v3 addresses this gap with human speech, annotated disfluencies, chained API scenarios, and evaluation across six configurations.
- Existing voice benchmarks often separate real speech from tool-use evaluation or rely on synthetic audio and single-step actions.
- FDB-v3 tests whether agents can update downstream API parameters after mid-utterance intent changes.Twenty-one of 100 scenarios test self-correction and state rollback.
- The benchmark uses real human recordings annotated across five disfluency categories, including fillers, pauses, hesitations, false starts, and self-corrections.
- Each scenario requires chained API calls across Travel & Identity, Finance & Billing, Housing & Location, or E-Commerce Support.
- Six configurations are evaluated across accuracy, latency, turn-taking, and disfluency robustness.GPT-Realtime leads accuracy and interruption avoidance; Gemini Live 3.1 is fastest but has the lowest turn-take rate; the Cascaded baseline has the highest latency.
- Self-correction handling remains difficult: even GPT-Realtime succeeds on fewer than 59% of these scenarios.
2 Related Works
Prior work has advanced real-time voice interaction and tool use separately, but existing systems and benchmarks remain limited by synthetic audio, single-step tasks, offline pipelines, or poor reproducibility. FDB-v3 targets reproducible evaluation of complex multi-step tool execution under natural speech conditions.
- Real-speech benchmarks evaluate turn-taking or dialogue but generally lack tool-use evaluation.
- Tool-use benchmarks often rely on synthetic audio or restrict evaluation to single-step calls, omitting natural disfluencies and real-time corrections.
- Recent voice-agent studies explore parallel query prediction or mid-turn tool execution, but incomplete model and inference-code access hinders reproducible comparison.
3 Benchmark Design
FDB-v3 combines deterministic mock APIs with naturalistic human recordings to evaluate multi-step voice-agent behavior across domains, difficulty levels, and annotated disfluency types.
- The benchmark uses locally executed mock APIs with deterministic, zero-latency responses to isolate model reasoning and parameter passing.Deterministic outputs also enable automatic scoring.
- Scenarios span four task domains and three difficulty tiers: single-step Easy, two-step Medium, and multi-step Hard cases with conflicting constraints.
- All audio comes from human speakers in uncontrolled environments, using everyday microphones and including mild background noise.
- Recordings are annotated for five disfluency categories targeting obsolete context, state rollback, redundant tokens, and end-of-turn detection.
- The dataset contains 100 recordings from 12 speakers, with 21 scenarios specifically testing self-correction and real-time state rollback.
4 Experimental Setup
The experiment compares five end-to-end speech-to-speech systems with a conventional cascaded baseline using shared streaming inputs and metrics for tool correctness, response quality, turn-taking, and latency.
- Six configurations are evaluated through LiveKit: five end-to-end speech-to-speech models and a Whisper–GPT-4o–TTS cascaded baseline.Each model receives the same audio input.
- Evaluation Metrics: Tool Selection F1 measures expected versus actual calls, while Argument Accuracy evaluates semantic correctness of generated arguments.
- Evaluation Metrics: Pass@1 requires exactly the expected tools and perfect argument accuracy for every call; any single failure produces a fail.
- Evaluation Metrics: Response Quality judges whether the spoken transcript fulfills the user’s intent and relays tool results effectively.
- Evaluation Metrics: Turn-take rate measures natural-timing responses, while an interruption occurs when Δt = tagent_start − tuser_end is negative.
- Evaluation Metrics: Latency is decomposed into first-response latency, tool-call latency, and task-completion latency.The first two measure time to any speech and time to the first API invocation, respectively.
- Evaluation Metrics: Filler rate records responses containing content-free filler sentences before substantive information, which may reduce perceived latency while risking interruption.
5 Results
GPT-Realtime is the strongest overall performer, while systems trade accuracy against speed and turn-taking reliability. Performance declines with scenario complexity and remains especially fragile for self-corrections, pauses, and multi-step reasoning.
- Overall performance: GPT-Realtime leads overall on tool selection F1 (0.876), argument accuracy (0.680), response quality (0.792), Pass@1 (0.600), and interruption rate (13.5%).
- Overall performance: Gemini Live 3.1 achieves the fastest latency (4.25 s) but the lowest turn-take rate (78.0%), while Gemini Live 2.5 is more conservative.
- Overall performance: The Cascaded baseline guarantees a 100% turn-take rate but combines Pass@1 (0.450) with the highest latency (10.12 s) and a 33.0% interruption rate.
- Performance by difficulty: Pass@1 declines with difficulty across systems; GPT-Realtime leads Easy (0.750), Medium (0.588), and Hard (0.433), while Grok reaches 0.200 on Hard.
- Robustness to disfluency: GPT-Realtime leads or ties every disfluency category and scores 0.588 on self-corrections, while pauses are weakest across most systems.
- Performance by domain: Finance is easiest and Housing hardest across domains, with GPT-Realtime leading all four; Ultravox shows low E-Commerce (0.345) and Housing (0.192) scores.
- Latency analysis: Ultravox speaks quickly before tool execution, pairing first-word latency (3.88 s) with slowest tool-call latency (6.01 s), 88.0% filler rate, and 47.9% interruption rate.
- Latency analysis: The Cascaded system has 10.12 s overall latency, with an 8.78 s first-word delay dominating its sequential pipeline.
6 Discussion
The results expose distinct trade-offs between speed, accuracy, turn-taking, and robustness to disfluency. Fast or concurrent processing can improve straightforward tool execution while preventing reliable rollback when users correct themselves.
- Turn-taking and pre-emption: 41.6% pre-emptive tool calls make Grok the most eager processor, while Ultravox interrupts most often at 47.9% despite a lower 23.2% pre-emptive rate.These rates indicate that early tool invocation and premature speech are distinct failure modes.
- Self-correction robustness: GPT-Realtime leads self-correction handling at 0.588, yet still fails on over 40% of scenarios; the Cascaded system scores 0.176.The core difficulty is updating committed intermediate parameters when intent changes mid-utterance.
- Silent-worker behavior: Gemini Live 3.1 produces no speech in 22% of examples, although 86% of those silent cases still execute tool calls.Its silent-worker behavior separates API execution from speech generation and costs potential Pass@1 improvements.
- Silent-worker behavior: 46.7% of hard scenarios receive no Gemini Live 3.1 response, compared with 23.5% of medium and 0% of easy scenarios.The no-response pattern concentrates as scenario difficulty increases.
- Cascaded baseline: The Cascaded baseline guarantees a 100% turn-take rate but has the highest task-completion latency at 10.12 s.Its sequential Whisper→LLM→TTS chain creates a reliability–latency trade-off.
- Qualitative case studies: Gemini Live 3.1 completes the clean three-tool finance case in 3.92 s versus GPT-Realtime’s 9.20 s, a 2.3× speed advantage.The case contains no disfluency and all four end-to-end models achieve perfect accuracy.
- Qualitative case studies: Only GPT-Realtime correctly applies both travel corrections; Gemini Live 3.1’s −2.27 s tool call locks in the stale destination.The contrasting cases show that eager processing helps clean chains but prevents state rollback under correction.
7 Conclusion
FDB-v3 evaluates real-time voice agents on multi-step tool execution using natural, unscripted human speech. It finds that faster end-to-end processing brings diverse turn-taking failures, while mid-sentence correction handling remains unresolved.
- Contribution: FDB-v3 is introduced as a benchmark for real-time voice agents performing multi-step tool execution on natural, unscripted human speech.The benchmark is designed to expose behavior beyond latency alone.
- Conclusion: Mid-sentence corrections remain an open challenge for all current models, because early processing can lock in outdated user intents.Future architectures must balance fast tool execution with flexibility during changing conversation.
Limitations
The evaluation controls latency comparisons with a fixed server region and high-bandwidth connections, but important environmental limitations remain. Local mock APIs isolate model reasoning while excluding robustness to network and service failures.
- Fixed-region, high-bandwidth evaluation supports fair latency comparisons across cloud-based models.
- Cloud-model latency includes nondeterministic network overhead and varying server-side loads.
- Zero-latency local mock APIs isolate model reasoning from external confounds.
- The mock-API setup does not test robustness to API timeouts, access denials, or malformed responses.
A Evaluation Prompts
The evaluation prompts use structured JSON judges for argument accuracy and response quality, while a separate identifier decomposes speech into filler and task-relevant information for latency measurement. Together, these prompts operationalize tool-call correctness, task completion, and key response timing.
- A Evaluation Prompts: Evaluation prompts use GPT-4o with structured JSON output for LLM-based assessments.
- A.1 Argument Accuracy Judge: Argument Accuracy Judge compares expected and actual function arguments, allowing dynamic references, aliases, formatting differences, and ±5% numeric tolerance.
- A.1 Argument Accuracy Judge: The argument-accuracy judge returns a JSON correctness decision with a brief explanation.
- A.2 Response Quality Judge: Response Quality Judge evaluates whether the agent performed the correct actions and indicated that the requested task was completed.
- A.2 Response Quality Judge: The response-quality judge returns only a JSON correctness decision with a brief explanation.
- A.3 Key Information Identifier (Latency): Key Information Identifier receives speech timestamps and tool calls to separate conversational fillers from the factual task-confirmation response.
- A.3 Key Information Identifier (Latency): The latency prompt models a sequence from user completion to filler, tool execution, and key information response, identifying the key-information start time.
- A.3 Key Information Identifier (Latency): The latency identifier outputs filler and key-information text alongside their start and end timestamps in a JSON object.