PaperlayerView exact revision on arXiv

Source-linked AI summary

$τ$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

Soham Ray, Keshav Dhandhania, Victor Barres, Karthik Narasimhan

arXiv:2603.13686v1cs.SDcs.AI

TL;DR

Voice-agent evaluations have largely separated conversational dynamics from grounded task completion, limiting evidence about real-world full-duplex reliability. τ-Voice extends τ 2-bench with verifiable tasks, full-duplex interaction, realistic audio, and a controllable simulator. Across 278 tasks, voice agents substantially trail GPT-5 under both clean and realistic conditions, with most analyzed failures attributed to agent behavior.

  • Problem

    Existing evaluations address conversational dynamics and grounded task completion in isolation, leaving their combined performance in realistic voice tasks insufficiently measured.

  • Method

    τ-Voice extends τ 2-bench with verifiable grounded tasks, full-duplex interaction, realistic audio, and a controllable voice user simulator.

  • Results

    Across 278 tasks, GPT-5 (reasoning) achieves 85%, while voice agents reach 31–51% under Clean and 26–38% under Realistic conditions, retaining 30–45% of text capability.

  • Takeaways & Limitations

    τ-Voice provides a reproducible testbed for measuring progress toward voice agents that reliably complete grounded tasks while sustaining natural conversation under realistic conditions.

  • Takeaways & Limitations

    The evaluation uses English TTS rather than recorded speech, so accent findings from TTS personas are indicative rather than definitive.

Abstract

from arXiv · show

Full-duplex voice agents--systems that listen and speak simultaneously--are rapidly moving from research to production. However, existing evaluations address conversational dynamics and task completion in isolation. We introduce $τ$-voice, a benchmark for evaluating voice agents on grounded tasks with real-world complexity: agents must navigate complex multi-turn conversations, adhere to domain policies, and interact with the environment. The framework extends $τ^2$-bench into a novel voice agent benchmark combining verifiable completion of complex grounded tasks, full-duplex interaction, and realistic audio--enabling direct comparison between voice and text performance. A controllable and realistic voice user simulator provides diverse accents, realistic audio environments, and rich turn-taking dynamics; by decoupling simulation from wall-clock time, the user simulator can use the most capable LLM without real-time constraints. We evaluate task completion (pass@1) and voice interaction quality across 278 tasks: while GPT-5 (reasoning) achieves 85%, voice agents reach only 31--51% under clean conditions and 26--38% under realistic conditions with noise and diverse accents--retaining only 30--45% of text capability; qualitative analysis confirms 79--90% of failures stem from agent behavior, suggesting that observed failures primarily reflect agent behavior under our evaluation setup. $τ$-voice provides a reproducible testbed for measuring progress toward voice agents that are natural, conversational, and reliable.

1. Introduction

Full-duplex voice agents must combine real-time conversation management with reliable grounded task completion, but existing evaluations largely isolate these capabilities. τ-Voice combines both dimensions with realistic audio and reports a substantial voice–text performance gap.

  • Motivation: Full-duplex voice agents must complete grounded tasks while managing interruptions, backchannels, and turn-taking in continuous time.Voice adds speech disfluencies, acoustic variation, and real-time conversational demands beyond text-based task execution.
  • Motivation: Existing benchmarks evaluate tool use or full-duplex interaction separately, leaving their interaction in realistic tasks unexplored.Tool-use benchmarks are text-only and turn-based, whereas full-duplex benchmarks generally lack real tool calls.
  • Implications: Realistic conditions remain important because non-standard accents, speech impediments, and noisy environments may systematically disadvantage voice users.The paper frames end-to-end evaluation as necessary because separate ASR, dialogue-state, and tool-use tests cannot capture compounded failures.
  • Contributions: τ-Voice combines verifiable grounded-task completion, full-duplex interaction, and realistic audio for direct voice–text comparison.The benchmark extends τ 2-bench and includes a controllable voice user simulator with diverse accents, realistic environments, and rich turn-taking.
  • Findings: 79–90% of failures in 91 analyzed simulations stem from agent behavior, suggesting the observed failures primarily reflect agent limitations under the evaluation setup.The benchmark also reports provider-specific trade-offs and no provider mastering both task completion and conversational dynamics.
  • Findings: 85% GPT-5 (reasoning) task completion contrasts with 31–51% for voice agents under Clean conditions and 26–38% under Realistic conditions.Voice agents retain only 30–45% of text capability under Realistic conditions.

2. Related Work

Prior benchmarks advance task completion, duplex interaction, or speech robustness separately, but do not combine all three evaluation dimensions. τ-Voice addresses this gap by evaluating grounded tasks alongside full-duplex behavior and realistic audio.

  • Evaluation dimensions: The comparison covers task completion, full-duplex interaction, and realistic audio environments as distinct evaluation dimensions.Realistic audio includes speaker diversity, accents, background noise, channel degradation, and disfluencies.
  • Task completion: τ-bench and τ 2-bench evaluate customer-service tasks with verifiable database outcomes, but operate entirely in text without acoustic variation or real-time constraints.τ 2-bench additionally supports dual-control settings in which users have tool access.
  • Full-duplex interaction: Full-Duplex-Bench and its second version measure pauses, backchanneling, turn-taking, interruptions, and multi-turn scenarios without real database tool calls.Their tasks remain scripted scenarios rather than grounded tool interactions.
  • Speech evaluation: Other benchmarks examine ASR robustness, vocal conversational abilities, memory, coherence, emotion, accent, or prosody, generally apart from task completion.These works reveal speech-processing gaps but do not evaluate end-to-end grounded task performance.
  • Gap: No existing benchmark combines task completion, full-duplex interaction, and realistic audio; τ-Voice addresses this gap.Its design therefore connects previously separate evaluation dimensions.

3. Methods

τ-Voice extends τ^2-bench with a controllable full-duplex voice evaluation framework that combines grounded task completion, realistic caller audio, and interaction-quality measurement. Its tick-based orchestration decouples simulation time from wall-clock time while preserving synchronized speech, interruptions, and environmental effects.

  • Framework: τ-Voice adds a full-duplex orchestrator, realistic voice user simulator, and metrics for task completion and interaction quality to τ^2-bench.Inherited task infrastructure includes instructions, tools, databases, and domain policies.
  • Full-Duplex Orchestrator: Decoupling simulation time from wall-clock time lets the user simulator use a capable LLM without real-time constraints and enables reproducible timing control.Conversational thresholds, interruption checks, audio conditions, and persona attributes are configurable.
  • Full-Duplex Orchestrator: 200ms ticks synchronize simultaneous audio exchange, buffering speech and clearing interrupted responses when the agent is cut off.The orchestrator distributes transcript text proportionally to audio duration each tick.
  • Voice User Simulator: The simulator generates speech, applies configurable personas and turn-taking policies, mixes environmental audio, and models telephony degradation.The pipeline includes background noise, burst sounds, out-of-turn speech, muffling, compression, and frame drops.
  • Evaluation: Evaluation logs turn-taking events, audio effects, and agent responses to measure verifiable task success alongside responsiveness, latency, interruption, and selectivity.Task success compares final environment state with annotated goals, while spoken communications use LLM evaluation.
  • Evaluation: A Retail trajectory illustrates how the framework identifies interruptions, no-response errors, incorrect yielding, backchannel handling, and acoustic disturbances.The example contains 8 user interruptions, 12 frame drops, 3 noise bursts, and 5 agent interruptions.

4. Experimental Setup

The experiments evaluate three audio-native providers across 278 grounded tasks in Retail, Airline, and Telecom domains under clean and realistic speech conditions. The design isolates acoustic and conversational factors while measuring both single-attempt task completion and voice interaction quality.

  • Domains and Tasks: 278 tasks span Retail, Airline, and Telecom, with Retail emphasized for its demanding slot-filling and ambiguous multi-turn requests.Retail and Telecom each contain 114 tasks, while Airline contains 50.
  • Models: Three audio-native providers are evaluated with identical voice-specific system prompts for spelling names, emails, and identifiers during information collection.The prompts also require agents to request spelling again after failed authentication.
  • Evaluation Conditions: The 3×3×2 design crosses three providers, three domains, and clean versus realistic speech conditions.Ablations on Retail isolate the effects of noise, accents, and turn-taking.
  • Evaluation Conditions: Clean uses clear American-accented speech without noise or interruptions, whereas Realistic adds diverse accents, environmental noise, channel degradation, and natural turn-taking.Intermediate conditions separately assess individual acoustic and conversational factors.
  • Simulation Parameters: Fixed seeds control noise timing and frame drops, while LLM responses remain nondeterministic; the default tick duration is 200ms.Other parameters include a 1200-second conversation limit, GPT-4.1 simulation, ElevenLabs v3 TTS, and 2-second checks.
  • Metrics: Task completion uses pass@1, defined as the proportion of single-attempt tasks whose final environment state matches a gold standard.Voice quality aggregates responsiveness, latency, interruption, and selectivity metrics.

5. Results

Voice agents show a large task-completion gap versus text models, with realistic audio worsening performance and acoustic factors affecting providers differently. Interaction-quality results reveal provider-specific trade-offs, while error analysis attributes most failures to agent behavior.

  • 5.1.1. Task Completion: 51% versus GPT-5’s 85% under Clean conditions, with voice performance falling to 38% under Realistic conditions.The best voice provider drops 34 percentage points from GPT-5 under Clean conditions, then loses an additional 12 points under Realistic conditions.
  • 5.1.1. Task Completion: Google loses 5pp from Clean to Realistic, compared with 12–14pp for xAI and OpenAI.For most providers, the larger voice-text gap comes from moving from text to Clean voice rather than from additional Realistic degradation.
  • 5.1.1. Task Completion: xAI leads task completion at 51% Clean and 38% Realistic, while Google is most robust to realistic degradation and OpenAI leads Retail at 71% Clean.xAI also reaches 58% Clean in Telecom, whereas OpenAI’s Retail score is the benchmark’s highest per-domain result.
  • 5.1.2. Impact of Acoustic Realism: Accents cause the largest average ablation drop at 10pp, followed by turn-taking at 7pp and noise at 4pp.The accent effect varies sharply by provider: xAI loses 18pp, while Google loses 1pp; the accent results are indicative because accents use TTS personas.
  • 5.1.3. Voice Interaction Quality: OpenAI has the best latency and responsiveness but worst selectivity, xAI has the best selectivity but interrupts most, and Google has the lowest responsiveness.OpenAI records 0.90s latency, 100% responsiveness, and 6% selectivity; xAI records 57% selectivity and an 84% interrupt rate; Google’s responsiveness is 69%.
  • 5.1.3. Voice Interaction Quality: No provider achieves both reliable responsiveness and appropriate restraint in real-time turn-taking.Each provider excels on different conversational dimensions while falling short on at least one.
  • 5.2. Qualitative Error Analysis: 79% of Voice-Fragile failures and 90% of Noise-Fragile failures are attributed to agents rather than the user simulator.Two raters reached 84% initial agreement and 100% agreement after resolving disagreements.
  • 5.2. Qualitative Error Analysis: Authentication is the dominant bottleneck, with failures to transcribe spelled names and emails blocking downstream actions.Other observed failures include hallucinated completions, forgotten multi-step requirements, unresponsiveness after repeated authentication failures, and timeouts from verbosity.

6. Conclusion

τ-Voice evaluates full-duplex voice agents across grounded tasks and realistic interaction conditions, while identifying important boundaries in what the benchmark measures and how faithfully it models users.

  • Accent findings are indicative rather than definitive because the study evaluates English using TTS instead of recorded speech.
  • The evaluation does not measure agent speech-generation quality, user satisfaction, or partial task success.
  • The simulator is more patient than real users and provides perfect memory and instantaneous tool calls.
  • τ-Voice evaluates 278 grounded tasks across retail, airline, and telecom domains, comparing voice-agent performance with GPT-5 (reasoning).
  • Voice agents reach 31–51% under clean conditions and 26–38% under realistic conditions, retaining only 30–45% of text capability.
  • 79–90% of failures are attributed to agent behavior rather than simulator artifacts in qualitative analysis of 91 failed simulations.

Impact Statement

The benchmark supports realistic, configurable voice-agent evaluation by combining controllable audio environments, diverse personas, and explicit full-duplex processing rules. Its open, modular design enables reproducible study across providers, languages, and domains.

  • Open and extensible: τ-Voice is open and extensible, allowing researchers to bring their own speech, agent, and VAD components.Audio effects, personas, turn-taking policies, and the simulator LLM are configurable.
  • Full-duplex processing: On interruption, buffered agent audio is cleared, truncating the in-progress response.The buffer formalism represents streamed audio across ticks and defines interruption semantics.
  • Temporal alignment: Transcript text is distributed in proportion to played audio so the simulator receives text in lockstep with speech.This prevents turn-taking decisions based on text that has not yet been spoken.
  • Evaluation representation: Overlapping full-duplex speech is converted to sequential messages using explicit ordering rules based on stopping and starting times.The rule inserts speech that occurs entirely during another turn where it stopped; otherwise, the earlier speaker goes first.
  • Realistic simulation: The simulator models diverse accents, background environments, and realistic speech events such as frame drops, burst noise, and non-directed speech.These effects are organized through configurable environment presets and scheduling parameters.

E. Voice Interaction Metrics

Voice interaction metrics evaluate whether agents respond appropriately, yield during interruptions, avoid mistimed interruptions, and ignore speech not directed at them. The simulator uses configurable timing and LLM-based decisions to produce realistic conversational behavior.

  • Metric dimensions: Voice interaction quality covers responsiveness, latency, interruption behavior, and selectivity.The metrics assess both whether agents act and whether they act at appropriate conversational moments.
  • Timing thresholds: Yielding is evaluated within a 2.0-second window after user interruption, while selectivity uses 1.0- and 2.0-second error windows.These thresholds define when an agent is counted as yielding or responding incorrectly.
  • Interruption decisions: The user simulator’s interruption decisions consider conversation history, the agent’s incomplete utterance, and whether the user has heard enough to respond.The simulator is instructed not to interrupt immediately or when the agent is asking a question.
  • Backchanneling: Backchannels are brief listener signals that occur during extended speech without responding to specific content.The simulator limits them when the user recently spoke and targets roughly one continuer per four to six sentences.
  • Simulator prompting: The simulator prompt combines global voice guidelines, persona behavior, and task-specific scenario information.It also specifies natural disfluencies, spoken formatting for special characters, and minimal verbosity.

H.1. Voice Interaction Quality: Full Metric Breakdown

The full metric breakdown groups voice interaction quality into latency, responsiveness, interruption, and selectivity domains. Clean and Realistic conditions are compared for selected metrics, while other measures use Realistic speech only.

  • Metric organization: Table 15 organizes metrics into Latency, Responsiveness, Interrupt, and Selectivity domains.Latency includes response and yield latency; responsiveness includes response and yield rates; selectivity includes backchannel, vocal-tic, and non-directed-speech correctness.
  • Condition reporting: Clean and Realistic conditions are reported separately for response latency, response rate, and agent interruption rate.Other voice interaction metrics are evaluated only under Realistic conditions.
  • Metric direction: Lower values are preferred for metrics marked with a downward arrow.The table caption defines the directionality convention for interpreting those measures.

H.2. Qualitative Error Analysis

The qualitative error analysis examines failures that arise when voice performance degrades relative to text or clean audio. It organizes observed failures into defined categories, including logical and transcription errors.

  • Failure cohorts: The analysis covers 91 failed simulations across Voice-Fragile and Noise-Fragile cohorts.The cohorts include 43 simulations failing in Clean audio after passing in text and 48 failing in Realistic audio after passing in Clean audio.
  • Cohort construction: Voice-Fragile failures compare simulations that pass in text but fail under Clean audio, while Noise-Fragile failures compare Clean-pass and Realistic-fail cases.The cohorts span 20 and 19 tasks, respectively.
  • Error taxonomy: Failures are categorized into six types based on observed failure patterns.The listed logical category includes incorrect tool calls, wrong actions, instruction violations, and lost conversation state.
  • Representative errors: Transcription errors refer to agent speech-to-text mistakes, especially when users spell names or email addresses letter by letter.Logical errors instead concern reasoning, execution, instruction following, or conversation-state tracking.

H.3. Statistical Reliability Analysis

The Retail reliability analysis uses paired permutation tests across independent runs and finds statistically significant degradation from text to Clean and from Clean to Realistic conditions. OpenAI has the strongest provider performance in both speech settings, while the task example illustrates the evaluated grounded workflow.

  • Statistical method: Two independent runs per condition use 114 paired task observations and 100k two-sided permutations, with Holm-Bonferroni correction.The analysis tests whether per-task differences between conditions are systematically non-zero.
  • Statistical results: All providers show significant text-to-Clean drops, with the narrowest gap between GPT-4.1 at 76.3% and OpenAI Clean at 67.5% (∆= −8.8pp, p = 0.032).
  • Statistical results: All three providers show significant Clean-to-Realistic degradation: Google and xAI each decline 13.2pp, while OpenAI declines 24.6pp.The reported p-values are 0.026 for Google, below 0.001 for OpenAI, and 0.044 for xAI.
  • Provider comparison: OpenAI leads provider performance under both Clean conditions at 67.5% and Realistic conditions at 43.0%.Under Clean, xAI reaches 50.0% and Google 42.1%; under Realistic, xAI reaches 36.8% and Google 28.9%.
  • Task example: The Retail example uses a grounded task requiring a puzzle exchange and shipping-address corrections under authentication and user-memory constraints.The example is presented as a complete evaluation-criteria and conversation-transcript case, corresponding to Figure 4.

I.1.2. EVALUATION CRITERIA

Task success is evaluated from the final database state and natural-language assertions, requiring several correctly argued write actions. Task 41 demonstrates how authentication, tool execution, interruptions, noise, and agent errors can determine whether a grounded voice task succeeds.

  • Evaluation criteria: Task success is assigned reward = 1.0 when the final database state and natural-language assertions satisfy the evaluation criteria.
  • Evaluation criteria: Only the final database state matters, so the exact sequence of read calls may vary even though the required writes must use correct arguments.
  • Conversation behavior: An attempted lookup by email returned “User not found” before the agent found the user through name and zip code.
Loading 2603.13686v1…