PaperlayerView exact revision on arXiv

Source-linked AI summary

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose, Fanny Riols, Hoang H. Nguyen, Raghav Mehndiratta, Lindsay Devon Brin, Joseph Marinier, Hari Subramani, Anil Madamala, Sridhar Krishna Nemala, Srinivas Sunkara

arXiv:2605.13841v2cs.SDcs.AIcs.CLcs.LG

TL;DR

Voice-agent benchmarks lack unified ways to simulate realistic conversations and measure voice-specific quality. EVA-Bench addresses both with validated bot-to-bot audio simulation and composite accuracy and experience metrics, finding that accuracy and experience remain jointly unsatisfied while the pass@k–pass^k gap is 0.44 on EVA-A.

  • Problem

    Existing text-oriented benchmarks do not jointly simulate realistic voice conversations and measure voice-specific quality across distinct speech-related failure modes.

  • Method

    EVA-Bench combines validation-gated bot-to-bot audio simulation with architecture-agnostic EVA-A and EVA-X metrics and multi-trial consistency measures.

  • Results

    Accuracy and experience remain jointly unsatisfied across evaluated systems, while the pass@k–pass^k gap is 0.44 on EVA-A.

  • Takeaways & Limitations

    Single-trial evaluation scores systematically overstate deployment-grade quality regardless of architecture.

  • Takeaways & Limitations

    The authors acknowledge that EVA-Bench has several limitations.

Abstract

from arXiv · show

Voice agents, artificial intelligence systems that conduct spoken conversations to complete tasks, are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues, with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy), capturing task completion, faithfulness, and audio-level speech fidelity; and EVA-X (Experience), capturing conversation progression, spoken conciseness, and turn-taking timing. Both metrics apply to all major agent architectures, enabling direct cross-architecture comparison. EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and pass@1, pass@k, pass^k measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k--pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean $Δ$ up to 0.314). We release the full framework, evaluation suite, and benchmark data under an open-source license.

1 Introduction

EVA-Bench addresses the distinct simulation and measurement challenges of evaluating voice agents by generating validated multi-turn audio conversations and measuring accuracy and experience across voice-specific failure modes. Its framework enables direct comparison across audio-native and cascade agents while supporting controlled acoustic perturbations and repeated trials.

  • Motivation: Voice agents face evaluation challenges absent from text because speech is ephemeral, timing-sensitive, and exposed to varying acoustic conditions.These properties create voice-specific failure modes that transcript-only evaluation may miss.
  • Simulation challenge: Realistic simulation requires complete task-oriented multi-turn interactions with natural spoken user behavior and real-world acoustic variation.Full conversations reveal whether agents recover from misunderstandings and resolve tasks end-to-end.
  • Measurement challenge: Existing measures can miss policy violations and incorrect spoken entities even when agents complete tasks or call the correct tools.Examples include wrong confirmation codes or monetary amounts that may be catastrophic in production.
  • EVA-Bench: EVA-Bench conducts fully automated bot-to-bot audio simulation over dynamic multi-turn dialogues with validation-gated quality control across repeated trials.The framework automatically validates simulated conversations to maintain consistency before scoring.
  • EVA-Bench: EVA-A measures task completion, policy and tool-output faithfulness, and audio-level entity fidelity, while EVA-X measures progression, spoken conciseness, and turn-taking timing.Together, the composite metrics expose voice-agent failures beyond task completion and latency.

2 Related Work

Prior voice-agent benchmarks typically isolate speech or conversational components instead of evaluating end-to-end behavior. Existing simulation and measurement approaches also leave gaps in realistic, validated multi-turn interaction and comprehensive conversational-quality assessment.

  • Benchmark scope: Most voice benchmarks target individual components—STT robustness, TTS quality, or conversational dynamics—rather than end-to-end voice-agent behavior.Table 1 is described as comparing benchmark features and summarizing these gaps.
  • Conversation Simulation: Prior simulation benchmarks are scripted, static, or single-turn, while τ-Voice and FDB-v2 provide live bot-to-bot multi-turn conversations without automated simulator validation across trials.FDB-v1/v1.5 are heavily scripted without tool use; VoiceAgentBench uses static TTS queries; FDB-v3 remains single-turn despite improved realism.
  • Voice Agent Quality Measurement: Existing quality metrics remain narrow: VoiceAgentBench measures tool-use accuracy, τ-Voice adds turn-taking, and FDB-v3 adds transcript-level response quality while omitting other fidelity and conversational-efficiency dimensions.The cited omissions include conversational quality, faithfulness, policy faithfulness, audio-level entity accuracy, efficient conversational progress, and cognitive-load avoidance.

3 Methodology

EVA-Bench constructs repeatable enterprise voice-agent scenarios and evaluates them through automated audio conversations, validation, controlled perturbations, and layered metrics. Its methodology separates Accuracy, Experience, and diagnostic measurements, with dimension-level passing requiring every constituent metric to meet its threshold.

  • Data Design: EVA-Bench spans three enterprise domains and defines each scenario with a constrained user goal, persona, scenario database, ground-truth state, and decision tree.Scenarios target high-contact, task-oriented interactions and are designed to expose voice-specific failure modes.
  • Multi-Turn Conversations: Fully automated bot-to-bot conversations connect a user simulator and agent over live audio WebSockets, supporting identical evaluation conditions for cascade and audio-native architectures.The simulator receives the user goal, decision tree, and persona as inputs.
  • Controlled Perturbations: Controlled perturbations independently vary accent, background noise, connection quality, personality, and speaking style, while validation checks user behavioral and speech fidelity before scoring.Independent control allows perturbations to be isolated or combined and detects simulator deviations that could corrupt evaluation.
  • Accuracy Metrics: EVA-A measures task completion, faithfulness, and speech fidelity, including exact database-state matching and accurate spoken reproduction of high-stakes entities.Faithfulness accounts for architecture-specific audio understanding responsibility, while S2S speech fidelity verifies key entities when no intended text exists.
  • Experience Metrics: EVA-X measures conversation progression, conciseness, and turn-taking, assessing forward progress, spoken brevity, and timing through interruption- and latency-sensitive functions.Turn-taking scores differ for interrupted, user-interrupted, uninterrupted, and tool-call turns.
  • Metric Aggregation: A conversation passes Accuracy only when task completion = 1.0, faithfulness ≥0.5, and speech fidelity ≥0.95, and passes Experience only when turn-taking ≥0.8, progression ≥0.5, and conciseness ≥0.5.Dimension scores use metric-specific thresholds and require every metric in the dimension to pass.

4 Empirical Analysis

EVA-Bench’s empirical analysis shows a sparse Accuracy–Experience frontier, substantial gaps between peak and reliable performance, architecture-dependent robustness failures, and measurement dominated by genuine behavioral differences rather than judge noise.

  • Measurement Validity: Trial stochasticity dominates variance across all metrics and models, while judge stochasticity is minimal, with p < 0.0001 for all 16 model × metric combinations.The judge meets human inter-annotator agreement on all four metrics, indicating that score differences reflect agent behavior rather than measurement artifacts.
  • Accuracy–Experience Frontier: No evaluated system clears 0.5 on both EVA-A pass@1 and EVA-X pass@1, while only GPT-Realtime-1.5 (0.47, 0.57) clears 0.4 on both.The pass@1 Pareto frontier contains four systems spanning two disjoint regions.
  • Accuracy–Experience Frontier: Turn-Taking separates architectures most clearly, with mean EVA-X scores of 0.28–0.58 for cascade and 0.82–0.83 for S2S.Conciseness and Conversation Progression show no comparable separation; hybrid systems fall within the cascade EVA-X range (0.000, 0.029).
  • Accuracy–Experience Frontier: No cascade system exceeds 0.25 on both accuracy and experience: higher-accuracy systems have tool-call latencies above 5 s, whereas faster systems stay below 2.7 s.The reported confidence intervals do not overlap, supporting a consistent cascade accuracy–experience trade-off.
  • Consistency Analysis: The median pass@k–pass^k gap is 0.44 on EVA-A and 0.24 on EVA-X, showing that peak performance substantially exceeds reliable performance across all 12 systems.Under pass^k, even the strongest systems fall well below peak, so single-trial scores overstate deployment-grade reliability.
  • Robustness Analysis: Cascade systems are most vulnerable on accuracy metrics, whereas S2S systems suffer most on experience metrics under accented speech, noise, and combined perturbations.Under the combined condition, cascade task completion drops a mean 19 points, while the S2S model mean remains within 5 points.

5 Conclusion

EVA-Bench jointly evaluates voice-agent simulation fidelity and measurement comprehensiveness across architectures, revealing sharp experience-quality, reliability, and acoustic-robustness differences. Its open-source, configurable design supports matched evaluation and deployment-specific adaptation.

  • Framework: EVA-Bench combines validation-gated bot-to-bot simulation, architecture-agnostic EVA-A and EVA-X metrics, and pass@1, pass@k, and pass^k consistency measures.These components enable comparison across cascade, hybrid, and S2S voice agents under identical conditions.
  • Findings: Best-performing cascade and S2S systems achieve comparable accuracy, but experience quality diverges sharply, driven almost entirely by turn-taking.The architecture-linked gap appears on EVA-X rather than accuracy.
  • Findings: 0.44 is the median pass@k–pass^k gap on EVA-A, showing that single-trial evaluation systematically overstates deployment-grade quality across architectures.Peak and reliable performance diverge substantially across all evaluated systems.
  • Findings: Cascade and S2S systems degrade differently under acoustic perturbation, exposing architecture-specific robustness patterns.These perturbation-specific degradation patterns are dimensions existing benchmarks leave unmeasured.
  • Framework: EVA-Bench is fully open-source and extensible, with modular metrics, configurable pass criteria and perturbations, and support for adding domains through agents and scenarios.It enables controlled, matched evaluation as organizations consider migrating from cascade architectures while S2S models mature.

Limitations … B.3 Turn Detection Configurations

EVA-Bench’s limitations concern judge reliability, simulation and infrastructure fidelity, experimental coverage, and reproduction cost. The appendices define core architecture and evaluation terms, document model-serving choices, and explain that turn detection uses framework defaults with configurable alternatives.

  • Limitations: LLM-based judges may favor stylistically familiar outputs, especially when evaluated systems share their model family, while LALM judges may be less reliable than text-only judges.These risks affect the reliability and comparability of automated scoring.
  • Metrics: Binary task-completion scoring grants no partial credit, potentially limiting discrimination between systems that differ mainly in terminal sub-goal robustness.Agents satisfying all but one sub-goal can therefore receive the same task-completion outcome as less complete agents.
  • Framework: Bot-to-bot simulation may not transfer to production callers, while audio conversion, interface timing, and imprecise event timestamps can distort fidelity and response-speed measurements.The benchmark also relies on commercial APIs, declarative mocks, and costly repeated trials; k=5 validated trials across 213 scenarios and 12 systems require several hundred model calls per trial.
  • Framework: The simulator is English-only, may under-represent natural disfluencies, and does not systematically generate interruptions, under-stressing turn-taking and barge-in robustness.Production latency-reduction strategies are also omitted, so reported latency may not represent achievable production lower bounds.
  • Experiments: The experiments cover 12 systems across three architectural classes, use default turn detection and unoptimized prompts, and test only one accent and one noise environment.These choices preserve fair baselines but limit broad inferential claims and may make observed robustness differences environment-dependent.
  • B Experiment Parameters: The experiment parameters specify H100-based self-hosting, model sampling settings, ElevenLabs API-hosted models, and separate documentation for prompts, hardware, and third-party packages.Gemma and Qwen configurations disable thinking mode, while ElevenLabs uses Scribe-v2.2-Realtime, Gemini-3-Flash, and TTS-Conversational-v3-Alpha with default agent parameters.
  • B.3 Turn Detection Configurations: Turn detection uses each framework’s default configuration because exact parameters and strategies are difficult to standardize, while EVA-Bench exposes available settings through its CLI except for separately configured ElevenAgents.Defaults include Pipecat VAD/transcription start with AI turn detection, OpenAI server VAD, normal ElevenAgents eagerness, and Gemini automatic VAD.

B.4 Experiments Compute Resources

EVA-Bench experiments require budgeting across simulator, evaluated-agent, and infrastructure costs. Full runs are especially resource-intensive when evaluating multiple proprietary models in parallel.

  • Cost Components: EVA-Bench incurs costs from the hosted ElevenAgents user simulator, proprietary-agent inference, and supporting infrastructure.ElevenAgents charges per simulated interaction; proprietary models add inference, speech-to-text, and text-to-speech costs.
  • Cost Components: Proprietary models are billed per token for LLM inference, per second of audio for STT, and per character for TTS.These charges apply whenever the evaluated agent is proprietary.
  • Planning: Full EVA-Bench runs should be planned as non-trivial compute investments, particularly when benchmarking multiple proprietary models simultaneously.Researchers are advised to arrange parallelization and cost budgeting before large-scale experiments.

C Data … D.1.1 Conversation Valid End.

EVA-Bench constructs domain-specific, multi-intent, and adversarial scenarios from jointly consistent synthetic records, then validates simulations and filters infrastructure failures before judging. Its datasets target transcription, policy, branching-flow, authentication, and compositional challenges across enterprise workflows.

  • C Data: Three enterprise datasets target distinct voice-agent difficulties while requiring accurate transcription of structured named entities.Airline CSM tests temporal reasoning and policy adherence; ITSM tests branching flows and tiered authentication.
  • C.1 Workflows: Scenarios cover Single-Intent, Multi-Intent, and Adversarial dimensions, including up to four concurrent workflows and hard policy constraints under social pressure.Multi-intent cases test compositional completion without context loss, while adversarial cases test refusal of ineligible requests.
  • C.2 Data Generation Pipeline: SyGra jointly generates each scenario’s user goal, initial database, and expected final database state from a workflow graph using GPT-5.2.Joint consistency prevents mismatches between requested entities, backend state, and ground-truth outcomes.
  • Synthetic Data Generation with SyGra: Multi-intent scenarios merge single-intent records, while adversarial scenarios are hand-designed around policy boundaries and verified against tool-executor behavior.The adversarial design ensures violations are achievable but detectable by a correctly behaving agent.
  • Human Review: Four review checks verify policy consistency, unique correct resolutions, internally consistent final states, and clearly specified adversarial violations before ambiguous records are removed.Records identified as ambiguous or inconsistent were corrected or discarded.
  • Frontier Model Stress Testing: Three frontier models were stress-tested on text-only scenarios, and zero task-completion cases were manually investigated for model error or dataset defects.The process supports high confidence that full-audio task-completion failures reflect real agent errors rather than evaluation artifacts.
  • D User Simulator Details: The user simulator combines Scribe v2.2 Realtime, GPT-5.1, and Eleven V3 Conversational, with four agents spanning English and French accents and two genders.It is prompted to stay on task, communicate required named entities clearly, and end calls after completion or when success is clearly unlikely.
  • D.1.1 Conversation Valid End.: A deterministic valid-end gate admits conversations only when the agent fails to respond or the user invokes end_call; other simulations are rerun before LLM judging.The gate catches WebSocket failures, failed starts, and simulator timeouts, preventing malformed simulations from consuming judge calls.

D.1.2 User Behavioral Fidelity. … E.4 Accuracy Metrics

EVA-Bench validates simulator behavior and speech, extracts architecture-aware evidence from synchronized logs, and uses validated judges and normalized metrics for equitable scoring. Its validation gates identify user-induced corruption, while judge agreement and pipeline-specific observability support reliable evaluation across architectures.

  • D.1.2 User Behavioral Fidelity.: User-behavior validation reruns conversations receiving an overall binary rating of 0 after a judge checks five corruption types.The judge analyzes extra modifications, premature ending, missing information, duplicate modifications, and decision tree violations.
  • D.1.2 User Behavioral Fidelity.: Across 714 flagged trials, premature ending was most frequent (63.9%), followed by decision tree violations (52.9%) and missing information (28.6%).Extra modifications occurred in 1.3% of trials, duplicate modifications in 0%, and 42.4% exhibited at least two corruption types.
  • D.1.3 User Speech Fidelity.: User speech fidelity requires accurate communication of key entities and major content, using a 3-point scale rather than exact word-level matching.The inherited judge validation achieved κ = 0.777, 95% CI [0.704, 0.835].
  • E Metric Details: All EVA-A and EVA-X scores are normalized to [0, 1], with pass@k and pass^k requiring all category metrics to exceed their respective thresholds simultaneously.The complete metric set is summarized in Table 11.
  • E.1 Log Processing and Variable Extraction: Three timestamped log streams are merged and replayed deterministically to recover per-turn intended text, transcriptions, audio timings, and a linearized conversation trace.Turn boundaries are driven by user audio_start events, with special handling for noise, buffered speech, and interruptions.
  • E.2 Equitable Evaluation Across Cascade, Hybrid, and S2S Architectures: Pipeline-aware metric variants use the most faithful observable view: internal text signals for cascade, intended user text plus assistant ASR for S2S, and a mixed view for hybrid systems.This prevents cascade STT/TTS errors from being attributed to the LLM while scoring S2S speech understanding and synthesis as system responsibilities.
  • E.3 Judge Development and Validation: Judge development combines targeted failure-mode generation, multi-model consensus labeling, prompt refinement, candidate evaluation, and held-out human validation.Three frontier models label generated samples by consensus, with human review for disagreements.
  • E.3 Judge Development and Validation: Linguist–judge agreement ranges from 0.777 to 0.845 across four metrics, meeting or exceeding linguist–linguist agreement in every case.This supports the reliability of the finalized judge prompts and selected models.

E.4.1 Task Completion … F.1 Key Entity Transcription Accuracy & Task Completion

The evaluation combines deterministic task-outcome checks with judge-based measures of faithfulness, speech fidelity, conversation progression, and turn-taking, while analyzing how entity transcription affects task completion. These metrics distinguish outcome correctness from path quality, audio realization, conversational efficiency, and architecture-sensitive latency.

  • E.4.1 Task Completion: Task Completion verifies whether the agent’s committed database changes match the expected scenario end state.It is a deterministic, code-based binary check with no LLM in the loop.
  • E.4.1 Task Completion: Task Completion is 1.0 when expected and actual database hashes match, and 0.0 otherwise.Authentication is verified separately, while the database comparison measures outcome independently of the conversational path.
  • E.4.2 Faithfulness: Faithfulness evaluates whether the assistant remains grounded in available information, policies, and instructions, penalizing errors even when the task ultimately succeeds.Its five disjoint dimensions cover fabricated tool parameters, misrepresented tool results, policy violations, failed disambiguation, and hallucination.
  • E.4.3 Speech Fidelity: Speech Fidelity checks whether spoken audio correctly conveys intended or source entities, with cascade and hybrid systems compared against intended text and S2S systems evaluated against trace context.The metric targets errors in digits, letters, amounts, numbers, codes, names, and other entities, while excluding faithfulness correctness for S2S.
  • E.5.1 Conversation Progression: Conversation Progression measures efficient movement toward the user’s goal, including avoiding unnecessary tool calls, information loss, redundant statements, and poor questions.It evaluates efficiency rather than correctness, so looping or re-asking can score poorly even when the final outcome is correct.
  • E.5.1 Conversation Progression: Conversation Progression aggregates dimension ratings by issue breadth: overall rating 3 requires no flagged dimensions, rating 2 allows one or two rating-2 dimensions, and rating 1 follows any rating-1 dimension or three or more flagged dimensions.This differs from Faithfulness’s minimum aggregation because efficiency is sensitive to the spread of issues across dimensions.
  • Tool-Call-Aware Turn-taking Score: Tool-call-aware Turn-Taking applies different latency thresholds to turns with and without tool execution, separating infrastructure delay from conversational responsiveness.The framework introduces this distinction because uniform thresholds would unfairly penalize unavoidable tool-call latency.
  • Tool-Call-Aware Turn-taking Score: Pearson r remains at least 0.910 across the τtt ∈ [0.50, 0.95] range, reaching 0.998 at τtt = 0.75 and 0.995 at τtt = 0.85.These results indicate that the per-system pass@1 vector is stable around the production anchor τtt = 0.8.

F.2 Faithfulness & Task Completion

Across clean trials, only 14.5% achieve both task completion and faithfulness, showing that task completion alone is a weak proxy for overall agent accuracy. Faithfulness failures occur nearly uniformly across task outcomes, indicating that the two dimensions capture distinct aspects of behavior.

  • Joint evaluation: 14.5% of clean trials achieve both task completion and faithfulness across all evaluated systems.The analysis covers 12,780 clean trials across the evaluated systems.
  • Joint evaluation: 38.3% versus 37.6% of faithfulness failures occur across the two task-outcome categories, respectively.The near-uniform distribution reinforces that task completion and faithfulness capture distinct aspects of agent behavior.
  • Interpretation: Task completion alone is a weak proxy for overall agent accuracy because it does not capture faithfulness.The confusion matrix jointly evaluates task completion and faithfulness, with faithfulness defined as a score of 1.0 versus below 1.0.

F.3 Patterns Across Pipeline Architectures

Across the evaluated systems, S2S pipelines tend to lead cascades narrowly on responsiveness but trail on policy adherence. These descriptive patterns are based on clean runs and should not be generalized beyond the evaluated systems.

  • Observed patterns: S2S systems tend to lead cascades narrowly on on-time turns and conversation completion, but trail on authentication success and policy adherence.The four facets are on-time turns, conversation completion, authentication success, and policy violations.
  • Observed patterns: 61% of conversations across all pipelines are flagged with at least one policy violation.Examples include unauthorized write actions and fabricated policies absent from instructions.
  • Analysis protocol: Pipeline-type means use unweighted averages of per-system means across domains, with 95% percentile bootstrap intervals from 10,000 resamples.All numbers use clean, unperturbed runs.
  • Analysis protocol: The analysis covers 7 cascade, 3 S2S, and 2 hybrid systems across three domains, yielding 1,065 clean conversations per system.Hybrid results are shown as individual system points because only two hybrid systems were evaluated.
  • Limitations: Because each pipeline type contains few systems, the findings are descriptive observations rather than inferential confidence claims or generalizations beyond the evaluated systems.Pipeline-level differences can conceal within-class exceptions, particularly among cascades.

F.3.1 Observations … F.6 Cross Domain Variability

Across domains, S2S systems generally lead pipeline-level performance, but cascade systems can approach them on selected metrics when completion failures are separated from conditional ability. The analysis attributes remaining gaps to authentication timeouts, entity mispronunciation, and system-specific turn-taking overhead, while also measuring cross-domain dispersion.

  • F.3.1 Observations: 0.28 is the lowest cascade policy-violation rate, below GPT-Realtime-1.5’s lowest S2S rate of 0.69.The policy-violation facet is the most variable within both pipeline classes.
  • F.3.2 Authentication: completion vs. ability: 0.94 is the conditional authentication rate for Nova-3 + GPT-5.4 + Sonic 3, matching GPT-Realtime-1.5 after rising from 0.85 raw.Ink-whisper + Haiku-4.5 + Sonic 3 rises from 0.65 to 0.86, while GPT-Realtime-1.5 finishes 99% and authenticates correctly 93% when finished.
  • F.3.3 Inactivity-timeout failures: 52–68% of timeouts are preceded by short turns for five of seven cascade systems, often following confirmation tokens without an agent follow-up.Spelled-content hangs persist across all pipelines and predominantly occur during authentication.
  • F.4 Speech Fidelity: The majority of flagged turns across all models involve entity mispronunciation, including character substitutions, omissions, insertions, and phonetic confusions.Examples include GPT-Realtime-1.5 dropping digits, Kokoro dropping letters, and Gemini-3.1-Flash-Live inserting text into codes.
  • F.5 Turn-Taking & Response Speed: 0.148 is the largest observed early-turn rate, while most systems remain below 0.06; turn-taking tracks on-time and late rates more closely.Early turns have latency < 200 ms, and late turns have latency ≥2.75 s, or ≥4 s for tool-call turns.
  • F.5 Turn-Taking & Response Speed: Every system shows higher latency with tool calls, ranging from roughly 0.1–0.2 s for the fastest cascades to several seconds for Ultravox-Realtime and Gemini-3-Flash + Gemini-3.1-Flash-TTS.The non-overlapping confidence intervals indicate consistent, system-specific tool-call slowdown.
  • F.6 Cross Domain Variability: Cross-domain variability is quantified as the sample standard deviation of each system–metric pair across CSM, ITSM, and HR.These statistics quantify how much pooled headline numbers conceal domain dispersion.

F.6.1 Metric-level findings … G.2 User Personas

EVA-Bench finds that speech fidelity is highly stable across domains, whereas faithfulness, task completion, and conversation progression vary more substantially. Its domain-wise results and perturbation framework expose model- and deployment-condition sensitivity through behavioral, accent, and audio-degradation variations.

  • F.6.1 Metric-level findings: Speech Fidelity has cross-domain standard deviations of 0.002–0.013 across all twelve systems, with a mean of 0.007 and point estimates ranging from 0.913–0.996.The metric is determined by the TTS/S2S audio path and depends less on conversational content.
  • F.6.1 Metric-level findings: Faithfulness and Task Completion have mean cross-domain standard deviations of 0.087 and 0.064, respectively, making accuracy more domain-coupled than speech rendering.Faithfulness reaches single-system standard deviations of 0.184 and 0.166.
  • F.6.1 Metric-level findings: Conversation Progression is the noisiest experience metric at mean standard deviation 0.052, while Conciseness is most stable at 0.020 and Turn-taking averages 0.042.Conversation Progression peaks at 0.125, whereas Conciseness never exceeds 0.043.
  • F.6.2 Model-level findings: GPT-Realtime-1.5 combines a pooled Task Completion mean of 0.739 ± 0.045 with a per-domain standard deviation of 0.174.The per-domain value is the largest Task Completion standard deviation in either table.
  • F.6.2 Model-level findings: Scribe-v2.2-Realtime + Gemini-3-Flash + TTS-Conversational-v3-Alpha shows the same pooled-versus-domain pattern for Faithfulness, with pooled 0.457 and standard deviation 0.184.These results show that pooled tables can conceal substantial across-domain variation.
  • F.6.3 Accuracy and Experience results per domain: EVA-Bench reports per-domain accuracy and experience results for CSM, HR, and ITSM, alongside cross-domain variability tables for both metric families.Accuracy appears in Tables 26–29, while experience appears in Tables 30–33 under clean-audio conditions.
  • G Perturbation Analysis: EVA-Bench’s robustness tests apply composable user-simulator perturbations along behavior, accent, and audio-degradation axes, while the evaluated agent receives no special configuration.Behavioral variants, four non-native English accents, background noise, and other degradation mechanisms model deployment conditions beyond clean audio.
  • G.1 Behavioral Perturbations in EVA-Bench: The user-persona framework includes a default direct caller plus aggressive_impatient, elderly_slow, and forgetful_- disorganized personas that vary speed, interruptions, pauses, frustration, and disfluencies.These personas stress-test non-canonical caller patterns, including rapid interruptions, slow delivery, and mid-utterance corrections.

G.3 Perturbation Experiment · H Measurement Reliability: Variance Decomposition and Trial Count Justification

The perturbation experiment compares accent, background noise, and combined conditions against clean baselines using paired scenario-level deltas and permutation testing. The supplied passages specify the evaluation design and statistical controls but provide no separate evidence for the measurement-reliability section.

  • G.3 Perturbation Experiment: Each perturbation condition is compared with a paired clean baseline using scenario-level mean deltas, averaging 3 perturbation trials and 5 clean-baseline trials.Comparisons are conducted for each model, metric, and domain or pooled across domains.
  • H Measurement Reliability: Variance Decomposition and Trial Count Justification: The provided passages contain no substantive methods, results, or trial-count justification for H Measurement Reliability: Variance Decomposition and Trial Count Justification.Accordingly, that merged subsection cannot be summarized from the supplied evidence.
  • G.3 Perturbation Experiment: Paired sign-flip permutation tests evaluate whether scenario-level perturbation deltas differ from zero.The method is designed for bounded, ordinal metrics without distributional assumptions.
  • G.3 Perturbation Experiment: 10,000 permutations independently flip each scenario delta with probability 0.5 to construct the null distribution and calculate two-sided p-values.The null hypothesis is E[δ] = 0, and p-values use the fraction of permutations with absolute means at least as large as observed.
  • G.3 Perturbation Experiment: Holm–Bonferroni correction controls multiple comparisons across the three perturbation conditions within each model × metric × domain combination.Effects are considered significant when the corrected p-value is below α = 0.05.
  • G.3 Perturbation Experiment: Tables 34–36 report pooled mean ∆(perturbed −clean) for accent, background noise, and accent + background noise across models and metrics.Significance markers reflect Holm–Bonferroni-corrected sign-flip permutation tests within each model–metric family.
  • G.3 Perturbation Experiment: Figure 8 summarizes perturbation effects across EVA-A, EVA-X, Task Completion, Transcription Accuracy (Key Entities), and Conversation Progression.Bars show mean deltas from clean trials, with negative values indicating drops under perturbation and whiskers representing 95% percentile bootstrap CIs.

H.1 Variance decomposition · H.2 Justification of trial count

Variance analyses show that trial stochasticity dominates score variation, while scenario effects are metric-dependent and strongly model-specific. Trial-count resampling supports k=3 for perturbation experiments because estimates stabilize closely to the k=5 anchor, while clean evaluations retain k=5.

  • H.1 Variance decomposition: Trial variance contributed 40–80% of observed variance for judge-graded metrics and 53–100% for deterministic metrics, exceeding scenario variance on 33 of 34 model–metric combinations.The only exception involved a 1.5-point overlap for Faithfulness in one model.
  • H.1 Variance decomposition: Scenario ICC ranged from near zero to 47%, with Task Completion at 33–47% and Faithfulness at 21–42% across models.These metrics showed the highest and most consistent scenario contributions.
  • H.1 Variance decomposition: The model × scenario interaction was significant in all 22 domain × metric combinations, explaining 4–18% of total variance with a median of 11%.Scenario difficulty was therefore ranked differently across models, while ICCscenario remained low overall at median 0.08 and range 0.00–0.27.
  • H.1 Variance decomposition: Trial variance exceeded judge variance in all 16 model × metric combinations, with permutation-test p < 0.0001 in every case.The binomial sign test likewise found trial SD exceeded judge SD in most scenarios for 13 of 16 model–metric combinations.
  • H.2 Justification of trial count: For each trial count k ∈ {1, 2, 3, 4}, 2,000 Monte Carlo subsamples independently sampled k trials per scenario, while k=5 used the single full-data anchor.The resulting model-level estimates measured sensitivity to using fewer than the five available trials.
  • H.2 Justification of trial count: Per-turn metrics had 95% CI width below 0.02 at k=1 across all models, whereas conversation-level and pass metrics narrowed from 0.05–0.09 at k=1 to roughly 0.02–0.03 at k=3.The metrics were Speech Fidelity, Conciseness, Turn-Taking, EVA-A pass@1, EVA-X pass@1, Task Completion, Faithfulness, and Conversation Progression.
  • H.2 Justification of trial count: At k=3, median 95% CI width was at most 0.034 for every metric, and at least 97.3% of subsamples fell within 0.02 of the k=5 anchor on every metric.These results indicate close agreement between three-trial estimates and the five-trial reference.
  • H.2 Justification of trial count: Cross-model gaps on headline pass metrics spanned 0.1–0.6, while k=3 trial-count uncertainty was approximately 0.03, below 10% of the smallest interesting effect size.Accordingly, perturbation experiments used k=3 to cover more scenarios, while clean evaluation retained k=5.

I Data Workflows … P Human Annotations

The supplied passages describe EVA-Bench’s enterprise workflow coverage, scenario specification, and a detailed airline rebooking example. They emphasize domain complexity, deterministic user behavior, explicit ground truth, and policy-constrained multi-turn interaction.

  • I.2 Healthcare HRSD Workflows: 83 scenarios span 12 healthcare HR service workflows backed by 47 tools, with average complexity of 8.7 expected tool calls.Single-intent workflows average 5.0 calls, while triple-intent workflows reach up to 18.
  • I.3 Enterprise ITSM Workflows: 80 IT service-desk scenarios cover 21 workflows and six categories, including 29 single-intent, 14 double-intent, 14 triple-intent, 14 quadruple-intent, and 9 adversarial cases.Incident flows branch between troubleshooting resolution and escalation to a ticket.
  • J Scenario Examples: Each evaluation record combines a deterministic user goal and decision tree, caller persona, scenario database, and ground truth for the expected final database state.These components jointly constrain simulation behavior and enable state-based evaluation.
  • Expected Flow, Database & Ground Truth: The expected airline flow applies a $75 same-day change fee, waived for Gold+, and searches earlier flights subject to availability.The ground truth tracks the reservation before and after the change, while Table 45 lists flight eligibility against the user’s criteria.

Q Ethics & Reproducibility … R Future Work

EVA-Bench uses fully synthetic data and publicly released implementation materials to support responsible, reproducible evaluation, while acknowledging limitations in safety, coverage, language, and API-dependent replication. Future work targets multilingual and multi-agent coverage and more realistic user simulation.

  • Ethics Statement: All evaluation scenarios are fully synthetic, using no real caller data, recordings, or personally identifiable information.The simulation tools also follow copyright and privacy guidelines.
  • Ethics Statement: EVA-Bench cannot guarantee that evaluated models will avoid harmful or biased outputs, motivating content filtering and bias detection before production deployment.This limitation applies despite the framework’s ability to support large-scale evaluation.
  • Ethics Statement: Current coverage is limited to English and a restricted set of domains, which may reinforce representational biases in audio AI systems.The authors encourage expansion to more diverse languages, accents, cultural contexts, and domains, and are extending the benchmark to other languages.
  • Reproducibility Statement: The authors publicly release EVA-Bench code, configurations, evaluation scripts, scenario data, documentation, setup instructions, environment specifications, and reproduction scripts under an anonymized open-source repository.These materials are intended to reproduce all reported evaluations.
  • Reproducibility Statement: Implementation details specify model configurations, domain data distributions, judge prompts, metric definitions, thresholds, and LLM-as-judge configurations for replication across research groups.These details are provided in Appendices B, C, M, and E.
  • Reproducibility Statement: Full score reproduction requires commercial model APIs, so results may vary across API versions and deployment configurations.The paper discusses this variability in Section 5.
  • R Future Work: Future work extends scenario coverage beyond English, supports multi-agent architectures, and improves the user simulator’s production of interruptions, disfluencies, and broader conversational behavior.Target architectures include supervisor-worker patterns, orchestration frameworks, and agentic planning or delegation.
Loading 2605.13841v2…