EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

Motivation

• Voice agents face speech-specific failure modes because audio is ephemeral and linear, timing affects naturalness, and acoustic conditions vary across callers, making text-agent evaluation insufficient.Rigorous evaluation therefore must address both how conversations are simulated and how quality is measured.Source: §1 · p. 1

Simulation challenge

• Realistic simulation requires complete task-oriented multi-turn dialogues, natural spoken behavior, varied accents and background noise, and validation that prevents simulator errors from being mistaken for agent failures.Invalid, drifting, or inconsistent simulated users undermine downstream evaluation by adding simulator variance.Source: §1 · p. 1

Measurement challenge

• Voice-agent measurement must detect failures beyond task completion and turn-taking, including policy violations, incorrect spoken entities, stalled progress, repetition, and excessive spoken options.These requirements motivate broader evaluation dimensions than existing benchmarks provide.Source: §1 · p. 2

Architectural scope

• Because cascade, speech-to-speech, and hybrid audio-native systems use fundamentally different mechanisms, benchmarks must evaluate them on equal footing for meaningful comparison.Cascade systems chain STT, LLM, and TTS components, whereas audio-native systems process audio directly or through a large audio language model paired with TTS.Source: §1 · p. 2

EVA-Bench

• EVA-Bench combines validation-gated bot-to-bot audio simulation with EVA-A for task completion, faithfulness, and audio-level entity fidelity, and EVA-X for progression, conciseness, and turn-taking timing.The framework includes three enterprise datasets totaling 213 scenarios, controlled acoustic perturbations, and direct comparison across audio-native and cascade agents.Source: §1 · p. 2

Overall gaps

• Existing benchmarks focus on STT robustness, TTS quality, or conversational dynamics instead of end-to-end voice-agent behavior.Table 1 compares benchmark features and motivates the gaps addressed by EVA-Bench.Source: §2 · p. 3

Conversation Simulation

• EVA-Bench addresses simulation gaps with multi-trial bot-to-bot, task-oriented multi-turn conversations, automatic simulator validation, and controlled perturbations that isolate acoustic and behavioral factors.Prior systems were scripted, static, single-turn, or lacked validation across trials; τ-Voice also confounded accent variation with persona changes.Source: §2 · p. 3

Voice Agent Quality Measurement

• Existing quality metrics cover limited combinations of tool-use accuracy, turn-taking, and transcript response quality, omitting faithfulness, audio-level entity accuracy, efficient progress, cognitive load, and spoken correctness.These omissions leave consequential enterprise voice-agent failure modes unmeasured.Source: §2 · p. 3

3.1 Conversation Simulation

• EVA-Bench simulates repeatable enterprise voice tasks across airline, healthcare HR, and IT service domains using goals, personas, decision trees, databases, ground truth, and voice-sensitive entities.Fully automated bot-to-bot audio conversations support cascade and audio-native architectures under identical conditions; independently controlled acoustic and behavioral perturbations isolate robustness factors, while validation regenerates corrupted simulations, with 12.0% of trials requiring reruns.Source: §3.1 · pp. 3–4

3.2.1 EVA-A: Accuracy Metrics

• EVA-A measures accuracy through exact task completion, instruction-grounded faithfulness, and audio-level speech fidelity, including high-stakes spoken entities and architecture-specific handling of mishearing.Task completion compares final database state with ground truth; faithfulness distinguishes cascade input from audio-native responsibility, while speech fidelity evaluates intended spoken text or key entities for S2S systems.Source: §3.2.1 · pp. 4–5

3.2.2 EVA-X: Experience Metrics

• EVA-X measures experience through purposeful conversation progression, spoken conciseness, and turn-taking timing, including interruption, silence, latency, and missed-response behavior.Turn-taking uses distinct scoring functions for interruptions, yields, uninterrupted turns, and tool-call turns, with Conversation Completion accounting for failures to respond.Source: §3.2.2 · p. 5

3.2.3 Diagnostic Metrics

• Diagnostic metrics remain separate from EVA-A and EVA-X while exposing actionable failures such as key-entity transcription errors that word error rate can miss.Additional diagnostics cover authentication outcomes and response latency; key-entity transcription uses semantic rather than exact matching for cascade systems.Source: §3.2.3 · p. 5

3.2.4 Aggregate Metrics

• EVA-A and EVA-X aggregate dimension metrics with per-metric thresholds, requiring every constituent metric to pass before a conversation passes accuracy or experience.Accuracy requires task completion = 1.0, faithfulness ≥0.5, and speech fidelity ≥0.95; experience requires turn-taking ≥0.8, progression ≥0.5, and conciseness ≥0.5.Source: §3.2.4 · p. 5
• The framework reports pass@1 for average performance, pass@k for ceiling performance, and pass^k for reliability, with the pass@k–pass^k difference quantifying peak-versus-consistent capability.pass@k counts scenarios with at least one successful trial, whereas pass^k estimates the probability of passing all k independent trials in a scenario.Source: §3.2.4 · pp. 5–6

4.2 Evaluation Reliability

• EVA-Bench’s scores reflect behavioral differences: judge stochasticity is minimal, trial stochasticity dominates variance, and judge–human agreement ranges from κ = 0.777 to 0.845.The judge meets human agreement on all four metrics, while trial stochasticity consistently exceeds scenario-level variance.Source: §4.2 · p. 6

4.3 Main Findings

• No system exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; only GPT-Realtime-1.5 clears 0.4 on both, while four systems form the Pareto frontier.The frontier contains two S2S and two cascade systems.Source: §4.3 · p. 6, Figure 2 · p. 7
• Peak performance exceeds reliable performance across all 12 systems, with median pass@k–pass^k gaps of 0.44 on EVA-A and 0.24 on EVA-X.Even the strongest systems fall well below peak under pass^k, which requires passing all five trials on a scenario.Source: §4.3 · p. 7
• Robustness failures diverge by architecture: accented speech reduces cascade task completion by 10 points on average, while background noise reduces S2S EVA-X by 0.16.Under combined perturbations, cascade task completion drops 19 points on average, whereas the S2S mean remains within 5 points; turn-taking is most sensitive overall, with 81% of measurements significantly degraded.Source: §4.3 · p. 7

4.4 Failure Mode Analysis

• Across cascade systems, key-entity transcription accuracy strongly correlates with task completion (r = 0.93, p = 0.002), and accuracy below 70% corresponds to task completion of 0.37 versus 0.60.The association holds within each domain, with r = 0.88–0.93 and all p < 0.01.Source: §4.4 · p. 8
• Faithfulness is distinct from task completion: 72.2% of task-complete conversations contain a faithfulness deviation, while 50.5% of deviations co-occur with task failure.Speech fidelity failures primarily involve alphanumeric entities, including letter substitutions, digit omissions, and phonetic character confusions.Source: §4.4 · pp. 8–9

• EVA-Bench combines validation-gated bot-to-bot simulation, architecture-agnostic EVA-A and EVA-X metrics, and configurable pass criteria and perturbations for matched evaluation across architectures.The framework is fully open-source and extensible: practitioners can add domains, languages, scenarios, and evaluation dimensions for deployment-specific requirements.Source: §5 · p. 9
• The best cascade and S2S systems achieve comparable accuracy, but experience quality diverges sharply by architecture because the EVA-X gap is driven almost entirely by turn-taking.The comparison spans cascade, hybrid, and S2S agents under identical conditions.Source: §5 · p. 9
• 0.44 is the median pass@k–pass^k gap on EVA-A across systems, showing that single-trial evaluation systematically overstates deployment-grade quality regardless of architecture.The multi-trial consistency framework distinguishes peak from reliable performance.Source: §5 · p. 9
• Cascade and S2S systems degrade differently under acoustic perturbation, exposing perturbation-specific robustness patterns that existing benchmarks leave unmeasured.These findings complement EVA-Bench’s joint accuracy-experience measurement and peak-versus-reliable performance analysis.Source: §5 · p. 9

Metrics

• LLM- and LALM-based judges may exhibit stylistic bias or lower reliability, while binary task-completion scoring provides no partial credit for near-complete tasks.Bias may be higher when evaluated systems share a model family with the judge.Source: Metrics · p. 10

Framework

• The benchmark may not transfer fully to production because bot-to-bot simulation, mock tools, audio conversion, interface timing, and default latency settings omit important real-world variation.Simulation is English-only, under-generates interruptions, and may change with commercial-system versions; production latency optimizations are also excluded.Source: Framework · p. 10
• Generating k=5 validated trials across 213 scenarios and 12 systems is costly, requiring several hundred model calls per trial, and full reproduction requires commercial model APIs.All tools are declarative mocks rather than live APIs, so they cannot capture failure modes, latency variance, partial responses, or schema drift.Source: Framework · p. 10

A Definitions & Key Terms

• The paper defines STT, TTS, S2S, LALM, Hybrid, Audio-Native, multi-turn, and bot-to-bot terminology for describing voice-agent architectures and evaluation interactions.It also distinguishes pass@1, pass@k, pass^k, peak, and reliable performance as complementary trial-level concepts.Source: A Definitions & Key Terms · p. 14

Audio-Native

• Audio-native systems process audio directly at one or more stages, while multi-turn and bot-to-bot protocols evaluate stateful spoken interactions through simulated callers.These definitions frame the benchmark’s architectural and interaction scope.Source: Audio-Native · p. 14

B Experiment Parameters

• The experiment-parameter appendix documents turn detection, prompts, third-party packages, self-hosted hardware and serving, and API-hosted model configurations.Self-hosted models run on NVIDIA H100 GPUs, while ElevenLabs uses Scribe-v2.2-Realtime, Gemini-3-Flash, and TTS-Conversational-v3-Alpha with default agent parameters.Source: B Experiment Parameters · p. 15, §B.1 · p. 15, Table 3 · p. 15

B.3 Turn Detection Configurations

• Turn detection uses each framework’s defaults because their configurability prevents exact standardization: Pipecat combines VAD/transcription starts with LocalSmartTurnAnalyzerV3 stops, whereas other systems use server, automatic, or eagerness defaults.EVA-Bench exposes available turn settings through the CLI, except ElevenAgents, which requires separate agent registration and configuration.Source: §B.3 · p. 15

Compute requirements

• 1,875 evaluation samples per model expand to approximately 2,327 after a 24.1% trial-regeneration rate.The baseline comprises 213 scenarios × 5 clean-data trials plus 90 scenarios × 3 trials × 3 perturbation types; regeneration is split roughly evenly between simulator errors and infrastructure failures or timeouts.Source: §B.4 · p. 16
• EVA-Bench costs span hosted user simulation, proprietary-agent inference, and model-based judge scoring.ElevenAgents incurs per-interaction costs; proprietary pipelines add token-, audio-second-, and character-based charges, while eight metrics use per-sample LLM-as-judge inference; conversations average 4–5 minutes.Source: §B.4 · p. 16
• Full EVA-Bench runs are non-trivial compute investments, especially when benchmarking multiple proprietary models, requiring advance parallelization and cost budgeting.The recommendation follows from compounded simulation, inference, judging, and regeneration costs.Source: §B.4 · p. 16

C Data

• The three domains target temporal policy reasoning, dense named-entity transcription, and branching workflows with tiered authentication.Airline CSM covers high-stakes flight rebooking; Healthcare HRSD stresses multiple registration and license numbers; Enterprise ITSM gates escalation on failed resolution attempts.Source: C Data · p. 18
• Each domain includes Single-Intent, Multi-Intent, and Adversarial scenarios to test isolated workflows, compositional completion, and policy-boundary behavior.Multi-Intent scenarios combine one to four concurrent workflows, while Adversarial scenarios impose hard policy constraints under social pressure.Source: C Data · p. 18

C.1 Workflows

• The benchmark spans 50 Airline CSM, 83 Healthcare HRSD, and 80 Enterprise ITSM scenarios, with increasing workflow and tool complexity across the domains.Healthcare HRSD averages 8.7 expected tool calls per scenario, while ITSM spans 21 workflows and 59 tools; CSM spans seven workflow categories backed by 15 tools.Source: §C.1 · p. 18

Synthetic Data Generation with SyGra

• SyGra jointly generates each scenario’s user goal, initial database, and expected final database state from a workflow graph, preventing inconsistencies that would corrupt evaluation.The pipeline uses GPT-5.2, composes multi-intent records from single-intent cases, and hand-designs adversarial cases verified against tool-executor behavior.Source: Synthetic Data Generation with SyGra · pp. 18–19

Human Review

• Manual review verifies policy consistency, unique correct resolutions, database consistency, and identifiable adversarial violations, discarding ambiguous or inconsistent records.These checks occur across multiple review rounds after generation.Source: Human Review · p. 19

Frontier Model Stress Testing

• All selected scenarios achieved task completion of 1 on at least one frontier model, supporting confidence that failures in full audio evaluation reflect agent errors rather than dataset artifacts.OpenAI/Gpt-5.4, Google/Gemini 3.1 Pro, and Anthropic/Claude Opus 4.6 were tested text-only; zero-task-completion cases were manually investigated, and problematic records were corrected or removed.Source: Frontier Model Stress Testing · p. 19

D User Simulator Details

• The user simulator uses an ElevenLabs cascade and four agents covering English and French accents plus two genders, while prompting users to stay on task, state named entities clearly, and end calls appropriately.The cascade combines Scribe v2.2 Realtime, GPT-5.1, and Eleven V3 Conversational; the end-call tool is configured to terminate after resolution, inability to progress, or an unfulfillable request.Source: D User Simulator Details · pp. 20–21

D.1 User Simulation Validation / D.1.1 Conversation Valid End.

• Before judging, EVA-Bench deterministically accepts only conversations where the agent failed to respond or the user invoked end_call; other simulations are rerun to filter infrastructure failures.This validation gate catches unexpected WebSocket closures, failed starts, and simulator timeouts before User Speech and Behavioral Fidelity judges are invoked.Source: §D.1.1 · p. 21, §D.1 · p. 21

D.1.2 User Behavioral Fidelity.

• The behavioral-fidelity gate checks extra or duplicate modifications, premature ending, missing information, and decision-tree violations before rerunning corrupted conversations.A judge analyzes the full conversation trace, tool calls, and available tools, then emits per-type and overall binary ratings; conversations rated 0 are rerun.Source: §D.1.2 · pp. 21–22
• 63.9% of 714 flagged trials involved premature ending, 52.9% decision-tree violations, and 42.4% contained at least two corruption types.The trials covered four systems across all domains; missing information occurred in 28.6%, extra modifications in 1.3%, and duplicate modifications in 0%.Source: §D.1.2 · p. 22

D.1.3 User Speech Fidelity.

• User speech is rerun when any turn receives fidelity rating 1, indicating an entity error or major omission that could prevent sensible conversation progress.Ratings of 3 require full fidelity, while 2 permits minor errors that preserve entities and progression; the motivation is especially strong for authentication entities.Source: §D.1.3 · p. 22

E Metric Details

• All EVA-A and EVA-X scores are normalized to [0, 1], and pass@k or pass^k requires every metric to exceed its respective threshold simultaneously.The complete metric set and category organization are summarized in Table 11.Source: Table 11 · p. 24, E Metric Details · p. 24

E.1 Log Processing and Variable Extraction

• Merged timestamped logs recover per-turn intended, transcribed, audio-timing, and tool-interleaved trace variables while explicitly handling empty sessions, buffering, barge-ins, and interruptions.Three independent streams are replayed into one timeline, and interruption labels remain probabilistic because text sent to TTS may not equal audio actually played.Source: §E.1 · pp. 24–25

E.2 Equitable Evaluation Across Cascade, Hybrid, and S2S Architectures

• Pipeline-aware metric views prevent architecture bias: cascade scores the LLM’s transcript-to-text view, S2S scores the whole audio agent, and hybrid combines intended text with audio-native inputs.These views assign STT and TTS errors according to whether each architecture exposes or owns the corresponding intermediate representation.Source: §E.2 · pp. 26–27

E.3 Judge Development and Validation

• Judge development combines explicit failure-mode scales, targeted in-distribution generation, consensus or human adjudication, prompt refinement, and held-out validation.Speech Fidelity intentionally oversampled failures, producing 122 failing versus 25 passing examples, to expose subtle acoustic artifacts and avoid trivial pass predictions.Source: §E.3 · pp. 27–28
• Linguist–judge agreement ranged from κ = 0.777 to 0.845 across four metrics, meeting or exceeding linguist–linguist agreement in every case.Held-out validation used 63 samples per metric and expert linguist annotations, with modality-appropriate evaluation for text and speech.Source: §E.3 · p. 28

E.4.1 Task Completion

• Task Completion is a deterministic binary check that passes only when the agent’s final scenario database exactly matches the expected end state.It compares canonicalized database states via SHA-256 hashes, separately verifies authentication with a session-field superset check, and reports 1.0 for a match and 0.0 otherwise.Source: §E.4.1 · pp. 28–30

E.4.2 Faithfulness

• Faithfulness judges whether the assistant stays grounded in available information, policies, and instructions, penalizing path-level violations even when the task outcome is correct.Its five disjoint dimensions cover fabricated tool parameters, misrepresented tool results, policy violations, failure to disambiguate, and hallucination; pipeline adaptations treat audio-native mishearing differently from cascade transcription errors.Source: §E.4.2 · pp. 30–31

E.4.3 Speech Fidelity

• Speech Fidelity evaluates whether assistant audio conveys critical entities correctly, using intended text for Cascade/Hybrid and trace-grounded entity articulation for S2S.S2S excludes faithfulness and hallucinated-entity judgments, while rated turns receive binary scores aggregated by mean across eligible turns.Source: §E.4.3 · pp. 32–33

E.5.1 Conversation Progression

• Conversation Progression measures whether the assistant advances the user’s goal efficiently without redundancy, repeated tool calls, repeated information requests, or loss of established facts.A conversation can reach the correct outcome yet receive a low progression score if it loops or re-asks for known information.Source: §E.5.1 · pp. 33–34
• Conversation Progression scores unnecessary tool calls, information loss, redundant statements, and question quality, while explicitly separating faithfulness violations and voice-interface artifacts.The overall rating is count-aware: any severe dimension issue or three flagged dimensions yields the lowest rating, and passes require a normalized score of at least 0.5.Source: §E.5.1 · pp. 34–35

E.5.2 Conciseness

• Conciseness rates each assistant turn for verbosity, excessive information density, exhaustive enumeration, and disproportionate detail, then averages normalized per-turn scores across the conversation.Interruptions, necessary identifier delivery, confirmation-code spell-outs, and end-of-call recaps are exempt; a conversation passes when its mean score is at least 0.5.Source: §E.5.2 · pp. 35–37

E.5.3 Turn-taking

• Turn-Taking classifies each turn by interruption condition and routes it to an event-appropriate score, covering agent interruptions, user interruptions, and uninterrupted response latency.The unified score is deterministically computed from simulation timestamps and latencies, rewarding prompt yielding and penalizing overlap, poor recovery, or unsuitable response timing.Source: §E.5.3 · p. 37, Unified Per-Turn Scoring Regime · p. 37, Table 16 · p. 37

Unified Per-Turn Scoring Regime

• Uninterrupted-turn scoring maps response latency to a piecewise curve with hard-zero premature and excessively late regions, an optimal window, and graduated early and late penalties.Agent interruptions are capped at M = 0.5, whereas immediate yielding after a user interruption receives a perfect score of 1 and yield latency is tolerated up to 2,000 ms.Source: Unified Per-Turn Scoring Regime · pp. 37–38

Tool-Call-Aware Turn-taking Score

• Tool-call turns receive more lenient upper latency breakpoints while lower breakpoints remain unchanged, separating unavoidable execution delay from conversational responsiveness.The EVA-X Turn-Taking pass threshold is τtt = 0.8, interpreted as approximately four of five turns being on time.Source: Table 17 · p. 38, Tool-Call-Aware Turn-taking Score · p. 39, Pass/fail thresholding · pp. 39–40
• EVA-X system rankings remain stable as τtt varies from 0.50 to 0.95, with Spearman ρ = 0.968 and 61/66 pairwise orderings preserved strictly across all thresholds.Figure 4 reports preserved architecture ordering, while the three pairwise flips involve systems within 0.15 pass@1 of each other.Source: Pass/fail thresholding · p. 40, Figure 4 · p. 41
• Per-system EVA-X pass@1 vectors remain highly correlated with the τtt = 0.8 anchor, reaching Pearson r = 0.998 at τtt = 0.75, r = 0.995 at 0.85, and never below 0.910.The evidence supports selecting τtt = 0.8 because the sweep shows no plateau or cliff and the threshold maps to a 4/5-turns-on-time interpretation without making the metric overly lenient.Source: Pass/fail thresholding · p. 40, Table 19 · p. 40

E.6 Diagnostic Metrics

• Diagnostic metrics separately measure authentication, latency, speakability, STT error, tool-call validity, and key-entity transcription to distinguish speech-recognition failures from reasoning failures.Key-entity transcription targets names, dates, and alphanumeric codes, while latency is decomposed by tool-call involvement.Source: §E.6 · p. 40

F.1 Key Entity Transcription Accuracy & Task Completion

• Entity transcription and task completion are tightly coupled in EVA-Bench’s entity-dense domains, but similar transcription accuracy can yield different task completion when LLM reasoning differs.Among two Nova-3 systems, GPT-5.4 achieves meaningfully higher task completion than GPT-5.4-mini despite similar transcription scores; this relationship may not generalize to domains with fewer key entities.Source: §F.1 · p. 42

F.2 Faithfulness & Task Completion

• 14.5% of 12,780 clean trials achieve both task completion and faithfulness, making task completion alone a weak proxy for overall accuracy.The joint distribution covers all evaluated systems.Source: Table 20 · p. 42, §F.2 · p. 42
• The analysis aggregates outcomes across all evaluated systems under clean conditions.The dataset contains 12,780 trials spanning 12 systems.Source: Table 20 · p. 42, §F.2 · p. 42
• 38.3% versus 37.6% of faithfulness failures occur across the two task-outcome categories, indicating the dimensions capture distinct behavior.Source: §F.2 · p. 42

Pipeline-level patterns

• S2S systems tend to outperform cascade systems narrowly on on-time turns and conversation completion but trail on authentication success and policy adherence.The comparison uses per-pipeline-type means on four clean-run facets; the strongest cascades remain close on responsiveness.Source: §F.3 · p. 43, Figure 7 · p. 44
• 61% of conversations contain at least one policy violation, commonly involving unauthorized write actions or fabricated policies.Policy violations are the dominant failure mode contributing to low faithfulness scores.Source: §F.3 · p. 44

Analysis setup

• The analysis averages per-domain system values into unweighted pipeline-type means across 7 cascade, 3 S2S, and 2 hybrid systems, using bootstrap intervals as descriptive uncertainty bands.All metrics come from clean runs across three domains, with 1,065 conversations per system; pairwise differences are reported in percentage points.Source: §F.3 · p. 44

Caveats

• Pipeline-level differences do not necessarily generalize: within-class exceptions occur, hybrid systems diverge substantially, and the analysis makes no inferential claims beyond the evaluated systems.The ordering of pipeline-type means is nevertheless preserved across all three domains.Source: §F.3 · p. 44

F.3.1 Observations

• S2S systems sit at the top of all evaluated facets, although several cascades narrow or close the gap at the system level.Table 21 summarizes per-pipeline means and S2S−Cascade differences on clean runs, while system-level patterns show where individual cascades excel or trail.Source: §F.3.1 · pp. 44–45, Table 21 · p. 45
• Policy violations are the most variable facet: Nova-3 + GPT-5.4 + Sonic 3 has the lowest raw rate at 0.28 and remains lowest conditionally at 0.32, while no S2S system matches it.Conditioning raises Ink-whisper + Haiku-4.5 + Sonic 3 from 0.48 to 0.63, showing that hanging can make raw violation rates appear artificially low.Source: §F.3.1 · p. 45

F.3.2 Authentication: completion vs. ability

• GPT-Realtime-1.5 combines 99% clean-conversation completion with 93% conditional authentication success, the highest values on both axes.Cascade raw authentication deficits often reflect hangs: Nova-3 + GPT-5.4 + Sonic 3 rises from 0.85 to 0.94 when restricted to finished conversations, matching GPT-Realtime-1.5.Source: §F.3.2 · pp. 45–46, Table 23 · p. 46

F.3.3 Inactivity-timeout failures

• Short-turn inactivity timeouts account for 52–68% of timeouts in five of seven cascades, while spelled-content failures persist across all pipelines and dominate among systems handling short turns well.Short-turn failures often follow confirmation tokens, whereas spelled-content hangs occur predominantly during authentication exchanges.Source: §F.3.3 · p. 46, Table 24 · p. 47

F.4 Speech Fidelity

• Entity mispronunciation is the dominant speech-fidelity failure class, spanning substitutions, omissions, insertions, and phonetic confusions in codes, identifiers, and numbers.Examples include GPT-Realtime-1.5 dropping digits, Kokoro omitting letters, Gemini-3.1-Flash-Live inserting text, and multiple systems confusing acoustically similar characters.Source: §F.4 · p. 47

F.5 Turn-Taking & Response Speed

• Every system incurs higher latency on tool-call turns, with overhead ranging from roughly 0.1–0.2 seconds for fast cascades to several seconds for Ultravox-Realtime and Gemini-3-Flash + Gemini-3.1-Flash-TTS.Turn-taking scores track on-time and late rates more closely than early rates, with the largest early rate at 0.148 for GPT-Realtime-mini.Source: §F.5 · p. 48, Table 25 · p. 48

F.6 Cross Domain Variability

• Cross-domain variability is quantified as the sample standard deviation of each system–metric pair across CSM, ITSM, and HR, complementing per-domain and pooled results.This analysis measures how much domain dispersion is hidden by pooled headline scores.Source: §F.6 · p. 48

F.6.1 Metric-level findings

• Speech Fidelity is highly domain-invariant, with cross-domain standard deviations of 0.002–0.013 across twelve systems, while Faithfulness and Task Completion vary more substantially.Speech Fidelity’s mean standard deviation is 0.007, whereas Faithfulness and Task Completion average 0.087 and 0.064, respectively.Source: §F.6.1 · p. 48
• EVA-X is less domain-sensitive than EVA-A: mean standard deviations are 0.029/0.037/0.018 versus 0.074/0.101/0.050 for pass@1/pass@k/pass^k.Conversation Progression is the noisiest experience metric, Conciseness the most stable, and domain choice perturbs task accuracy more than experience.Source: §F.6.1 · p. 48

F.6.2 Model-level findings

• GPT-Realtime-1.5’s pooled Task Completion mean of 0.739 ± 0.045 conceals a per-domain standard deviation of 0.174, the largest in either table.Faithfulness shows the same masking effect for Scribe-v2.2-Realtime + Gemini-3-Flash + TTS-Conversational-v3-Alpha, with pooled 0.457 and standard deviation 0.184.Source: §F.6.2 · p. 49

F.6.3 Accuracy and Experience results per domain

• Per-domain clean-audio accuracy and experience results cover CSM, HR, and ITSM, with separate tables for each domain and cross-domain variability summaries.Accuracy appears in Tables 26–29, while experience appears in Tables 30–33.Source: §F.6.3 · p. 49, Table 26 · p. 49, Table 27 · p. 49

G Perturbation Analysis

• EVA-Bench applies user-simulator perturbations along independent behavior, accent, and audio-degradation axes, optionally composing them without configuring the evaluated agent.This design makes robustness scores reflect system sensitivity rather than special evaluation-side configuration.Source: G Perturbation Analysis · p. 52

G.1 Behavioral Perturbations in EVA-Bench

• Behavioral perturbations combine prompt and voice changes to test interruptions, rapid speech, slow delivery, long silences, disfluencies, and mid-utterance corrections.The three personas are aggressive_impatient, elderly_slow, and forgetful_disorganized.Source: §G.1 · p. 52
• Accent testing uses French, Indian, Spanish, and Chinese simulator voices, while audio degradation adds configurable noise or composable VoIP artifacts.Noise defaults to 15 dB SNR across eight environments; connection degradation combines codec compression, simulated packet loss, and volume fluctuation.Source: §G.1 · p. 52

G.2 User Personas

• User personas define a direct default caller plus aggressive_impatient, elderly_slow, and forgetful_disorganized behaviors, each maintaining behavioral and acoustic consistency through prompt and voice-model changes.They respectively model frustration and interruption, slow speech and repetition requests, or disfluency, memory lapses, and self-correction.Source: §G.2 · pp. 52–53

G.3 Perturbation Experiment

• Perturbation effects are computed as scenario-level mean deltas between perturbation runs and paired clean baselines, averaging three perturbation trials and five clean trials.The analysis evaluates accent, background noise, and accent-plus-noise conditions for each model, metric, and domain or pooled domain set.Source: §G.3 · p. 53
• Sign-flip permutation tests assess whether mean scenario deltas differ from zero without distributional assumptions, using 10,000 permutations and 95% percentile bootstrap confidence intervals from 1,000 resamples.Two-sided p-values are based on the fraction of permuted absolute means at least as large as the observed absolute mean.Source: §G.3 · p. 53
• Holm–Bonferroni correction controls the three-condition comparison family within each model, metric, and domain combination, with corrected p < 0.05 defining significance.The table captions use asterisks to denote corrected significance levels of p < 0.05, p < 0.01, and p < 0.001.Source: §G.3 · p. 53, Table 34 · p. 54, Table 35 · p. 54
• Complete pooled perturbation results are provided separately for accent, background noise, and accent-plus-background-noise conditions in Tables 34–36.The supplied passages identify the evaluated model ordering and the table scopes, but do not provide the cells’ numerical values or comparisons.Source: §G.3 · p. 53, Table 34 · p. 54, Table 35 · p. 54
• Figure 8 visualizes pooled perturbation deltas across EVA-A, EVA-X, Task Completion, Transcription Accuracy for Key Entities, and Conversation Progression, with negative bars indicating degradation from clean trials.Whiskers show 95% bootstrap confidence intervals, colors distinguish the three perturbation conditions, and asterisks indicate corrected significance.Source: Figure 8 · p. 55, §G.3 · p. 56

H.1 Variance decomposition

• Trial variance contributed 40–80% of judge-graded variance and 53–100% of deterministic variance, exceeding scenario variance in 33 of 34 model–metric combinations.The sole overlap was a 1.5-point faithfulness case in one model.Source: §H.1 · p. 57
• Scenario contributions ranged from near zero to 47%, with Task Completion and Faithfulness most sensitive to scenario difficulty.Task Completion had ICC 33–47% across models, while Faithfulness had ICC 21–42%.Source: §H.1 · p. 57
• Model × scenario interactions were significant in all 22 domain–metric combinations, explaining 4–18% of variance while scenario ICC remained low overall.Interaction significance was p < 0.01 in every case and p < 0.001 in 19 of 22 cases; median interaction variance was 11%.Source: §H.1 · p. 58
• Trial standard deviation exceeded judge standard deviation for all 16 model–metric combinations, with permutation-test p < 0.0001 in every case.The binomial sign test confirmed trial SD was larger in most scenarios for 13 of 16 combinations.Source: §H.1 · p. 60

H.2 Justification of trial count

• Per-turn metrics had CI width below 0.02 at k=1, whereas conversation-level and pass metrics narrowed from 0.05–0.09 at k=1 to roughly 0.02–0.03 at k=3.The per-turn metrics were Speech Fidelity, Conciseness, and Turn-Taking; noisier metrics included EVA-A pass@1, EVA-X pass@1, Task Completion, Faithfulness, and Conversation Progression.Source: §H.2 · p. 62
• At k=3, median 95% CI width was at most 0.034 for every metric, and at least 97.3% of subsamples fell within 0.02 of the k=5 anchor.Figure 9 showed approximately k^-1/2 decay in interval width as trial count increased.Source: §H.2 · p. 62, Table 41 · p. 63
• Because k=3 uncertainty was approximately 0.03, below 10% of the smallest interesting effect size, perturbation experiments used k=3 while clean evaluation retained k=5.The clean setting reused identical point estimates across scatter plots and Pareto frontiers, and the k=4-to-k=5 shrinkage was approximately 0.02 absolute.Source: §H.2 · p. 62

I.1 Airline CSM Workflows

• 50 airline CSM scenarios cover seven workflow categories backed by 15 tools, emphasizing time-pressured handling of named entities such as confirmation codes and flight numbers.The domain includes rebooking, voluntary changes, missed connections, standby, cancellations and refunds, escalation and availability constraints, and adversarial compensation claims.Source: §I.1 · p. 65

I.2 Healthcare HRSD Workflows

• 83 healthcare HRSD scenarios span 12 workflows with 47 tools and the benchmark’s highest workflow complexity, averaging 8.7 expected tool calls.Single-intent workflows average 5.0 calls, while triple-intent workflows reach 18; dual-intent, triple-intent, and adversarial variants extend the coverage.Source: §I.2 · p. 65

I.3 Enterprise ITSM Workflows

• 80 enterprise ITSM scenarios cover 21 workflows and 59 tools, testing branching resolution-versus-escalation flows and three authentication levels.The scenarios include 29 single-intent, 14 double-intent, 14 triple-intent, 14 quadruple-intent, and 9 adversarial cases; authentication ranges from standard to OTP-elevated and manager-level.Source: §I.3 · p. 65
• ITSM adversarial callers attempt to bypass mandatory troubleshooting, urgency classification, authorization, budget, asset-return, calendar, or approval steps.Source: §I.3 · p. 71

J Scenario Examples

• Each evaluation record specifies a user goal and decision tree, persona, scenario database, and ground truth for the expected final database state.The benchmark provides domain examples with sample transcripts and per-metric scores, including single-intent CSM, multi-intent ITSM, and adversarial HRSD scenarios.Source: J Scenario Examples · p. 72

User Simulator Specification

• User simulators combine a speaking persona with explicit goals, must-have criteria, branching acceptance or rejection rules, resolution and failure conditions, and escalation behavior.The flight example requires a direct same-day LAX-to-SFO flight before 2:00 PM with a fee under $80, and ends only after confirmed rebooking or failure.Source: User Simulator Specification · pp. 72–73

Expected Flow, Database & Ground Truth

• The expected-flow specification connects the user’s request to agent actions, while database tables define eligible flight inventory and pre- and post-change reservation states.For the flight scenario, the agent applies a same-day change fee and searches earlier available options.Source: Expected Flow, Database & Ground Truth · p. 74, Table 45 · p. 74, Table 46 · p. 75

Sample Conversation Transcript

• The sample transcript demonstrates verification, repeated flight-option presentation, user constraint enforcement, successful rebooking and seat assignment, final confirmation, and associated metric scoring.The transcript includes successful reservation lookup, rebooking to flight SK 130, window-seat assignment, and a reference to Table 47 for metric scores and failure explanations.Source: Sample Conversation Transcript · pp. 74, 76, 77, 78

User Simulator Specification

• The user simulator authenticates only when asked, addresses email, AD lockout, Confluence, and Figma sequentially, and ends after required confirmations or three turns without progress.It fixes choices such as read_only Confluence access and a temporary 30-day Figma license, while prohibiting invented goals and supervisor requests.Source: User Simulator Specification · p. 80

Expected Flow, Database & Ground Truth

• The expected flow processes identity verification and four items sequentially: an email incident, security-held AD lockout, auto-approved read-only Confluence access, and a 30-day Figma license expiring 2026-08-26.Ground-truth incident tickets, software requests, and catalog entries are recorded in Tables 48–50.Source: Expected Flow, Database & Ground Truth · p. 81, Table 48 · p. 81, Table 49 · p. 81

Sample Conversation Transcript

• The transcript shows successful employee authentication, creation of a high-urgency email-exchange incident ticket, tier-1 targets of 1h response and 4h resolution, and an Active Directory security hold.Source: Sample Conversation Transcript · pp. 82–83

User Simulator Specification

• The user simulator follows scenario-specific eligibility and negotiation rules, such as challenging only a backdating denial while accepting the underlying eligibility decision.The expected FMLA flow requires authentication, OTP verification, eligibility checking, repeated pushback on denied backdating, and no case creation.Source: User Simulator Specification · p. 86, Expected Flow, Database & Ground Truth · p. 87

Sample Conversation Transcript

• A sample conversation demonstrates multi-turn voice interaction with authentication, OTP verification, eligibility checking, leave submission, manager notification, and appointment scheduling.The transcript includes successful and failed tool calls, spoken structured data, and a final summary of completed actions.Source: Sample Conversation Transcript · pp. 87–89

K User Simulator Prompts

• EVA-Bench provides three domain-specific user-simulator prompts for Airline CSM, Enterprise ITSM, and Healthcare HRSD, populated at runtime from shared scenario variables.The shared inputs include persona, goals, criteria, negotiation behavior, supporting information, and other scenario context.Source: K User Simulator Prompts · p. 91, Table 54 · p. 91

K.1 Airline Customer Service Management

• The Airline CSM prompt models flight-service calls with prioritized seat preferences, spoken normalization of structured data, transcription-error recovery, concise responses, and explicit escalation and call-ending rules.The simulator must remain faithful to scenario details, avoid invented goals, wait for confirmation that requested actions completed, and use NATO phonetics after an initial mishearing.Source: §K.1 · pp. 91–94

K User Simulator Prompts

• Across the prompts, the simulator receives transcript text that may contain speech-recognition errors and is instructed to request clarification rather than treat garbled text as agent confusion.This voice-channel assumption is stated for both Airline CSM and Enterprise ITSM simulations.Source: §K.1 · p. 91, §K.2 · p. 94

K.2 Enterprise IT Service Management

• The Enterprise ITSM prompt adapts the simulator to internal service-desk calls by requiring confirmation that no outstanding actions remain and using IT artifacts such as case IDs, request numbers, and ticket confirmations.Its success and failure conditions also cover ITSM-specific outcomes such as desk assignments and cost centers.Source: §K.2 · p. 94
• ITSM simulator prompts constrain callers to their stated goals, scenario details, brief human-like turns, escalation rules, and explicit end_call conditions after confirmed completion or failure.Callers must verify that requested actions actually completed, wait after approvals or transfers, and end only with a brief goodbye once no actions remain.Source: §K.2 · pp. 95–97

K.3 Healthcare HR Service Delivery

• Healthcare HRSD prompts adapt the same strict turn-management and completion logic to employees or credentialed providers handling credentialing, benefits, or onboarding tasks.The simulator provides supporting information only when requested, treats transcript problems as possible transcription errors, and applies the same transfer, failure, and end_call rules.Source: §K.3 · pp. 97–100

L Agent Prompts

• Agent prompts are provided for the cascade, hybrid, and speech-to-speech architectures evaluated in the benchmark.The supplied material identifies these three architecture-specific prompt variants without reporting comparative results here.Source: L Agent Prompts · p. 102

Input variables

• Runtime-resolved template fields populate prompts containing persona, authentication, operating principles, tools, domain rules, and shared evaluation context.The judge infrastructure also derives conversation traces and intended or transcribed turns from pipeline-specific raw event streams, while injecting common interruption and turn-disclaimer fragments.Source: Input variables · p. 102, §M.1 · p. 125

Core differences between the three system prompts

• All three agent prompts share voice-formatting, conversational-behavior, and template-variable foundations, while architecture-specific constraints determine transcription handling, information density, and tool-calling posture.Cascade prompts address transcription artifacts and information overload; hybrid prompts retain information-density guidance; S2S prompts prioritize rapid repeated function calls and use compact response guidance.Source: Core differences between the three… · p. 102

L.1 Cascade System Prompt

• The cascade system prompt requires natural, concise, voice-friendly speech, spoken-form numbers and acronyms, and conversational descriptions instead of visual-only formatting.It instructs the agent to use complete sentences with pauses, avoid run-ons and shorthand, and omit tables, links, symbols, and text formatting unsuitable for audio.Source: §L.1 · p. 103
• Cascade’s prompt requires brief, conversational responses, natural transitions, warm professionalism, limited information per turn, and explicit uncertainty rather than hallucination.It also instructs the agent to summarize long lists, spread requests across turns, and use only current-conversation information.Source: §L.1 · pp. 103–104

L.2 Hybrid System Prompt

• Hybrid’s prompt adapts language for speech by requiring 5–20-word sentences, spoken forms for numbers and acronyms, and conversational summaries instead of visual-only formatting.It further prohibits symbols, URLs, text formatting, and awkward abbreviations while emphasizing brief, natural, listener-friendly turns.Source: §L.2 · pp. 104–105

L.3 Speech-to-Speech System Prompt

• Speech-to-Speech adds rapid tool invocation and information requests to the same voice-friendly rules for natural speech, audio-appropriate content, concise turns, and grounded responses.The agent calls tools until the task is complete, asks for missing information, and avoids hallucinations by relying on the current conversation.Source: §L.3 · pp. 105–106

L.4 Airline CSM Agent

• The airline agent authenticates callers, distinguishes voluntary from involuntary changes, explains fees and consequences before action, offers alternatives, transfers ancillaries, and summarizes the completed change.Its policy covers disruptions, cancellations, refunds, standby, elite benefits, escalation, upset callers, and exhausting availability before refund or credit.Source: §L.4 · pp. 106–107

L.4 Airline CSM Agent / L.5 Enterprise ITSM Agent

• Both agents should keep responses concise, listen when interrupted, and offer live-agent transfer for explicit requests, unresolved issues, policy exceptions, or unsupported flows.The ITSM agent must confirm the caller wants the transfer and summarize what has happened before transferring.Source: §L.4 · p. 107, §L.5 · p. 112

L.4 Airline CSM Agent

• The Airline CSM Agent classifies passenger-initiated changes as same-day when the new flight departs today and voluntary otherwise; Basic Economy changes incur a $75 fee plus fare difference, while IRROPS fees are waived.Source: §L.4 · pp. 107–108
• The Airline CSM Agent provides refunds, travel credits, meal vouchers, hotel vouchers, and same-day standby according to disruption, rebooking, timing, and passenger-choice conditions.Meal and hotel vouchers are issued after rebooking decisions and not when passengers choose refunds; hotel coverage is capped at 3 nights for later-date rebooking.Source: §L.4 · pp. 108–109

L.5 Enterprise ITSM Agent

• The Enterprise ITSM Agent authenticates every caller using standard, elevated, or manager verification based on the requested action, and prohibits action until verification is complete.Access changes require standard verification plus OTP, while representatives acting for another employee require manager verification plus OTP; other supported requests use standard verification.Source: §L.5 · pp. 110–112, §L.6 · p. 118
• The Enterprise ITSM Agent must verify identity, inspect records, confirm eligibility, read back error-prone values, complete required notifications, and close calls with outcomes and next steps.Readback is required for identifiers, codes, amounts, dates, and spelled-out names before changes, but is optional for harmless read-only lookups.Source: §L.5 · p. 111
• The agent applies the highest required verification level call-wide, uses OTP and manager authorization for elevated operations, and verifies identifiers and current records before changes.System-returned identifiers must be used exactly, and callers must confirm dictated identifiers and reviewed record state.Source: §L.5 · pp. 112, 115
• Incidents are categorized from caller-described symptoms into mutually exclusive login, outage, hardware, or network issues, with clarifying questions when categories overlap.The agent must determine the category rather than accept the caller’s chosen label, including distinguishing VPN connectivity from a VPN outage.Source: §L.5 · pp. 112–113
• Hardware, software, and facilities requests require domain-specific eligibility, budget, availability, catalog, approval, and delivery checks before submission.Examples include 36-month laptop and 12-month monitor replacement thresholds, 30-, 60-, or 90-day temporary licenses, active cost-center validation for permanent licenses, and ergonomic assessments for selected equipment.Source: §L.5 · pp. 113–114
• Account and access operations require elevated authentication and, when acting for another employee, manager authorization; provisioning, group changes, permission changes, and removals each require record, eligibility, approval, or HR checks.Access removals use staged or full removal according to handoff needs, and a valid HR off-boarding record is required before initiation.Source: §L.5 · p. 115
• Troubleshooting guides are followed step by step for login, network, and hardware problems; resolved issues are confirmed and closed without tickets, while unresolved issues receive tickets with completed-troubleshooting notes.Hardware failures additionally require asset lookup and technician scheduling, while network failures require attaching the caller’s diagnostic reference code.Source: §L.5 · pp. 115–116
• New incident tickets receive SLA tiers inferred from business impact—Tier 1 provides 1hr response and 4hr resolution, Tier 2 provides 4hr response and 8hr resolution, and Tier 3 provides 8hr response and 24hr resolution.Existing outages retain their original SLA, and callers are told the expected response time.Source: §L.5 · p. 116
• The Enterprise ITSM agent specifies post-action workflows for bookings, provisioning, permission changes, off-boarding, security incidents, MFA resets, and software-request escalation.These rules require actions such as sending calendar invites, scheduling 90-day access reviews, initiating asset recovery or remote wipes, transferring callers when necessary, and escalating only after an approval SLA breach.Source: §L.5 · p. 117

L.6 Healthcare HRSD Agent

• The Healthcare HRSD agent supports clinical and non-clinical staff across licensing, credentialing, scheduling, payroll, leave, onboarding, I-9 verification, and immigration, with authentication determined by request sensitivity.Standard verification uses employee ID and date of birth; provider verification uses NPI, facility code, and PIN; sensitive actions require OTP, and DEA transfers require provider verification followed by OTP.Source: §L.6 · pp. 117–118
• Healthcare HRSD processing requires complete verification, current-record lookup, eligibility checks, readback of error-prone identifiers, required notifications, and a clear call closure.No action may occur before authentication; blocking eligibility conditions stop processing, and completed changes must be followed by team notifications, caller updates, and any required appointments.Source: §L.6 · p. 118
• The agent uses explicit escalation and scheduling policies: it transfers only with caller agreement under defined failure or authority conditions and offers only system-available appointment slots.Scheduling asks for a preferred date, checks availability, offers alternatives when necessary, confirms the selected slot, and then books it.Source: §L.5 · p. 112, §L.6 · p. 119
• Domain-specific workflows impose precise compliance constraints for licenses, malpractice coverage, DEA transfers, clinical privileges, shift swaps, and on-call registration.Examples include 60-day license-extension eligibility, $1,000,000 minimum per-occurrence malpractice coverage, dual verification for DEA transfers, competency-review scheduling before privilege reactivation, certification checks for shift swaps, and leave and certification requirements for on-call registration.Source: §L.6 · pp. 120–121
• Healthcare HRSD procedures enforce eligibility, exact data capture, validation, confirmation, and required notifications across payroll, leave, onboarding, PTO, I-9, and visa-dependent workflows.Examples include FMLA eligibility and balance checks, schedule-specific PTO calculations, exact document identifiers, and manager, HR, compliance, or immigration-counsel notifications.Source: §L.6 · pp. 122–123

M Judge Prompts

• Judge prompts use explicit rubrics and structured binary issue flags with severity ratings, while interruption metadata is treated as non-spoken annotation rather than a faithfulness violation.Judges are instructed not to penalize truncated or incomplete content caused by overlap, and to flag only the assistant’s chosen content when it violates an evaluation dimension.Source: M Judge Prompts · p. 125, §M.1.1 · p. 125, §M.2 · p. 127

M.1.2 User turns disclaimer

• Pipeline-specific user-turn disclaimers distinguish cascade STT transcripts from raw-audio inputs, assigning transcription accountability to the assistant only for hybrid and S2S systems.Cascade evaluation uses the received transcript and does not penalize STT accuracy; hybrid and S2S evaluation treats intended user text as reference and expects clarification when audio is uncertain.Source: §M.1.2 · p. 126

M.1.3 Assistant turns disclaimer

• Assistant-turn disclaimers likewise distinguish cascade intended text from S2S audio transcriptions, separating conversational judgment from audio articulation fidelity and preserving literal tool-call evidence.For S2S, speech fidelity is scored on actual audio, while tool parameters and tool responses remain the literal API values used for evaluation.Source: §M.1.3 · p. 126

M.2 Faithfulness

• Faithfulness evaluation covers non-overlapping violation dimensions, including fabricated tool parameters that are invented, contextually unsupported, miscategorized, or hallucinated in free-text fields.The scope includes unsupported IDs, guessed placeholders, cross-context values, unsupported enum choices, and fabricated tool-field details.Source: §M.2 · pp. 127–128
• Faithful tool parameters may derive from user statements, prior tool results, standard mappings, policy entitlements, contextual inferences, simple arithmetic, or valid identifier disambiguation.Before flagging fabrication, evaluators must trace parameters to all such sources and verify enum values against the tool specification.Source: §M.2 · p. 128
• Misrepresenting tool results includes incorrect values, contradictions, omitted consequential information, undisclosed costs, and arithmetic errors, while accurate paraphrases, filtering, and harmless formatting differences are permitted.Verification requires independent arithmetic, conversion checks, and cross-referencing all relevant tool-result fields.Source: §M.2 · pp. 128–129
• Policy violations include skipping required procedures, taking prohibited actions, misapplying policies, or performing consequential writes without required disclosure and confirmation.Read and write operations requiring explanation or confirmation must be separated by user confirmation; favorable applicable policies and no-cost entitled benefits are not violations.Source: §M.2 · pp. 129–130
• Assistants must clarify ambiguous, contradictory, or failed lookup inputs rather than silently choosing among options, dates, names, codes, or downstream-impacting interpretations.Proceeding is acceptable when intent is clear, a reasonable inference is obvious, or corrected lookup variations are grounded in the conversation.Source: §M.2 · pp. 130–131
• Hallucination is information stated without support from tools, user utterances, instructions, or system context; evaluators must check every source before flagging it.The framework excludes grounded computations, current-date references, courtesies, hedged caveats, and reasonable domain knowledge, and rates issues by user impact from 3 to 2 in the supplied scale.Source: §M.2 · pp. 131–132
• Faithfulness’s overall rating is the minimum across dimensions: any 1 yields overall 1, all 3s yield 3, and no 1 with at least one 2 yields 2; major violations materially affect the user.Examples include wrong or fabricated write-tool parameters, unresolved transcription ambiguity before committing, irreversible actions without explicit confirmation, and harmful financial or reputational misstatements.Source: §M.2 · pp. 132–133

M.3 Speech Fidelity

• Speech fidelity compares the exact intended text word-for-word with the audio, prioritizing TTS-critical entities; rating 1 means all entities are correct and non-entity words are faithfully reproduced without meaningful omissions or additions.Spelled-out codes and segmented reference IDs must be verified letter- and digit-by-digit; wrong digits, letters, amounts, or numbers are entity errors.Source: §M.3 · pp. 134–135

M.4 Conversation Progression

• Conversation progression flags four mutually exclusive types of inefficient action, including unjustified tool calls such as needless repetition, missing parameters, or requests for information already available.The metric excludes faithfulness or policy violations, expected repetition after speech-recognition errors, and interruptions themselves unless they cause information loss or unnecessary repetition; corrected retries and sequential calls yielding new information are allowed.Source: §M.4 · pp. 136–137
• The final rating considers both the severity within each dimension and the total number of flagged dimensions.A rating of 3 means no dimension is flagged; rating 2 means one or two dimensions are flagged at rating 2 with no dimension rated 1, while rating 1 reflects any dimension rated 1 or three or more flagged dimensions.Source: §M.4 · p. 140
• Speech-to-speech progression scoring treats transcription variations in consistently spoken identifiers as speech-fidelity artifacts, flagging information loss only for structural or semantic discrepancies or downstream evidence of a wrong value.Examples include a different entity or field, incorrect follow-up tool calls, stale-data actions, or failure to incorporate user objections.Source: §M.4 · p. 141

M.5 Conciseness

• Conciseness evaluation rates only assistant speech, expecting brief, conversational responses that summarize information, avoid overload, and spread multiple requests across turns.Interrupted or truncated content is not penalized when interruption tags indicate the assistant could not finish speaking.Source: §M.5 · pp. 141–142
• Conciseness requires assistant turns to be short enough for an average person to follow in real time and retain in working memory, avoiding unnecessary wording, excess information, long lists, and disproportionate detail.Source: §3 · pp. 142–143

M.6 User Speech Fidelity

• User speech fidelity checks expected words and TTS-critical entities while exempting non-spoken tags, interruption-marked omissions, minor pronunciation differences, natural fillers, and final-turn cutoff omissions.Per-turn ratings are 3 for high fidelity, 2 for noncritical omissions or errors that preserve progress, and 1 for entity errors or major failures that impede conversation.Source: §M.6 · pp. 144–145

M.7 User Behavioral Fidelity

• User behavioral fidelity attributes evaluation corruption to the user only when visible behavior plausibly caused a modification problem, while tool traces may exonerate users from unknowable agent failures.The analysis covers off-script requests that trigger modification tools and premature user-ended calls, but excludes read-only deviations, agent errors, and hallucinated tool calls.Source: §M.7 · pp. 146–147
• User-behavioral fidelity labels corruption when user actions cause premature endings, missing information, duplicate or incorrect modifications, and distinguishes clean from corrupted evaluations.The rubric also covers off-script demands, skipped wrap-up, unhelpful agents, valid resolution, and transfer-specific termination behavior.Source: §M.7 · pp. 147–148

M.8 Speakability

• Speakability scoring marks text voice-unfriendly if it contains visual formatting, unspoken structural syntax, or missing spaces that could cause TTS failure.Examples include Markdown, tables, repeated symbols, JSON brackets, and concatenated words.Source: §M.8 · p. 148

M.9 Transcription Accuracy (Key Entities)

• The transcription-accuracy prompt evaluates Speech-to-Text accuracy for key entities across an entire conversation.Source: §M.9 · p. 149

N.1 Third Party Licenses

• EVA-Bench depends on third-party packages documented with their licenses, while the benchmark itself is released under the MIT license for research, enterprise evaluation, and framework extension.The dependency licenses are listed in Table 56, with full texts in the repository’s THIRD_PARTY_NOTICES file.Source: §N.2 · p. 153, Table 56 · p. 153

N.2 EVA-Bench License

• The license discourages selective or misleading reporting and prohibits using EVA-Bench scenarios as training data because this could undermine evaluation integrity and inflate performance.Outputs should be interpreted alongside the statistical methodology and stated limitations.Source: §N.2 · p. 153

O AI Usage

• The authors used LLM assistants for selected writing and phrasing and AI coding agents for scaffolding, while humans conducted the research design, analysis, interpretation, and conclusions.Seven internal AI data linguists independently annotated each sample by two annotators using the same prompts and rubrics as the LLM judge, without additional guidance.Source: O AI Usage · p. 154, P Human Annotations · p. 155

Ethics Statement

• All EVA-Bench scenarios are fully synthetic, using no real caller data, recordings, or personally identifiable information.Source: Ethics Statement · p. 156
• The tools follow copyright and privacy guidelines, but evaluated models may still produce harmful or biased outputs, requiring filtering and bias detection before deployment.Source: Ethics Statement · p. 156
• EVA-Bench currently covers limited domains and English-only simulation, potentially reinforcing representational biases; the authors encourage broader languages, accents, cultures, and domains.The authors are actively extending the benchmark to other languages.Source: Ethics Statement · p. 156

Reproducibility Statement

• The released open-source repository includes code, configurations, scripts, scenario data, documentation, setup instructions, and environment specifications for reproducing evaluations.Source: Reproducibility Statement · p. 156
• Reproducibility is supported by model configurations, domain data distributions, judge prompts, metric definitions, thresholds, and fully specified LLM-as-judge settings.Source: Reproducibility Statement · p. 156
• Full score reproduction requires commercial model APIs, and results may vary across API versions and deployment configurations.Source: Reproducibility Statement · p. 156

R Future Work

• Future work will add multilingual scenarios, multi-agent architectures, more realistic interruptions and caller behaviors, safety and privacy evaluation, and broader coverage of implementations, architectures, and providers.Planned safety checks include harmful outputs and inadvertent exposure of sensitive caller information such as PII.Source: R Future Work · p. 157