Source-linked AI summary
VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation
Joshua Meyer, Sahar Shayegan, Ritiz Tambi, Ali Khan, Sun Kim, Victor Shih, Mehdi Jamei, Andi Partovi
TL;DR
Production voice calls can combine several requests, competing speech, and impatient callers, while evaluation must account for both what agents say and do. VAmoS Energy benchmarks these challenges in utility assistance calls and finds widely varying completion, with television particularly difficult. Its results underscore the value of evaluating the whole call, while verifier audits and selective exclusions leave some uncertainty.
Problem
Production voice calls can involve several requests and background speech, yet voice-agent evaluation must assess conversation and execution together.
Source: §1 · p. 1Method
VAmoS Energy evaluates fourteen voice stacks on multi-request utility calls with varied callers and backgrounds, using stateful services and an LLM verifier for spoken and tool-action requirements.
Source: §1 · p. 1Results
Takeaways & Limitations
Evaluating conversation and execution together exposes failures that a caller’s apparent satisfaction would miss.
Source: §8 · p. 7Unaudited verdicts can still be wrong, and selective exclusions may affect the task mix.
Source: §7 · p. 7
Abstract
from arXiv · showhide
Voice agents in production must handle several requests, background speech, and customers who lose patience. We introduce VAmoS Energy, a benchmark that combines these challenges in 100 calls about utility billing and payment assistance. Each caller makes two to four requests. The agent has sixteen tools backed by a stateful Stripe billing twin and the Apache Fineract loan engine, with account access blocked until caller verification succeeds. The tasks use public household electricity data and a policy based on Pennsylvania's residential billing rules. An LLM-as-a-verifier checks the agent's actions and spoken figures against explicit requirements. On a calibration run, it agrees with a code verifier on 99.1% of checks. Across fourteen voice stacks and three repeats per task, completion ranges from 17.3% to 44.7%. Grok Voice leads, and Gemini 3.8 Live and GPT-Live follow at about the same cost per call. Background television reduces pooled completion from 38.7% to 8.6%. The simulated caller often accepts an incorrect result because it hears the agent's words but cannot inspect its actions. These findings show why voice agents need evaluation across the whole call, including what they say, what they change, and how they handle competing speech.
1 Introduction
VAmoS Energy evaluates voice agents on multi-request utility calls that combine account verification, policy-compliant actions, and background speech. Agents often fail to complete required account changes, and performance varies across stacks and audio conditions.
1 Introduction
- VAmoS Energy benchmarks utility billing and payment assistance with two to four requests per call, stateful services, caller verification, and a Pennsylvania-rule-based policy.The benchmark uses public household electricity profiles and compares fourteen voice stacks under calm and angry callers and four background conditions.Source: §1 · p. 1
- Table 1 summarizes the differences between VAmoS Bench and VAmoS Energy, but their completion rates do not isolate task difficulty because domains and agent configurations also differ.The table caption likewise cautions that observed completion depends on both benchmark and evaluated configurations.Source: §1 · p. 1, Table 1 · p. 2
- Tasks combine requests while preserving validity after earlier actions, and callers may be angry or speak over street noise, café chatter, or television.Each stack receives the same persona and background assignment for each task and repeat.Source: §1 · p. 1
- The verifier checks spoken facts, tool results, and consent-action order, while the simulated caller can hear only the agent’s words.The benchmark excludes calls affected by caller deviations, grading errors, or platform errors from reported scores.Source: §1 · p. 2
- Agents usually respect the verification gate but often fail required account changes; success also falls as callers add requests, and television is the hardest background.Repeated trials vary, and 54.2% of calls the simulated caller considers successful fail verification.Source: §1 · p. 2
2 Related work
Related benchmarks address stateful task completion, speech conditions, or joint evaluation of conversation and execution. VAmoS Energy combines multiple requests, caller disposition, and background conditions in a utility domain.
2 Related work
- EVA-Bench evaluates final database state, faithfulness, and speech fidelity, while τ-Voice varies noise, accents, and turn-taking for full-duplex agents.VAmoS Energy combines multiple requests, caller disposition, and background conditions in a utility domain with two vendor services.Source: §2 · p. 2
- VAmoS Bench grades conversation and execution together across production voice stacks [Meyer et al., 2026].VAmoS Energy extends this benchmark context to multiple requests per call and varied caller and background conditions.Source: §2 · p. 2
- τ-bench checks final database state and required response information, while VAmoS Energy uses explicit task checks interpreted by an LLM.LLM-as-a-judge can show position and verbosity biases [Zheng et al., 2023].Source: §2 · p. 2
- RST recursively extends tasks and validates them in a sandbox, whereas VAmoS Energy adds valid requests and selects tasks using measured difficulty.Its executable policy model checks admission but does not establish solvability through a successful reference agent.Source: §2 · p. 2
3 The benchmark
The benchmark combines verified, policy-constrained utility tasks with stateful services, simulated electricity profiles, recursively constructed multi-request calls, varied callers and audio, and checks of both speech and tool execution.
3.1 Task, verification, and policy
- The agent handles billing, payments, instalments, and billing preferences; account tools remain blocked until the caller passes verification, and actions must satisfy policy and consent requirements.A promise to pay later must not be recorded as a payment, and arrangements must be quoted before creation.Source: §3.1 · pp. 2–3
3.2 Stateful services and public data
- Each call uses stateful Stripe billing and Apache Fineract loan services cloned from a consistent snapshot; the 24 exam accounts use modeled Pennsylvania electricity profiles and a fixed, simplified policy.The profiles describe simulated homes rather than customer meter readings, and the encoded policy is not a complete statement of current law.Source: §3.2 · p. 3
3.3 Building difficult tasks
- Tasks grow by appending requests that remain valid after prior actions; development-run success estimates rank combinations, and the final set contains 100 difficult tasks.The set includes 6 tasks with two requests, 30 with three, and 64 with four; not every generated combination was run.Source: §3.3 · p. 3, Figure 1 · p. 4
3.4 Callers and background conditions
- Simulated callers use calm or angry personas, while calls mix clean audio or street noise, café chatter, or television speech across repeats.The persona remains fixed for a task’s repeats, while background assignment rotates across repeats and is shared across stacks.Source: §3.4 · p. 4
3.5 Grading
- Three task-derived assertions check verification and disclosure, ordered account changes with required values, and remaining spoken requirements; all must pass.The verifier reads transcripts and recorded tool calls, arguments, and results, inferring account changes from successful tool results rather than independently inspecting the backend ledger on the main run.Source: §3.5 · p. 4
4 LLM-as-a-verifier
The paper uses a constrained LLM verifier to interpret conversational evidence against explicit checks, then calibrates its judgments against a code verifier. The two methods agree frequently, but agreement is not ground-truth accuracy.
4 LLM-as-a-verifier
- Rule-based speech parsing can confuse quantities with postal codes or payment counts with month counts, while conversational checks also require identifying speaker, topic, and event order.The same amount may be spoken in different valid forms, such as “sixty-five fifty-two” or “sixty-five dollars and fifty-two cents.”Source: §4 · p. 4
- The LLM receives explicit checks for value, tolerance, evidence window, and topic, and returns a verdict with supporting evidence.The method remains fallible and permits semantic equivalents for statements such as whether an account is past due.Source: §4 · p. 5
- 99.1% of 744 checks were judged identically by gpt-5.6-terra and the code verifier, but agreement between fallible methods does not establish accuracy against ground truth.Review of combined disagreements found seven cases where the LLM was right and eight where the code was right.Source: §4 · p. 5
5 Evaluation
Evaluation covers fourteen voice stacks across three runs per task. Scores exclude calls affected by benchmark faults while counting agent-side run failures as misses.
Evaluation
- 4,200 calls cover fourteen stacks, with three runs per task.Six stacks use speech-to-speech models, three use bundled voice platforms, and five use assembled pipelines.Source: §5 · p. 5
- 402 benchmark-fault calls are excluded, leaving 3,798 counted attempts; agent-side run failures count as misses.Completion is the fraction of counted attempts passing all assertions.Source: §5 · p. 5
- Across repeats, background changes while each task retains its persona and uses another conversation.Intervals are Wilson 95% intervals over each stack’s counted attempts.Source: §5 · p. 5
6 Results
Completion varies substantially across stacks and task conditions, with television the hardest background. Results also show a gap between satisfying access restrictions, completing account changes, and callers believing their goals were met.
Completion and account changes
- Completion ranges from 17.3% to 44.7%; Grok Voice leads, followed by Gemini 3.8 Live and GPT-Live, whose intervals overlap.The three cost about the same per call: $0.21, $0.193, and $0.192, respectively.Source: Table 2 · p. 5, §6 · p. 5
Verification and account changes
- The verification gate passes on 99.0% of calls, while required account changes pass on 54.8%.On 44.4% of calls with verdicts, the gate passes while the account-change assertion fails.Source: §6 · pp. 5–6
Background speech
- Pooled completion is 38.7% in clean audio, 41.4% with street noise, 27.4% with café chatter, and 8.6% with television.Nine stacks complete fewer than 5% of TV calls; traces show agents treating broadcasts as caller speech, answering them, or cancelling their replies.Source: Figure 2 · p. 6, §6 · p. 6
More requests, lower success
- Success rates fall from 56.3% for two requests to 34.9% for three and 23.8% for four, though these are different task groups.Agents may miss a confirmed action, omit a requested figure, or state the wrong policy limit; the rates do not isolate the effect of adding a request to an otherwise identical call.Source: §6 · p. 6
Angry callers and transfers
- Angry calls complete at 30.4%, versus 27.8% for calm calls, but different persona task groups prevent attributing the difference to anger.Agents transfer 7.6% of angry calls and 4.1% of calm calls to humans, although no exam task requires a transfer.Source: §6 · p. 6
Variation across repeats
- The median stack’s completion-rate standard deviation is 3.8 percentage points across repeats, and 45 tasks have both passing and failing attempts.Variation combines conversation variation with intended background changes; overlapping intervals warrant caution when ranking adjacent stacks.Source: §6 · p. 6
Caller satisfaction
- Of 2,072 calls the simulated caller deemed successful, 1,122 fail verification (54.2%).The caller hears assurances but cannot see whether a tool recorded the right amount, so satisfaction alone is insufficient for scoring.Source: §6 · pp. 6–7
Three unsolved tasks
- Three tasks remain unsolved: they require reporting an overdue arrangement instalment even though the bill balance is current.The shared account tool reports only bill arrears, and without a successful reference solution the authors cannot claim these tasks are solvable.Source: §6 · p. 7
7 Benchmark validity and limitations
Reported results are bounded by caller-simulation and verification errors, implementation and cost choices, and limited task coverage.
7 Benchmark validity and limitations
- 402 calls were excluded for caller deviations, confirmed verdict errors, or lost sessions; 36 exclusions had passed, avoiding retention of artificially easy calls.The authors neither changed verdicts nor reran selected calls.Source: §7 · p. 7
- The 228-call audit targeted suspected errors, so its frequency is not an unbiased verifier-accuracy estimate; unaudited verdicts may still be wrong and exclusions may alter task mix.Exclusions raised pooled completion from 27.2% to 29.1% while preserving the top five stacks’ order.Source: §7 · p. 7
- Integration defects count against candidates, Soundex matching may favor some stacks, and one configuration per vendor leaves results sensitive to vendor tuning.Costs omit telephony and hosting; GPT-Live’s cost omits a delegated model that could add at most 17%, while Gemini 3.1 Live’s cost is estimated.Source: §7 · p. 7
- The exam covers one utility policy and selected modeled usage, not all customers or regulations; two personas and three repeats provide limited coverage.Policy review does not prove every task is solvable, and noise or persona comparisons remain sensitive to task mix.Source: §7 · p. 7
8 Conclusion
VAmoS Energy tests multi-request utility calls under policy, stateful-service, and background-speech demands. Completion varies across stacks, while television and the gap between spoken acceptance and correct execution expose failures that whole-call evaluation can detect.
8 Conclusion
- Completion ranges from 17.3% to 44.7% across fourteen configurations.Source: §8 · p. 7
- Television disrupts most stacks, account changes fail more often than verification, and callers often accept incorrect outcomes.Source: §8 · p. 7
- Evaluating conversation and execution together exposes failures that apparent caller satisfaction would miss.Source: §8 · p. 7
A Policy encoded in the benchmark
The benchmark encodes a simplified utility policy and authored business rules, selects difficult multi-request tasks, and compares verifier judgments against code on a calibration run.
A Policy encoded in the benchmark
- The encoded policy includes income-banded arrangement caps and seasonal or medical protections, alongside authored arrears, down-payment, extension, and transfer rules.The cap table and protections are benchmark encodings, not a complete statement of statutory requirements; final tasks exclude transfers and successful extensions.Source: A Policy encoded in the benchmark · p. 9
B Task selection
- Task construction excludes invalid request combinations and selects 100 tasks using measured difficulty, with a predicted mean success of 0.27.The selection uses 26 starting requests across 24 accounts; its model ranks tasks and is not a calibrated performance forecast across stacks.Source: B Task selection · p. 9
C Verifier comparison
- Explicit checks with matched evidence raised agreement with code from 76/88 to 85 calls, but changed specifications and evidence prevent isolating wording alone.The calibration comparison used 97 attempts; code is a comparator, not ground truth.Source: Table 3 · p. 9, C Verifier comparison · pp. 9–10
D Additional metrics
The section defines how completion, other rates, latency, and cost are measured, and reports completion breakdown dimensions and cost-per-pass comparisons.
D Additional metrics
- Completion counts all attempts, including agent-side run failures; assertion and transfer rates use different eligible-call denominators.A transfer is the transfer_to_human tool call, not a spoken promise; latency runs from caller speech ending to agent speech starting.Source: D Additional metrics · p. 10
- Cost per completed call is total cost divided by passes: $0.22 and $0.26 for the Hugging Face and LiveKit cascades, about $0.47 for three leading stacks, and $1.62 and $1.72 for the two OpenAI Realtime models.Cascade estimates use published prices and measured usage; speech-to-speech costs use vendor bills.Source: D Additional metrics · p. 10
- Table 4 groups completion by request count, caller persona, and last-request use case; the last-request category need not identify the failure source.Before exclusions, each stack had 18/90/192 attempts for two/three/four requests, respectively; counted denominators are smaller.Source: Table 4 · p. 10, D Additional metrics · p. 10
E Audit and exclusions
The section audits verifier judgments and caller deviations, describes the limits of Soundex matching, and inventories the benchmark’s voice stacks, tools, and task requests.
E Audit and exclusions
- Auditors found no wrong verdicts in 72 stratified calls; targeted reviews surfaced false failures, chiefly stale interruption markers and tolerance handling.The 116-call sweep found 43 false failures and one false pass; another 40 calls yielded one false failure. The leading causes were stale interruption markers (17 calls) and incorrect tolerance handling (13).Source: E Audit and exclusions · pp. 10–11
- Figure 3 plots completion against median response latency and mean cost per call, with Wilson intervals and a logarithmic cost axis.Source: Figure 3 · p. 11
- Caller deviations led to 85 Vapi exclusions, while Table 5 compares all-attempt and counted completion and categorizes caller, verifier, and lost-session exclusions.Caller deviations could make tasks easier by omitting a request or supplying an answer, so exclusions include passes as well as failures.Source: E Audit and exclusions · p. 11, Table 5 · p. 11
- Soundex retains first letters while allowing spelling differences; 1,027 calls used sound-alike-only verification, including 336 passing calls.These counts use raw denominators across all attempts, including those later excluded.Source: E Audit and exclusions · p. 11
F Voice stacks
- All fourteen stacks share the prompt, tool declarations, and dispatcher; Table 6 orders them by observed completion and lists their recorded configurations.Source: Table 6 · p. 12
G Tools
- The sixteen listed tools include five world-changing actions: payments, payment arrangements, due-date extensions, and paperless billing.Source: G Tools · p. 12
H Requests in the exam
- Table 7 catalogs the requests drawn on by the 100 tasks and the number of tasks containing each request.Source: Table 7 · p. 13