Source-linked AI summary
$\tau$-Elicitation: Benchmarking multi-turn entity extraction in voice agents
Soham Ray, Victor Barres
TL;DR
Voice-agent benchmarks often score complete tasks without isolating exact entity capture or recovery. τ-Elicitation benchmarks controlled spoken entity collection and finds that even scaffolded systems succeed across all three environments on only 37–54% of tasks.
Problem
End-to-end voice-agent benchmarks do not isolate whether an interactive agent recognizes uncertain entity capture, verifies it, repairs errors, and stores the exact value.
Source: §1 · p. 1Method
τ-Elicitation is a reproducible 200-task benchmark spanning 10 entity types and controlled difficulty and caller realisms, scored against exact database values across three environments.
Source: §1 · p. 1, §3.3 · p. 2Results
Even with a strict elicitation protocol, systems succeed across all three environments on only 37–54% of tasks, and only 24–37% of verified errors are repaired.
Source: §6 · p. 4Takeaways & Limitations
Systems execute a supplied protocol more reliably than they select one unaided, while the pattern locates a bottleneck after risk is noticed.
Source: §5.1 · p. 3, §5.2 · p. 3The study covers English, three providers, one simulator, and one run per main environment; caller-side ASR is omitted, and agent speech synthesis defects do not affect task reward.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Voice agents often need to collect names, addresses, identifiers, dates, and times exactly, yet end-to-end benchmarks obscure where capture fails. We introduce $\tau$-Elicitation, a 200-task voice benchmark spanning 10 entity types, controlled difficulty, caller realisms, and three environments. A matched text agent passes all tasks, but four voice configurations achieve robust exact success from 0.14 to 0.41. Agents increase verification for hard and unfamiliar entities and sometimes for incorrect captures, but not for their weakest caller voice; only 24 to 37 percent of verified errors are repaired. A scaffold that prescribes spelling, read-back, correction, and confirmation raises robust Pass$^3$ by 14 to 31 points, at a cost of 21 to 28 seconds per call. Realisms such as spelling variations and restarts do not detectably affect exact success; mispronunciation increases repair effort. These results identify strategy selection and successful recovery as the central bottlenecks in exact spoken entity collection.
1. INTRODUCTION
End-to-end voice-agent benchmarks do not isolate whether agents capture and recover exact entity values, despite the importance of named entities to dialogue. τ-Elicitation introduces a controlled benchmark that separates agent-selected verification from a prescribed protocol.
1. INTRODUCTION
- Existing benchmarks score complete tasks, but do not isolate uncertain capture, verification decisions, error repair, and exact database writes.The gap also includes how these capabilities change with entity type, difficulty, repeated conditions, and caller effort.Source: §1 · p. 1
- The default condition leaves verification strategy to the agent, while a matched scaffold prescribes asking, spelling or reading back, confirming, and correcting.Both conditions share consent, never-guess, logging, and one-submission rules, and database comparison scores results without an LLM judge.Source: §1 · p. 1
- τ-Elicitation contributes a reproducible 200-task benchmark spanning 10 entity types, controlled difficulty and caller realisms, and multi-entity calls.It measures capture, verification, repair, effort, and robust success across three environments.Source: §1 · p. 1
2. RELATED WORK
Prior voice-agent benchmarks cover end-to-end task performance and interaction quality, while related work studies tracking, recognition, and repair separately. τ-Elicitation focuses on controlled exact entity collection during live tool use.
2. RELATED WORK
- Existing voice-agent benchmarks measure end-to-end task performance and interaction quality, but do not isolate exact entity capture [1] [2].Several also exercise tool use [1] [2] [3] [5], and authentication is identified as a dominant bottleneck in τ-Voice [1].Source: §2 · p. 1
- Related work covers dialogue-state tracking, ASR and spoken entities, and dialogue repair, but does not combine live tool execution with exact database writes and controlled entity conditions.The missing combination includes controlled entity type, difficulty and count, and the agent’s choice to verify.Source: §2 · p. 1
3. METHODS
τ-Elicitation constructs controlled callbacks from entity banks and runs them through a voice interaction loop. It scores exact values across prespecified environments and records verification effort.
3.1. Task and elicitation policy
- Each task is a callback about a record with missing fields; the caller reveals a value only after the agent requests it.The caller spells accurately when asked and confirms or corrects read-backs, but does not volunteer information.Source: §3.1 · pp. 1–2, Figure 1 · p. 2
3.2. Values, difficulty, and realisms
- Ten entity banks contain easy and hard values, with difficulty defined relative to entity type.A seeded catalog adds six caller realisms, including spelling variation, restarts, partial or wrong-field answers, and mispronunciation; linked two- and three-field tasks are also generated.Source: §3.2 · p. 2
3.3. Scoring and behavioral measures
- Pass@1 is exact-value success in the regular realization, while Pass3 requires the same task to succeed across regular, noise-heavy, and speech-heavy realizations.Pass3 tests robustness across prespecified conditions rather than averaging away failure in one condition.Source: §3.3 · p. 2
- Verification effort is the sum of logged spelling requests and read-backs; timed-out calls remain in the denominator and score zero.A conservative four-minute call limit follows a 700-call audit whose 90th-percentile duration was 1.89 minutes.Source: §3.3 · p. 2
4. EXPERIMENTS
Experiments compare four voice configurations under matched task and environment conditions, with a scaffolded-prompt repeat. They also audit failed calls and validate a sample of agent speech.
4. EXPERIMENTS
- Each configuration runs 200 tasks in three environments under both agent-directed and scaffolded prompts, for 4,800 calls total.Task-level bootstrap intervals and aligned pairwise tests are used for uncertainty and significance.Source: §4 · p. 2
- Two raters review 90 failed calls; after adjudication, failure source is resolved for 86, and subtype for 66 of 81 calls resolved as agent failures.A frozen, stratified sample of 60 utterances enriched for human flags is scored for speech fidelity.Source: §4 · pp. 2–3
- Figure 2 presents task success for agent-directed and scaffolded runs, with Pass@1 as full bars and robust Pass3 as inset bars.The figure covers the combined task set.Source: Figure 2 · p. 3
5. RESULTS
Scaffolding improves robust success but costs call time, while voice agents still struggle to recognize risk and repair incorrect captures. Caller realisms show no detectable exact-success effect, and errors compound across multi-field calls.
5.1. Strategy selection and robustness
- Pass3 spans 14–41% agent-directed and 37–54% scaffolded; scaffolding raises Pass3 by 14–31 points while adding 21–28 seconds per call.Grok remains best and significantly exceeds every other agent-directed system on both measures; no other pair differs.Source: §5.1 · p. 3
5.2. Risk recognition and recovery
- Systems increase verification for hard and unfamiliar entities, but not for their weakest caller voice; only Grok increases effort under noise.GPT xhigh, Gemini, and Grok initially capture fields incorrectly at rates of 69%, 65%, and 42%, respectively.Source: §5.2 · p. 3, Figure 3 · p. 3
- Only 27%, 37%, and 24% of verified GPT xhigh, Gemini, and Grok errors end correct.Among 347 initially wrong captures, recovery is 10% without a repair step, 26% with read-back only, 24% with spelling only, and 35% with both; because agents choose when to verify, these are associations.Source: §5.2 · p. 3
- Caller-voice success gaps reach 27 points for GPT xhigh, 28 for Gemini, and 15 for Grok, but no system increases verification for its lowest-performing voice.Only Gemini shows evidence of success differences across all five voices after correction (p = .040); no individual voice pair remains significant.Source: §5.2 · p. 3
5.3. Stress tests and diagnostics
- No assigned caller realism detectably changes exact success in either arm; in agent-directed runs, mispronunciation adds 24 points of spelling requests and 26 seconds.Its success estimate changes by only −0.5 points.Source: §5.3 · p. 3, Figure 4 · p. 4
- Dates reached 60.0%; the pattern is consistent with format-constrained versus lexical capture, since dates and phone numbers follow tightly scoped numeric schemas.The agent is shown neither the medication nor coined-name list.Source: §5.3 · p. 4
- At three fields, only 0.53 of calls pass despite 0.75 individual-field accuracy; validation and retry raise task accuracy from 0.64 to 0.82.The intervention combines validation and retry, so the gain is not attributed to either mechanism alone.Source: §5.3 · p. 4, Figure 5 · p. 4
- A matched text run passes all 200 tasks, implicating spoken capture and repair rather than task understanding.Human review assigns 81 of 90 failed calls to the agent, two to user-simulator logic, and none solely to infrastructure.Source: §5.3 · p. 4
6. CONCLUSION
Voice agents remain unreliable at exact entity collection, although a strict elicitation protocol helps substantially. The study’s scope is limited to simulated English settings and a small set of providers and environments.
6. CONCLUSION
- Scaffolded systems succeed across all three environments on only 37–54% of tasks, and just 24–37% of verified errors are repaired.Failures arise both because agents do not consistently recognize risky captures and because verification often fails to repair errors.Source: §6 · p. 4
- The study covers English, three providers, one simulator, and one run per main environment, while caller-side ASR and end-to-end speech-synthesis effects are excluded from task reward.The multi-field study is formative; broader provider, human-caller, and multilingual validation remain future work.Source: §6 · p. 4
- The authors thank Ben Shi, Ola Zytek, Vijay Iyengar, Ajeet Grewal, and Clay Bavor for feedback and support.Source: §6 · p. 4
7. COMPLIANCE WITH ETHICAL STANDARDS
The benchmark uses synthetic or public non-sensitive values and simulated callers and provider voices. The paper notes that exact-value elicitation is dual use and that simulated English results do not establish safety or deployment readiness.
7. COMPLIANCE WITH ETHICAL STANDARDS
- The benchmark uses no real customer records or recordings; values are synthetic or public and non-sensitive, and calls use simulated callers and provider voices.Artifacts will be screened for incidental personal data or secrets before release, with flagged items excluded.Source: §7 · p. 5
- Exact-value elicitation can improve service calls but can also facilitate collection of sensitive identifiers.The authors recommend minimizing collection, explaining its purpose, protecting stored values, and supporting human escalation.Source: §7 · p. 5
- The authors report support from Sierra and Mercor and disclose that they are employees of those companies, respectively, with no other relevant interests.Source: §7 · p. 5