Source-linked AI summary
Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Terumi Chiba, Guangzhi Sun, Zheqi Yuan, Chao Zhang
TL;DR
Clean written instructions dominate computer-use agent evaluation, although spoken transcription can alter task-critical details and conventional ASR metrics do not directly measure whether those details remain useful for execution. Talk2Agent benchmarks human-spoken instructions and introduces a task-conditioned, execution-free grader projection; on WCB, its Native measure improves PCC over WER/CER and correlates with downstream task performance.
Problem
Computer-use benchmarks assume clean written instructions, while conventional ASR metrics do not directly assess whether spoken task-critical information is preserved for execution.
Source: §1 · p. 1Method
Results
0.246 absolute PCC improvement over WER/CER: Native projection better correlates with downstream task performance on WCB.
Source: §1 · p. 1Takeaways & Limitations
Abstract
from arXiv · showhide
Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems. Speech input introduces an additional failure point: transcription errors can alter task-critical entities, constraints, or targets before the agent begins reasoning, while conventional ASR metrics do not directly measure whether the information required for successful execution has been preserved. We introduce Talk2Agent, a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. Talk2Agent builds human-spoken versions of tasks from WildClawBench and OSWorld and evaluates a range of voice interfaces, including dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair. Because repeatedly executing long-horizon computer-use tasks is costly and stochastic, we further propose an execution-free, task-conditioned evaluation framework that projects the original task grader onto prompt-addressable intentions and measures how much task-relevant information is retained after the voice interface. On WildClawBench, Talk2Agent's execution-free native projection provides a practical, execution-grounded measure of voice-interface quality, correlating with downstream task completion and improving Pearson correlation by 0.246 over WER/CER on 32 hours of real human speech.
1. INTRODUCTION
Talk2Agent addresses the gap between clean-text evaluation and spoken instructions, whose transcription can alter task-critical details that conventional ASR metrics do not directly assess. It evaluates voice interfaces for computer-use agents and proposes an execution-free, task-conditioned measure validated against agent execution.
1. INTRODUCTION
- Talk2Agent builds human-spoken versions of WildClawBench and OSWorld tasks to evaluate how voice interfaces convey instructions to computer-use agents.The benchmark uses fixed downstream agents and contains 31.82 hours of human-recorded instructions.Source: §1 · p. 1
- 0.246 absolute PCC improvement over WER/CER: Native projection correlates better with downstream task performance on WCB.The execution-free measure projects the source benchmark grader onto prompt-addressable intentions and weights them by their contribution to the grader.Source: §1 · p. 1
- Its systematic study covers dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair.The interventions incorporate task-relevant environmental information or attempt to recover corrupted entities and terminology.Source: §1 · p. 1
- The execution-free framework measures retention of prompt-addressable intentions weighted by their contribution to the original task grader, avoiding repeated long-horizon task execution.Full execution is expensive as a routine evaluation mechanism.Source: §1 · p. 1
2. RELATED WORK
Prior spoken-language benchmarks address intent and entity recognition, semantic parsing, and voice-agent tasks, while Talk2Agent targets information needed for computer-use execution. It uses real human speech and isolates the voice interface before a fixed downstream text agent, while retaining task execution as the endpoint.
2. RELATED WORK
- Earlier spoken-language benchmarks evaluate intent and entity recognition and compositional semantic parsing, whereas Talk2Agent targets information needed for computer-use execution.Talk2Agent's focus is task execution rather than these earlier benchmark targets.Source: §2 · p. 1
- Voice-agent benchmarks cover tool use [8], entity-sensitive task completion and conversation quality [10], and full-duplex tasks [11]; one related framework converts text tool-calling benchmarks to synthetic audio [12].The synthetic-audio framework retains tool schemas and labels.Source: §2 · pp. 1–2
- Unlike approaches that jointly evaluate recognition, reasoning, and action, Talk2Agent uses real human speech and isolates the voice interface before a fixed downstream text agent.Task execution remains the endpoint for evaluation.Source: §2 · p. 2
- Semantic and application-oriented ASR metrics do not directly encode the source computer-use benchmark's grading logic, which Talk2Agent projects onto prompt-addressable intentions.The projection retains task-specific weights and non-additive interactions.Source: §2 · p. 2
3. TALK2AGENT BENCHMARK
Talk2Agent combines command-line tasks from WildClawBench with graphical desktop tasks from OSWorld, preserving each source task and evaluator. It collects human recordings of speakable task prompts and compares voice-interface methods, with execution passed to fixed agents and scored by the original evaluator.
3.1. Benchmark construction
- WildClawBench contributes 60 formal command-line tasks and OSWorld contributes 351 English-language graphical desktop tasks, each retaining its original evaluator.The benchmark assesses actual agent execution rather than transcript similarity alone.Source: §3.1 · p. 2
- Prompts are rewritten into speakable scripts while preserving execution-critical entities, constraints, literal symbols, and requested outputs; human review checks task intent.WCB tasks are recorded by ten Chinese–English proficient speakers; each OSWorld task has five recordings, totaling 31.82 hours.Source: §3.1 · p. 2
- WCB recordings are manually checked for completeness and intelligibility, while OSWorld recordings are collected through Prolific in 20-task batches under the same device and script-fidelity requirements.WCB uses smartphone or computer microphones in quiet indoor settings, with rerecording as needed.Source: §3.1 · p. 2
- The evaluation pipeline passes voice-interface text unchanged to a fixed downstream agent, then scores execution with the source benchmark evaluator.This endpoint also combines interface information loss with agent capability, environment failures, and run stochasticity; an execution-free diagnostic evaluates the interface before downstream action.Source: §3.1 · p. 2
3.2. Voice-interface methods
- The benchmark compares vanilla ASR, audio-capable LLMs, contextual biasing, and ontology repair, with original and colloquialized prompts as controls.Contextual biasing uses task-local workspace terms; ontology repair restores punctuation and technical terms only when supported by task-local context.Source: §3.2 · p. 2
4. EXECUTION-FREE EVALUATION
Execution-free evaluation estimates voice-interface information retention using the source benchmark’s grading logic, avoiding repeated costly, noisy task execution. It evaluates retained intentions through both native projection and Shapley retention, which capture different aspects of task value.
4. EXECUTION-FREE EVALUATION
- Direct execution is costly and noisy, so the framework estimates retained task information under an idealized downstream agent while grounding evaluation in the source grader.Observed execution scores also conflate interface information loss with planning errors, limited agent capability, environment failures, and run-to-run variation.Source: §4 · p. 2
- Each task’s evaluator is compiled into prompt-addressable intentions and fixed criteria, with the fixed-criteria baseline removed when scoring retained intentions.The compilation preserves sums, products, gates, thresholds, caps, and floors; continuous criteria use unsatisfied and fully satisfied endpoints.Source: §4 · p. 2
- The conditioned evaluator game yields exact Shapley values that allocate task value across intentions.These values support the Shapley-retention diagnostic.Source: §4 · p. 2
- Native projection substitutes retained intentions into the original grader, while Shapley retention measures how much additively attributed task value survives.Native projection is an execution-oriented surrogate, not a success probability; neither diagnostic subsumes the other.Source: §4 · p. 2
- Changed or omitted intentions count as lost, while unresolved judgments produce an exact interval; execution outcomes are withheld during retention assessment.Midpoints are used only when a single value is needed for aggregate analysis.Source: §4 · pp. 2–3
5. EXPERIMENTAL SETUP
The experiments compare voice interfaces and task-local interventions across command-line and graphical tasks, then validate execution-free scores against direct execution and other metrics on matched WCB conditions.
5.1. Interface and agent configurations
- WCB and OSWorld experiments compare dedicated recognizers, audio-capable LLMs, contextual biasing, and ontology repair with fixed downstream agents.WCB uses ten interface conditions and three executions per task–interface pair; OSWorld compares five voice-interface options with two downstream agents.Source: §5.1 · p. 3
5.2. Execution-free validation protocol
- The WCB validation protocol audits graders before execution results, yielding 549 prompt-addressable intentions across 60 tasks and 3.00% unresolved retention judgments.Suspected errors are independently re-reviewed, and remaining uncertainty is propagated through the interval procedure.Source: §5.2 · p. 3
- Correlation validation uses 500 matched task–interface observations from 50 non-Safety WCB tasks; safety tasks are excluded because omitted unsafe intentions can improve safety-aligned scores.It compares execution-free scores with direct execution and higher-is-better WER/CER, SemDist, AER, Atomic Rubric, and Semantic WER, reporting PCC at three aggregation levels with task-grouped bootstrap intervals.Source: §5.2 · p. 3
6. RESULTS
Execution-free task-conditioned scores track WildClawBench execution better than generic metrics at the task–interface level, while OSWorld results show that intervention effects vary by recognizer and agent. Targeted diagnosis and cross-benchmark transfer reveal interface-level gains that end-to-end execution can obscure.
6.1. Operational benchmark results
- On the frozen 60-task WCB execution artifact, no intervention is uniformly beneficial across the full cohort, and Parakeet changes little across settings.The artifact uses ten matched interfaces and three runs per task–interface pair, equally weighted across tasks and runs.Source: §6.1 · p. 3, Table 2 · p. 3
- On OSWorld, ontology repair raises GPT-4o and Whisper execution means by 0.79 and 0.70 points with Gemini, while Parakeet biasing changes them by +0.13 with Gemini and −0.11 with Qwen.These execution effects depend on both the recognizer and downstream agent.Source: §6.1 · p. 3, Table 3 · p. 4
6.2. Does execution-free scoring track execution?
- Native exceeds AER, the best generic metric, by 0.180 at the task–interface level, while task-conditioned metrics remain statistically comparable to AER at the interface-mean level.Shapley and Native exceed AER in 98.95% and 99.89% of task-grouped bootstrap replicates by PCC; at the interface-mean level, AER is numerically higher than Shapley (0.916 versus 0.902), but not significantly so.Source: Table 4 · p. 4, §6.2 · p. 4
6.3. What does execution-free diagnosis reveal?
- In a preselected workspace-intensive probe, ontology repair improves Parakeet execution by 11.85% without biasing and 16.09% with biasing, while biasing itself has no consistent benefit.Across all 50 tasks, ontology repair still recovers interface-level value, while end-to-end execution changes by less than 0.3%.Source: §6.3 · p. 4, Table 5 · p. 4
- On OSWorld, ontology repair improves execution-free intent accuracy by 7.96 percentage points across five paired Whisper takes.This transfer result concerns graphical tasks; execution gains are smaller because downstream-agent factors are absent from interface-level retention.Source: Table 6 · p. 4, §6.3 · p. 4
- The targeted probe exposes intervention effects where workspace content is relevant, while full-cohort retention isolates interface improvement obscured by downstream execution variability.The authors report this pattern for the targeted probe and full-cohort comparison.Source: §6.3 · p. 4
- Shapley retention and Native projection provide complementary diagnoses: on the full cohort without biasing, ontology repair yields 54.84% Shapley retention but 23.24% Native projection.The difference suggests many individually valuable requirements are restored, but some joint requirements are not satisfied.Source: §6.3 · p. 4
7. CONCLUSION
Talk2Agent combines a human-spoken benchmark with execution-free evaluation that projects benchmark graders onto prompt-addressable intentions. Its measures offer complementary diagnoses without repeated downstream execution, but natural voice use and context selection remain open challenges.
7. CONCLUSION
- Talk2Agent presents a human-spoken benchmark and execution-free framework for evaluating voice interfaces to LLM agents.Native and Shapley measures provide complementary diagnoses of task-relevant information loss while retaining execution-based validation.Source: §7 · p. 4
- Long written prompts may not reflect natural voice use, as speakers may omit, paraphrase, or interactively resolve URLs, paths, code, and other structured content.The conclusion identifies these differences as an open challenge for the benchmark setting.Source: §7 · p. 4
- Workspace-derived bias lists did not consistently improve agentic task execution, leaving context selection and the division of recovery between interface and agent unresolved.Source: §7 · p. 4