Source-linked AI summary
Voice Agents under Acoustic Stress: From Signal Degradation to Interaction and Action
Amir Ivry, Kai-Wei Chang, Lin Zhang, Sharon Gannot, Carlos Busso
TL;DR
Acoustic stress can affect what a voice agent understands and does, while existing benchmarks use varied tasks, conditions, and scoring rules. TRACE offers a practical workflow comparing the same agent’s performance on original and stressed audio, then scoring task outcomes, recovery, and user effort. Its conclusions apply to the tested tasks and conditions, and the paper calls for empirical study across agents, tasks, and acoustics.
Problem
Existing robustness benchmarks use different tasks, conditions, and scoring rules, and answering accurately does not ensure that an agent seeks needed information or completes a task correctly.
Source: §1 · p. 1Method
Results
TRACE provides one practical workflow for evaluating acoustic robustness through task outcomes, errors, and user effort, rather than reporting empirical results across agents or tasks.
Source: §7 · p. 4Takeaways & Limitations
TRACE findings can guide targeted agent changes, such as asking directly for a missing street number and testing whether that clarification policy improves task success or reduces extra user turns.
Source: §1 · p. 1, §5 · p. 4TRACE conclusions are limited to the tasks and acoustic conditions tested, so usefulness in other settings requires further assessment.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Voice agents must complete users' tasks despite noise, reverberation, and competing speech. Evaluating agents' robustness therefore requires following how acoustic conditions affect the conversation and the actions taken on the user's behalf. This overview examines what existing benchmarks reveal about agents' ability to complete tasks under acoustic stress and where further task-based evaluation is required. We then introduce TRACE, a practical workflow for designing, running, and interpreting evaluations of acoustic robustness in task-oriented human-agent interactions: the same agent attempts a specified task with an original recording and an acoustically stressed copy, and the resulting conversations are scored for task completion, wrong actions, recovery, and user effort. Finally, we explain how results from these evaluations can guide changes to an agent to prevent wrong actions and improve recovery.
1. INTRODUCTION
Acoustic robustness must be judged by what an agent does with a noisy request and the task outcome, not just by answer quality. TRACE is a practical workflow for testing application-specific behavior and interpreting task completion, wrong actions, recovery, and user effort.
1. INTRODUCTION
- Answer quality alone may miss whether an agent recognizes when it needs more information, while broader benchmarks examine task completion and conversational behavior –.Pardon? finds models can answer successfully yet fail to request clarification when essential information is missing.Source: §1 · p. 1
- Practitioners still need a reproducible test tailored to an application, with explicit expected behavior and criteria for judging outcomes.Existing studies use different tasks, conditions, and scoring rules.Source: §1 · p. 1
- TRACE guides practitioners in identifying what to test, running tests that score completion, wrong actions, recovery, and user effort, and using findings to guide agent changes.The workflow retains original audio, stresses a copy, and tests both versions with the same agent.Source: §1 · p. 1, Figure 1 · p. 1
2. HOW ACOUSTIC STRESS AFFECTS AGENT BEHAVIOR
Acoustic stress can change what an agent understands, whose words it follows, and when it acts. Each can alter the task outcome, including by causing a wrong action or requiring extra user effort.
2. HOW ACOUSTIC STRESS AFFECTS AGENT BEHAVIOR
- Noise or reverberation can make the agent mishear a word, while clipped recordings can yield transcribed words that were never spoken.A misheard “fifteen” as “fifty” can send a package to the wrong address.Source: §2 · p. 1
- In the delivery example, immediately acting on a misheard number sends the package to the wrong address, whereas clarification can correct it at the cost of an extra user answer.The task outcome depends on how the assistant responds to the noisy request.Source: Figure 2 · p. 2
- Competing speech can make an agent attribute a bystander’s words to the user, while overlap can hide a correction until after the agent acts [8].An early tool call may retain an outdated destination even if the agent later understands the correction.Source: §2 · p. 2
3. WHAT EXISTING BENCHMARKS TELL US
Existing benchmarks offer complementary tests, but their scores can leave important aspects of task-oriented robustness unmeasured. TRACE calls for matched clean and stressed tasks and assessment of outcomes throughout the conversation.
3. WHAT EXISTING BENCHMARKS TELL US
- Noisy-condition accuracy alone does not measure sensitivity to noise; matched clean and stressed versions estimate the added effect, and tool-calling performance changed only modestly across tested noise levels [19].A noisy score measures task accuracy under that condition; comparison with its matched clean condition estimates sensitivity.Source: §3 · p. 2
- The agent with the highest overall task completion rate in τ-Voice also interrupts users most often, so task completion does not characterize the whole interaction.Interruptions need to be measured separately from final task outcome.Source: §3 · p. 2
- A clarification request or corrected transcript does not establish that the right order was placed or that no wrong order preceded it [23] [24].Recovery requires both obtaining a correction and using it to complete the task.Source: §3 · p. 2
- TRACE compares the same agent on the same tasks with and without stress, tracking wrong actions, task completion, and user effort throughout conversations.The checks include mistakes the agent later corrects.Source: §3 · p. 2
4. TRACE: A WORKFLOW FOR TESTING VOICE AGENTS
TRACE organizes acoustic-robustness evaluation into five steps: define the task, establish its recording, modify acoustics, compare matched runs, and score effects. Its delivery example illustrates explicit rules and measures rather than reporting measured agent performance.
4.1. T—Task: The goal and rules for completing the goal
- The Task step defines correct behavior, permitted actions, and stopping limits before testing, distinguishing goal attainment from task success without violations.For the delivery example, guessing the missing number or placing a wrong order is prohibited even if the eventual goal is reached.Source: §4.1 · p. 2, §4.5 · p. 3
4.2. R—Recording: The speech that the agent receives
- The Recording step selects representative speech, documents existing room and microphone effects, and marks the target word while preserving the original audio.The correct address is retained for scoring but not given to the agent.Source: §4.2 · pp. 2–3
4.3. A—Acoustics: The acoustic modification to the original speech and its effect on the available information
- The Acoustics step modifies a copy and checks what information remains, because the expected behavior depends on whether the answer is preserved, missing, or changed.In the example, replacing “fifteen” with silence preserves duration; clarification is required only if the number cannot be determined from remaining audio or context.Source: §4.3 · p. 3
4.4. C—Comparison: Testing original and stressed recordings
- The Comparison step tests the same agent and task in separate fresh conversations using original and stressed recordings, with user-reply rules held fixed.Cooperative users provide the missing number when asked; abandonment cases end without a reply.Source: §4.4 · p. 3
4.5. E—Effects: Task completion, recovery, and user effort
- The Effects step separately scores goal attainment, violations, task success, recovery, user effort, and delay across original and stressed conditions.The hypothetical ten-recording example shows success falling from 90% to 60%, violations rising from 10% to 20%, and mean extra turns rising from 0.1 to 0.8; these are illustrative counts, not measured performance.Source: §4.5 · pp. 3–4, Table 2 · p. 4
5. USING TRACE RESULTS TO IMPROVE THE AGENT
TRACE results can guide targeted agent changes by revealing where task performance breaks down. Repeating the same tests before and after a change measures its effects, including on cases not used in development.
5. USING TRACE RESULTS TO IMPROVE THE AGENT
- TRACE can test speech enhancement and target speaker extraction by comparing task-success rates and response delays with processing enabled and disabled.The comparison uses the same recordings under original and stressed audio conditions.Source: §5 · p. 4
- Comparing broad-repeat requests with a targeted street-number question tests whether clarification improves task success or reduces additional user turns.Recovery is assessed by whether the agent uses the answer to complete the delivery task.Source: §5 · p. 4
- Applying TRACE to cases excluded from development tests whether an improvement extends to new inputs and preserves previously correct task completion.Source: §5 · p. 4
6. LIMITATIONS
TRACE’s guidance is bounded by how well the selected tests represent the intended application; conclusions apply to the tasks and conditions tested.
6. LIMITATIONS
- TRACE’s guidance depends on whether selected robustness tests represent the intended application.Source: §6 · p. 4
- Testing noise only in the initial instruction may miss failures when clarification replies are also noisy.Source: §6 · p. 4
- Conclusions about improvements support only the tested tasks and conditions; usefulness in other settings requires further assessment.Source: §6 · p. 4
7. CONCLUSION
The conclusion presents TRACE as a practical way to evaluate task-oriented voice-agent robustness through outcomes, errors, and user effort. It calls for empirical studies across agents, tasks, and acoustic conditions, including comparison with existing evaluation practices.
7. CONCLUSION
- TRACE evaluates acoustic robustness in task-oriented interactions through what the agent accomplishes, its errors, and the effort required from the user.Source: §7 · p. 4
- Empirical studies should test TRACE across different voice agents, tasks, and acoustic conditions.Source: §7 · p. 4
- Comparisons with existing evaluation practices should determine whether TRACE reveals additional failures and helps practitioners improve task success, while measuring changes in user effort.Source: §7 · p. 4