Source-linked AI summary
From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench
Ke Xu, Yuhao Wang, Yu Wang
TL;DR
Existing benchmarks emphasize reactive responses, while proactive voice agents must infer when assistance is appropriate and remain quiet until relevant triggers occur. ProVoice-Bench evaluates this behavior across four tasks, and state-of-the-art MLLMs show substantial gaps, especially in over-triggering and moving from decisions to execution.
Problem
Existing voice agents mainly respond to explicit instructions, while proactive interaction requires inferring implicit needs and detecting intervention triggers across audio and context.
Source: §1 · p. 1Method
ProVoice-Bench evaluates proactive voice agents with 1,182 curated samples across four tasks involving intent, user-defined triggers, acoustic events, and digital-context contradictions.
Source: §1 · p. 1Results
Current MLLMs show a substantial performance gap, including widespread over-triggering and a divide between deciding to speak and executing an appropriate task.
Source: §4 · p. 4, §3.3 · p. 4Takeaways & Limitations
The evaluation indicates that current models are not yet fully equipped for reliable proactive interaction, particularly when context awareness and environment sensing are needed.
Source: §1 · p. 1, §3.3 · p. 4
Abstract
from arXiv · showhide
Recent advancements in LLM agents are gradually shifting from reactive, text-based paradigms toward proactive, multimodal interaction. However, existing benchmarks primarily focus on reactive responses, overlooking the complexities of proactive intervention and monitoring. To bridge this gap, we introduce ProVoice-Bench, the first evaluation framework specifically designed for proactive voice agents, featuring four novel tasks. By leveraging a multi-stage data synthesis pipeline, we curate 1,182 high-quality samples for rigorous testing. Our evaluation of state-of-the-art Multimodal LLMs reveals a significant performance gap, particularly regarding over-triggering and reasoning capabilities. These findings highlight the limitations of current models and offer a roadmap for developing more natural, context-aware proactive agents.
1. Introduction
Existing voice agents primarily wait for explicit instructions, while proactive interaction requires sensing when help is useful and remaining dormant until triggers occur. ProVoice-Bench evaluates this gap across audio and digital-context tasks.
1. Introduction
- Current voice agents mainly respond to explicit instructions, limiting their ability to infer implicit needs or detect when intervention is warranted.The paper contrasts this reactive paradigm with human communication, which flexibly responds to ongoing context and environment.Source: §1 · p. 1
- Prior proactive-agent studies focus mainly on visual cues and implicit triggers, leaving audio information and user-defined trigger points less explored.The paper names ContextAgent [14] and ProAgent [15] as examples of the visual focus.Source: §1 · p. 1
- ProVoice-Bench introduces 1,182 curated samples across four proactive tasks: implicit-intent tool calls, semantic and acoustic triggers, and context contradiction checks.The tasks are Proactive Intent Capture, Latent Topic Monitor, Environment Sound Sensing, and Context Fact Checking.Source: §1 · p. 1
- The four tasks target distinct proactive capabilities, and evaluation of state-of-the-art MLLMs reveals a substantial gap for reliable proactive interaction.The benchmark is supported by a specialized data-synthesis pipeline.Source: §1 · p. 1
2. ProVoice-Bench Construction
ProVoice-Bench evaluates agents that combine conversational audio and digital context to decide when to intervene and what response or tool call to make. Its construction pipeline builds scenarios, speech, and acoustic conditions around four proactive tasks.
2.1. Task Overview
- The benchmark maps conversational audio and digital context to tool requests and textual responses, covering implicit intent, user-set triggers, acoustic events, and context contradictions.Each sample also includes a semantic cue and ground-truth tool-call and response references.Source: §2.1 · p. 2
2.2.1. Digital State Construction
- Digital states are synthesized from randomly selected dialogue-topic themes and include implicit cues such as appointments, dietary restrictions, or medical conditions.These contextual variables ground personas and inform whether later intervention is justified.Source: §2.2.1 · p. 2
2.2.2. Scene Synthesis
- Scenes combine digital context, a task category, and available tools, specifying trigger cues, conversational context, and temporal metadata.The metadata is designed to support interpretation of relative timing and event sequences.Source: §2.2.2 · pp. 2–3
2.2. Data Construction Pipeline
- The five-stage pipeline constructs digital states and scenes, generates speech, simulates acoustics, and assembles conversations; Figure 2(b) summarizes cue-to-audio synthesis.Figure 2(a) shows the distribution of data across the four tasks.Source: §2.2 · p. 2, Figure 2 · p. 3
2.2.3. Conversation Generation
- Naturalistic dialogue scripts are converted to speech with gender-matched audio prompts; User1 is the assisted user, while User2 is an interlocutor that may be omitted.Environment Sound Sensing also incorporates sound events from ESC-50.Source: §2.2.3 · p. 3
2.2.4. Acoustic Simulation
- Acoustic simulation normalizes loudness, models far-field User2 speech with filtering and attenuation, and adds reverberation using a stochastic room impulse response.The output uses a wet/dry ratio of 0.3.Source: §2.2.4 · p. 3
2.2.5. Conversation Assembly
- Conversation assembly samples inter-turn intervals from clipped Gaussian distributions, using longer intervals for ESS, and mixes overlapping speech additively.Ambient noise is also incorporated from CochlScene [23].Source: §2.2.5 · p. 3
3. Experiment and Result
ProVoice-Bench evaluates proactive voice agents on 1,182 multimodal samples, using measures of intervention decisions and response quality. Experiments reveal over-triggering, gains from Chain-of-Thought on analysis-intensive tasks, and a gap between deciding to intervene and executing an appropriate response.
3.1. ProVoice-Bench Overview
- ProVoice-Bench contains 1,182 multimodal samples balanced between positive and negative instances.The benchmark is designed to evaluate proactive interaction across diverse scenarios.Source: §3.1 · p. 3
3.2. Metrics
- Evaluation measures intervention decisions with Accuracy, False Positive Rate, and Recall, and response quality with Response Accuracy.Response Accuracy integrates tool-calling precision and semantic response alignment, then averages instance scores across the benchmark.Source: §3.2 · pp. 3–4
3.3. Results on ProVoice-Bench
- Models commonly over-trigger in LTM and fail to recognize conversations without digital-context violations in CFC.In LTM, most models respond regardless of whether a trigger exists.Source: §3.3 · p. 4
- Chain-of-Thought significantly improves performance on CFC, LTM, and PIC, and helps bridge interaction decisions with appropriate responses.These tasks require analysis of implicit cues in digital context or user speech.Source: §3.3 · p. 4
- Models often show semantic drift or hallucinated tool calls, exposing a gap between knowing when to speak and knowing what to do.The authors call for prioritizing context awareness and environment sensing in MLLMs.Source: §3.3 · p. 4
3.4. Impact of Digital Context
- Removing digital context substantially reduces CFC Recall and lowers PIC Accuracy and Recall.CFC relies on digital context to determine interaction timing, while PIC uses it to infer implicit intentions.Source: §3.4 · p. 4, Figure 3 · p. 4
4. Conclusion
ProVoice-Bench evaluates proactive audio agents across four tasks using 1,182 high-quality samples. State-of-the-art MLLMs show a significant performance gap, particularly in over-triggering and the decision-to-execution divide.
4. Conclusion
- The evaluated state-of-the-art MLLMs show a significant performance gap, particularly in over-triggering and bridging the “decision-to-execution” divide.These findings underscore the necessity of enhanced context awareness for future multimodal agents.Source: §4 · p. 4
- ProVoice-Bench is the first evaluation suite for proactive audio agents, with 1,182 high-quality samples across four novel tasks.It integrates digital context with audio input to shift from reactive responses toward context-aware, proactive interaction.Source: §4 · p. 4
- The authors hope ProVoice-Bench will catalyze development of more natural and autonomous proactive voice assistants.They identify enhanced context awareness as necessary for future multimodal agents.Source: §4 · pp. 4–5
5. Generative AI Use Disclosure
Gemini was used only to edit and polish the manuscript’s language and grammar. The human authors performed the scientific work and remain responsible for the manuscript and its integrity.
5. Generative AI Use Disclosure
- Gemini was used solely to edit and polish the manuscript’s language and grammar for readability.Source: §5 · p. 5
- The human authors performed all scientific analysis and interpretation of results.Source: §5 · p. 5
- The human authors wrote the manuscript and remain fully responsible for the work and its integrity.Source: §5 · p. 5