Source-linked AI summary
VoxMind: An End-to-End Agentic Spoken Dialogue System
Tianle Liang, Yifu Chen, Shengpeng Ji, Yijun Chen, Zhiyang Jia, Jingyu Lu, Fan Zhuo, Xueyi Pu, Yangzhuo Li, Zhou Zhao
TL;DR
End-to-end spoken dialogue models remain limited in complex tasks requiring reasoning, planning, and external knowledge, while tool-augmented spoken agents are comparatively underexplored. VoxMind combines Think-before-Speak reasoning, AgentChat supervision, and asynchronous dynamic tool management. Its overall agent score reaches 74.57, and the paper reports significant gains over strong baselines.
Problem
End-to-end spoken dialogue models remain limited in complex, goal-oriented tasks requiring reasoning, planning, and external knowledge, while spoken agent research lacks unified standards.
Method
VoxMind combines explicit Think-before-Speak reasoning, the reasoning- and tool-annotated AgentChat dataset, and asynchronous dynamic management of a compact local tool space.
Results
VoxMind achieves an overall agent score of 74.57, outperforming StepAudio2 at 34.88, Kimi-Audio at 54.94, and Qwen3-8B + Whisper at 64.00.
Takeaways & Limitations
VoxMind provides an end-to-end spoken agent framework with intrinsic reasoning and tool use for complex agentic tasks.
Takeaways & Limitations
Think-before-Speak introduces inference latency from generating internal reasoning trajectories, and AgentChat’s synthesized text reasoning may reflect written-language structure.
Abstract
from arXiv · showhide
Recent end-to-end spoken dialogue models enable natural interaction. However, as user demands become increasingly complex, models that rely solely on conversational abilities often struggle to cope. Incorporating agentic capabilities is therefore essential: by enabling tool use, these models can extend their knowledge boundaries and better solve real-world tasks. Yet, existing research has largely concentrated on core perception and generation, with comparatively limited exploration of such tool-augmented extensions. To bridge this gap, we present VoxMind, an integrated framework designed to equip end-to-end spoken dialogue models with comprehensive agentic abilities. Leveraging our curated 470-hour AgentChat dataset, we incorporate a "Think-before-Speak" mechanism, enabling the model to internalize structured reasoning as a critical prerequisite for planning and response generation. Furthermore, to mitigate latency bottlenecks caused by large-scale tool integration, we propose a Multi-Agent Dynamic Tool Management architecture. By asynchronously delegating retrieval tasks to an auxiliary agent aligned with the main model's reasoning trajectory, this system effectively decouples inference latency from toolset size. Experimental results confirm that VoxMind achieves significant improvements in agent performance: compared with strong baselines, the task completion rate increases from 34.88% to 74.57%, outperforming Gemini-2.5-Pro on spoken agent tasks while preserving general conversational quality. The source code and associated data are publicly available at https://github.com/MM-Speech/VoxMind.
1 Introduction
End-to-end spoken dialogue models have advanced natural interaction but remain limited in complex tasks requiring reasoning, planning, and external knowledge. VoxMind addresses this gap with explicit reasoning, tool use, AgentChat supervision, and dynamic tool management.
- Motivation: Existing end-to-end spoken dialogue models improve naturalness and responsiveness but remain primarily optimized for reactive conversation.They have limited capacity for complex, goal-oriented tasks requiring reasoning, planning, and external knowledge.
- VoxMind framework: VoxMind introduces a Think-before-Speak mechanism that performs explicit internal reasoning before response generation.The framework integrates autonomous reasoning, tool utilization, and natural spoken interaction.
- AgentChat dataset: AgentChat is a spoken corpus annotated with structured reasoning trajectories and tool interactions for reasoning-aware training.The dataset supports structured reasoning and direct tool invocations from spoken context.
- Dynamic tool management: VoxMind dynamically updates a compact local tool space with candidate tools selected from a global pool.This avoids repeatedly processing the entire tool library and supports responsive decision making in tool-rich environments.
- Dynamic tool management: The proposed dynamic tool management architecture decouples inference latency from tool-library size through asynchronous parallel execution.The mechanism is designed to maintain consistent performance and accuracy as the toolset expands.
2 Related Work
Text-based agents have established reasoning and tool-use mechanisms, while end-to-end spoken agents remain comparatively underexplored and lack unified standards. VoxMind’s architecture places explicit reasoning and dynamically managed tools within spoken interaction.
- Existing agent research: Text-based agents use tool calling and planning to support real-time knowledge access and complex reasoning.End-to-end voice interaction remains a nascent extension of these approaches.
- VoxMind architecture: VoxMind’s architecture uses explicit reasoning before response generation and dynamically updates a local tool space from a global pool.The architecture is presented as a way to integrate agentic capabilities into spoken dialogue.
3 Methodology
VoxMind defines spoken agents around memory, planning, execution, and adaptive interaction, then implements these principles through explicit reasoning, dynamic tool management, and AgentChat training data.
- Unified agent definition: The framework defines spoken agents as autonomous systems with cognitive and executable capabilities beyond reactive speech generation.Its formulation organizes the agent around four essential dimensions.
- Unified agent definition: The agent profile combines static semantic and acoustic attributes with dynamically adapted prosody and emotional expression.Dynamic attributes respond to real-time environmental interaction and self-reflection.
- Planning capability: Planning transforms direct input-to-output mapping into x →z →y, inserting intermediate reasoning for task decomposition and structured execution.The intermediate reasoning stage can represent chain-of-thought, task decomposition, or latent logic generation.
- VoxMind architecture: VoxMind generates explicit reasoning before selecting either a spoken response or an external-tool invocation.The reasoning trajectory captures intent understanding, contextual analysis, and task planning before action execution.
- VoxMind architecture: Parallel dynamic tool updates let the main agent act with local tools while an auxiliary model retrieves candidates from the global pool.Tool expansion is triggered when the local set is insufficient, enabling scalable usage with minimal inference overhead.
- AgentChat dataset: AgentChat combines tool-interaction and general-dialogue data, with cleaning, speech synthesis, and structured reasoning trajectories supporting agent training.The dataset includes tool-related subsets and general dialogue subsets, while its chains are generated from task inputs and final answers.
4 Experiments
VoxMind is evaluated on agent capabilities, general conversation, and dynamic tool management. It achieves higher agent-task performance than strong baselines, while Think-before-Speak supports general speech quality and auxiliary tool management stabilizes inference as tool pools expand.
- Evaluation setup: The evaluation covers six capabilities: single-task processing, task decomposition, parallel processing, proactive seeking, result feedback, and contextual planning.Outputs are checked against predefined ground-truth answers by Gemini-2.5-Flash, with three evaluations averaged for each output.
- Core agent capabilities: 74.57 overall score gives VoxMind a 113.79% relative improvement over StepAudio2’s 34.88 on core agent capabilities.VoxMind also exceeds Kimi-Audio (54.94), Qwen3-8B + Whisper (64.00), and Gemini-2.5-Pro (71.51).
- Ablation study: Without deep reasoning, shifting the data ratio from 1:1 to 1:0.5 improves agent performance only from 68.83 to 70.97 and reduces VoiceBench from 59.72 to 54.80.Think-before-Speak variants show negligible VoiceBench degradation, with a maximum decrease of 0.53 points.
- Ablation study: 74.57 is the peak agent-task score for the Think-before-Speak 1:0.5 configuration, 2.6 points above the 1:1 baseline.Its general evaluation reaches 64.21, exceeding the 1:1 variant and the base model’s 64.15.
- Dynamic tool management: As tool pools grow, single-agent inference time increases exponentially, whereas VoxMind maintains stable inference time and task performance through auxiliary-agent tool management.The controlled study compares configurations with and without the auxiliary tool-management agent across varying tool counts.
- Additional analysis: Additional experiments report strong robustness under realistic speech conditions with only minimal and bounded overhead.These analyses cover robustness to real-world speech, latency-scale decoupling, and token-level overhead.
5 Conclusion
VoxMind establishes a theoretical standard for End-to-End Spoken Agents and presents an end-to-end spoken agent with intrinsic reasoning and tool use. Experiments show strong performance on complex agentic tasks.
- VoxMind establishes a comprehensive definition and theoretical standard for End-to-End Spoken Agents.
- VoxMind integrates intrinsic reasoning and tool use into an end-to-end spoken agent.
- VoxMind significantly outperforms strong baselines on complex agentic tasks.
Limitations
The paper identifies inference latency and dataset naturalness as limitations. Think-before-Speak adds computational overhead, while AgentChat may reflect written-language precision more than spontaneous speech.
- Think-before-Speak introduces inference latency because internal reasoning trajectories precede the final verbal response.The authors describe minimizing this latency as future work.
- AgentChat may reflect written-language precision rather than the spontaneity and disfluencies of authentic daily speech.Future iterations are intended to use datasets rooted natively in spoken scenarios.
F Evaluation of Core Competencies
The evaluation extracts tool calls and separately assesses tool selection and parameter filling under strict matching rules. This yields a reproducible measure of speech-agent tool-call capability.
- The evaluation extracts tool names and parameter name-value pairs from target and model outputs while ignoring textual formatting.No correctness judgment is made during extraction.
- Tool selection is correct only when case-sensitive tool names and occurrence counts match exactly.Evaluation stops immediately when these conditions fail.
- Parameter filling is evaluated only after correct tool selection, with exact parameter values and flexible numeric equivalence or quoting.Parameter names ignore case and spaces, and argument order does not affect evaluation.
- Results are returned as JSON booleans for function-selection correctness and parameter-filling correctness.
- The procedure avoids direct string comparison and provides a precise, reproducible measure of speech-agent tool-call capabilities.
G.1 Training Environment Setup
The experiments describe the training environment, dataset mixing and supplementary data, acoustic robustness evaluation, and analyses of toolset-size latency and output-mode overhead.
- Training Environment Setup: All models were trained with PyTorch 2.6.0, CUDA 12.4, Python 3.10, and NVIDIA GPUs.
- Training Data: The 1:1 data mixture uses the full collected dataset across categories as the baseline.
- Training Data: The 1:0.5 mixture downsamples general conversational and knowledge-intensive tasks by approximately 50% while preserving tool-related categories.
- Supplementary Data: Supplementary data include no-tool audio-text pairs, safety-focused reasoning chains, and text-only dialogues for language-generation stability.The no-tool set contains 2,717 turns and 5.09 hours of audio.
- Acoustic Robustness: The acoustic robustness study compares real recorded speech with TTS-synthesized speech on matched task settings using 150 queries from ToolMind.Real speech includes stutters, hesitations, noisy conditions, and normal speech; TTS uses CosyVoice.
- Efficiency Analysis: VoxMind evaluates latency-scale decoupling across global toolset sizes and token-level overhead across output modes.
H.3 Analysis
Real-speech inputs cause moderate degradation, while VoxMind retains strong task performance and stable Think-before-Speak reasoning under realistic conditions.
- Approximately 7.3% degradation occurs in FS and 6.7% in PF when moving from TTS-synthesized inputs to real speech.
- 86% task success is maintained under realistic speech conditions despite disfluencies and acoustic variability.
- Think-before-Speak remains stable across noisy and disfluent inputs.
- TTS-based evaluation is slightly optimistic but remains informative for estimating real-world performance.
I Experimental Validation of Latency-Scale Decoupling
As the global toolset expands, auxiliary retrieval becomes slower, but parallel execution keeps the main agent’s waiting overhead negligible and task latency practically constant.
- Auxiliary LLM retrieval latency increases from 1.3s to 2.6s as the global toolset expands from 10 to 100 tools.
- Below 15 ms average waiting overhead is maintained as the global toolset grows from 10 to 100 tools.
- Parallel reasoning effectively hides retrieval latency, yielding practical O(1) task execution latency with respect to total tool count.
J Token-Level Analysis of Overhead
Token analysis indicates that Think-before-Speak adds a bounded, predictable overhead rather than a cost that scales with the tool library, especially in speech output.
- 12.6% of speech-output tokens are THINK tokens, indicating negligible additional overhead compared with speech generation.
- Approximately 84 THINK tokens are generated on average in text-output scenarios despite a high THINK-to-answer ratio.
- THINK-token usage remains approximately 80–90 tokens and does not increase with tool-library size.
- The reasoning stage introduces bounded and predictable constant overhead because generation latency scales approximately linearly with token count.