Source-linked AI summary
Asynchronous Tool Usage for Real-Time Agents
Antonio A. Ginart, Naveen Kodali, Jason Lee, Caiming Xiong, Silvio Savarese, John Emmons
TL;DR
Frontier agents can use tools, but current systems remain synchronous and may struggle with asynchronous messages. This paper proposes an event-driven finite-state-machine architecture for asynchronous agents, with speech peripherals, and reports that fine-tuning helps dispatch models generate valid ledger messages consistently.
Problem
Frontier agents are described as largely synchronous, and specialized prompting may still leave them confused by out-of-order asynchronous messages.
Source: §1 · p. 1Method
The paper proposes an event-driven finite-state-machine execution environment for asynchronous agents, with ASR and TTS peripherals, and fine-tunes Llama 3.1 and GPT-4o to operate it.
Source: §1 · p. 1Results
Qualitatively, fine-tuning helps dispatch models consistently generate valid ledger messages according to the prescribed instruction set.
Source: §4 · p. 6Takeaways & Limitations
The architecture is presented as supporting more natural and responsive AI interactions, particularly in voice-based applications.
Source: §4 · p. 6The paper does not delve into context-length and memory management beyond the cited context-length range enabling voice conversations with tool calls of up to 30 minutes or more.
Source: §3.3.1 · p. 4
Abstract
from arXiv · showhide
While frontier large language models (LLMs) are capable tool-using agents, current AI systems still operate in a strict turn-based fashion, oblivious to passage of time. This synchronous design forces user queries and tool-use to occur sequentially, preventing the systems from multitasking and reducing interactivity. To address this limitation, we introduce asynchronous AI agents capable of parallel processing and real-time tool-use. Our key contribution is an event-driven finite-state machine architecture for agent execution and prompting, integrated with automatic speech recognition and text-to-speech. Drawing inspiration from the concepts originally developed for real-time operating systems, this work presents both a conceptual framework and practical tools for creating AI agents capable of fluid, multitasking interactions.
1 INTRODUCTION
The introduction argues that frontier agents are capable of tool use but remain largely synchronous, limiting interaction when tasks take time or messages arrive out of order. It proposes an asynchronous agent framework and execution architecture to address this gap.
1 INTRODUCTION
- Current agents are described as inherently synchronous, while asynchronous concurrency and early replies could reduce perceived delay and improve user experience.The introduction says users may otherwise wait during tool use and cannot submit another query while a serial agent is busy.Source: §1 · p. 1
- Specialized prompting enables some asynchronous operation, but frontier LLMs may still become confused by out-of-order messages.The introduction also notes difficulty stemming from limited asynchronous-tool instruction data and the need for systems-and-ML co-design expertise.Source: §1 · p. 1
- The paper draws on concurrent programming and asynchronous systems to present a conceptual framework and practical architecture for asynchronous agents using frontier language models.Source: §1 · p. 1
- The proposed execution environment uses an event-driven finite-state machine, includes ASR and TTS peripherals, and is modular for LLMs that generate valid messages.The authors say they fine-tune Llama 3.1 and GPT-4o to operate it.Source: §1 · p. 1
2 BACKGROUND & RELATED WORKS
The background section prioritizes seminal systems work and contemporary generative machine-learning work, reflecting the different maturity and pace of those fields.
2 BACKGROUND & RELATED WORKS
- The authors emphasize foundational systems literature and newer generative-machine-learning work because systems paradigms are established while generative ML advances rapidly.Source: §2 · p. 2
2.1 Asynchronous Computer Systems
The section frames asynchronous agents through established work on asynchronous execution and concurrency, alongside real-time systems that combine timing requirements with concurrent peripherals.
2.1.1 Asynchronous Execution.
- Asynchronous execution in the framework draws on Dijkstra’s cooperating sequential processes and Hoare’s formal treatment of asynchronous systems.Source: §2.1.1 · p. 2
- The concurrency discussion contrasts polling and interrupt-based approaches, citing ROS as an event-driven model relevant to responsiveness to inputs such as ASR.It also draws on and Lampson and Redell for trade-offs in concurrency and system responsiveness.Source: §2.1.2 · p. 2
2.1.3 Real-Time Systems.
- Real-time computing systems address timing requirements in concurrent environments with integrated peripheral sensors and actuators.Source: §2.1.3 · p. 2
2.2 Generative AI
The section surveys tool-use models, multi-agent systems, speech models, and dialogue systems as related areas. It distinguishes this paper’s focus on asynchronous real-time operation from work centered on agent coordination.
2.2.1 Large Action Models and Tool-Use.
- Large action models target agent tasks, with xLAM reporting performance on agent benchmarks and APIGen generating function-calling data across APIs and categories.The cited xLAM series ranges from 1B to 8x22B parameters, while APIGen uses over 3,600 APIs across 21 categories.Source: §2.2.1 · p. 2
2.2.2 Multi-Agent Systems.
- Generative-AI multi-agent work includes coordination platforms such as AutoGen [WBZ+23], a DEI-guided committee reported to outperform individual agents, and task-oriented AgentLite.Source: §2.2.2 · p. 2
- The paper focuses on asynchronous, real-time agent operation rather than multi-agent cooperation, while noting that prior coordination ideas could translate to this setting.The section also cites related generative multi-agent systems.Source: §2.2.2 · p. 2
2.2.3 Speech Models.
- Speech-model advances include multilingual and accent-robust ASR with Whisper and TTS work on voice cloning, style transfer, and other generation tasks [CLZ+24].The section identifies virtual assistants and accessibility tools as potential applications of more natural speech interfaces.Source: §2.2.3 · p. 2
2.2.4 Spoken Dialogue Systems.
- Spoken-dialogue research has advanced in dialogue state tracking and dialogue management, with recent developments discussed in.Source: §2.2.4 · p. 2
3 REAL-TIME AGENTS
The real-time-agent framework pairs an asynchronous execution environment with a markup-language prompting specification. Its dialog system schedules messages while a dispatcher manages model generation, context, and function calls.
3 REAL-TIME AGENTS
- A markup-language prompting specification lets the environment asynchronously enqueue function calls and handle chat through speech-to-text and text-to-speech peripherals.The environment relies on the language model producing valid generations according to the specification.Source: §3 · pp. 2–3
- The execution environment centers on a dialog system comprising an event-driven finite-state machine and priority queue, alongside a dispatcher for model generation, context, and function calling.Messages are appended to the ledger as the dialog system processes the scheduling queue; speech peripherals can be replaced with text streams.Source: §3 · p. 3, Figure 2 · p. 3
3.1 Architecture
The architecture combines clock-aware prompting and parallel thought processes with an event-driven dialog system. Priority-scheduled events update the system state and may append messages to the ledger.
3.1.1 Clock Awareness.
- Timestamps and periodic clock updates, such as every 5 seconds, pair with clock awareness in the language model.Clock awareness is supplied through specialized fine-tuning or prompting.Source: §3.1.1 · p. 3
3.1.2 Parallel Thought Processes: Fork vs. Spawn Semantics
- Parallel thought processes are child execution environments: fork copies the parent's ledger, while spawn starts a new ledger with child instructions.Forking can cost more context and include distracting messages; spawning is the usual default, and a reserved kill call can terminate either process.Source: §3.1.2 · p. 3
3.1.3 Event Processing.
- Events from speech input, model generation, speech output, and tool requests enter the dialog system and may trigger state transitions or ledger messages.The event table maps priorities, message presence, and next states across idle, listening, generating, and emitting.Source: §3.1.3 · p. 3, Table 1 · p. 4
- Dispatcher and peripheral state-transition events should receive minimum priority for immediate processing, while tool responses use developer-defined priorities through the scheduling queue.The dispatcher and peripherals may instead use locking to update state atomically.Source: §3.1.3 · pp. 3–4
3.2 Peripherals
The system integrates speech peripherals with an open-source framework, while implementing asynchronous tools separately. It streams model output to a sentence aggregator for phrase-by-phrase speech synthesis.
3.2 Peripherals
- The system uses Pipecat to integrate the language model with speech recognition and synthesis and to help handle user interruptions, but implements asynchronous tools separately.Model generations are streamed through a sentence aggregator and then processed phrase by phrase by text-to-speech.Source: §3.2 · p. 4
3.2.1 Speech-to-Text.
- Deepgram was found to be a good speech-to-text API option, while an optimized Whisper Turbo implementation also achieved low latency.Source: §3.2.1 · p. 4
3.2.2 Text-to-Speech.
- The authors preferred the Sonic API for text-to-speech because of its speed and quality.They observed end-to-end latency below 300 ms, which they considered sufficient to mimic human conversation.Source: §3.2.2 · p. 4, §3.2 · p. 4
3.3 Execution
The execution framework extends ChatML for asynchronous agents and pairs it with a ledger-based environment that handles tool calls, streaming output, and interruptions. Its finite-state machine and scheduling queue coordinate concurrent events while keeping the ledger consistent.
3.3 Execution
- Valid asynchronous ChatML acts as an instruction-set abstraction between the model and execution environment, allowing compatible models to run on environments implementing its specification.The framework treats the prompting format as more than an interface: it separates model messages from environment implementation.Source: §3.3 · p. 4
3.3.1 Prompt Template and Context Management.
- The asynchronous ledger stores messages and tool interactions, while the prompt format allows back-to-back user messages and adds a system-managed notification role.The synchronous context C is a list of role/content messages; the asynchronous ledger also supports tools, reserved fork and spawn calls, and hash strings.Source: §3.3.1 · p. 4
3.3.2 Instruction Set.
- The dispatch model generates assistant messages; the environment streams chat to the user and executes function requests asynchronously, with speech output able to finish before generation does.Request-sent and response-received notifications record tool-call progress in the ledger.Source: §3.3.2 · p. 5
3.3.3 Interruption Handling.
- The finite-state machine tracks idle, generating, emitting, or listening states, according to whether the system is producing tokens, sending output, or receiving user input.The environment must keep the recorded state aligned with the system’s actual state.Source: §3.3.3 · p. 5
- A priority queue makes ledger updates atomic despite concurrency, and the system queues a time-passage notification every 5 seconds.User and assistant messages have example default priorities of -1 and 1, respectively; priorities can be configured per deployment.Source: §3.3.3 · p. 5
- During interruption, the environment reconciles generated chat with what was actually emitted, so the ledger records only output the user heard.Thought streams can be interrupted directly, while streamed chat requires reconciliation because generation and output can be out of sync.Source: §3.3.3 · p. 5
4 DISCUSSION
Fine-tuning helped dispatch models consistently produce valid ledger messages, while the discussion outlines clock-aware, deadline-bound tool use and future directions for real-time agents. The implementation uses separate speech and language components, with multimodal integration as a possible alternative.
Results
- Fine-tuning helped Llama 3.1 and GPT-4o dispatch models consistently generate valid ledger messages.The authors share sample voice-call recordings in the supplementary materials.Source: §4 · p. 6
Pipelined vs. Multi-modal Systems
- A foundation model with text and speech modalities could provide tighter peripheral integration and possibly reduce latency or improve speech quality.Source: §4 · p. 6
- The implementation pipelines independent speech-recognition, dispatch-language, and speech-synthesis networks, using text between them.A multimodal alternative could use speech tokens between speech components rather than neural STT and TTS components.Source: §4 · p. 6
Reasoning for Time-Constrained Tasks
- Clock awareness can coordinate long-running reasoning and repeated tool use across task-specific agents against an allotted deadline.The discussion illustrates this with a research report on text-to-speech models due within 10 hours.Source: §4 · p. 6
Conclusion and Future Work
- Future work includes clock-aware prompting and fine-tuning strategies, multimodal integration, and seamless connections to multi-agent and long-running task-specific agents.Source: §4 · p. 6
- The proposed asynchronous tool use and parallel thought processes support more natural, responsive interactions, particularly in voice applications.Source: §4 · p. 6