Source-linked AI summary
Omni Interaction Agent Technical Report
Orantqing, Shengpeng Ji, Junlong Tong, Jialong Zuo, Dongjie Fu, Di Cao, Yangzhuo Li, Shangda Wu, Franz, Evan, Theron Veyra, Changhao Pan, Jingyu Lu, Dongchao Yang, Zhifei Xie, Yang Tan, Xiaoyu Shen, Xiaoda Yang, Wenfu Wang, Teddysun, Steveyves, Zhou Zhao, Bryanytian
TL;DR
Gander addresses the challenge of unifying continuous multimodal interaction with complex agentic task execution. It combines a streaming Thinker-Talker Cerebellum with a Brain–Cerebellum collaboration framework and orchestration runtime, achieving strong omni interaction and agentic capabilities while retaining natural spoken dialogue. The paper also identifies limits in its spoken-dialogue evaluation and presents Gander as an initial foundation requiring further advances for reliable long-horizon deployment.
Problem
Existing human–AI systems remain largely turn-based, while unified systems must support continuous multimodal interaction alongside long-horizon agentic reasoning.
Method
Gander uses a streaming Thinker-Talker Cerebellum for realtime interaction and a Brain–Cerebellum architecture with orchestration for complex agentic tasks.
Results
Gander achieves strong performance across dialogue, omni understanding, realtime interaction, and agentic capabilities, including robustness to interference, multiparty interaction, and backchanneling.
Takeaways & Limitations
Gander provides an initial foundation for unified omni interaction agents spanning realtime conversation and long-horizon agentic execution.
Takeaways & Limitations
Spoken interaction training does not supply extended discursive composition or broader accented-speech coverage, and the reported front-cerebellum table does not assess back-brain execution quality.
Abstract
from arXiv · showhide
In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework. In contrast to turn-based conventional paradigms, Gander continuously receives streaming inputs across multiple modalities, including video, speech, and text, enabling natural full-duplex interaction in both everyday conversations and complex workflow-oriented agent scenarios. Users can interrupt the model at any time, while the model can also proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) It employs a Cerebellum-Brain collaborative framework, in which the Cerebellum is responsible for realtime interaction and omni conversational capabilities, while the Brain handles complex reasoning and higher-level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, user inputs and model outputs are further flattened into an ordered token stream at the chunk level, providing a unified representation for low latency, continuous interaction. We conduct comprehensive evaluations of Gander across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Internal human evaluations demonstrate that Gander maintains the natural and expressive spoken dialogue capabilities of SOTA open source models while achieving competitive performance in omni interaction. Gander also demonstrates robustness in challenging real-world scenarios, including background noise interference, multi-party interactions, and backchannel communication. We release Gander together with its models, code, and data to facilitate further research and development in the community.
1. Introduction
Gander addresses the mismatch between turn-based human–AI dialogue and fluid multimodal collaboration by unifying realtime interaction with agentic reasoning. It combines intrinsic streaming interactivity with a Brain–Cerebellum design and reports strong performance across dialogue, multimodal understanding, interaction, and agentic capabilities.
- Human–AI interaction remains largely turn-based, unlike human communication, which supports continuous multimodal exchange, interruption, and redirection.
- Gander targets two challenges: making interactivity intrinsic to the model and combining low-latency responsiveness with long-horizon agentic reasoning.
- Its streaming design aligns audio-visual inputs and generated text into temporally ordered chunks, with the model predicting whether to listen or speak within each chunk.
- A Brain–Cerebellum framework separates realtime conversational interaction from complex reasoning and sustained agentic workflows.
- Gander achieves strong capabilities in omni dialogue, interruption, proactive interaction, interference robustness, multiparty backchanneling, reasoning, and agentic tasks.
- The authors release Gander’s model weights and code to support open research, practical deployment, and further development.
2. Related Work
Related work moves from turn-based speech systems toward continuous multimodal interaction and practical voice agents, but conversational control and task execution remain difficult to unify. Gander addresses this gap with a single end-to-end model for realtime conversation and agentic workflows.
- Speech interaction research has progressed from turn-based systems toward models that continuously listen, speak, and regulate participation.
- Omni models extend speech interaction by combining auditory and visual context, including continuous perception of environmental changes.
- Voice agents increasingly connect spoken interaction with code generation, file operations, tool use, and multi-step workflows.
- Realistic workflows require handling interruptions, revised constraints, follow-up questions, proactive updates, clarification, backchanneling, multiple speakers, noise, and visual changes.
- Gander studies whether conversational interaction, multimodal perception, and task execution can operate as one continuous end-to-end process.
3.1. Overview
Gander uses three coordinated components to support continuous realtime interaction alongside complex long-horizon task execution. The front cerebellum handles interaction, the back brain handles reasoning and workflows, and the runtime coordinates them.
- Gander comprises a front cerebellum, an agent orchestration runtime, and a back brain.
- The front cerebellum is a realtime full-duplex multimodal model for continuous perception and interactive communication.
- The back brain is a general-purpose task-execution agent for complex reasoning and long-horizon workflows.
- The orchestration runtime coordinates realtime multimodal inference with asynchronous background-task orchestration.
- Together, the components support continuous realtime interaction and complex long-horizon task execution.
3.2. Cerebellum-Brain Collaborative Framework
Gander separates realtime multimodal interaction from long-horizon task execution through a front cerebellum, an orchestration runtime, and a back brain. Structured tool calls let users continue conversing, refine tasks, and control execution while workflows run.
- Architecture: The front cerebellum handles continuous full-duplex multimodal interaction and routes tasks, while the back brain performs general-purpose long-horizon execution.The back brain supports retrieval, code and file manipulation, document processing, and other tool-mediated workflows without task-specific training.
- Tool interface: Structured tool calls expose task_start, task_send, and task_resolve operations for creating tasks, refining active work, and controlling permissions or termination.The runtime deterministically enforces the associated task-state transitions and execution semantics.
- Interactive execution: During back-brain execution, users can continue realtime conversation or modify previously assigned tasks through the front cerebellum.The workflow illustration depicts task initiation followed by concurrent conversation and task refinement.
- Runtime: The orchestration runtime coordinates realtime data transport, backend task execution, scheduling, and interaction between the two components.It provides a unified interface for multimodal stream ingestion and backend task orchestration.
- Runtime modes: The runtime offers lean and coordinator control modes, trading shorter, more deterministic control paths against additional planning overhead and nondeterminism.Lean mode directly executes front-cerebellum actions, whereas coordinator mode inserts an independent control-plane model for execution directives.
3.3. Streaming Thinker-Talker Architecture
The front cerebellum converts streaming audio-visual perception and generated responses into a unified, chunk-level causal process. Each one-second chunk lets the model decide whether to listen or speak while maintaining bounded context and streaming speech output.
- Architecture: The front cerebellum uses a Thinker-Talker architecture that ingests streaming audio and video and produces text and speech.Its visual and acoustic encoders operate concurrently and incrementally, projecting observations into the LLM token space.
- Streaming chunk flattening: A streaming chunk-flattening mechanism serializes perceptual inputs, a control token, and generated text into one unified causal stream.The model attends to newly arrived perception before generating each chunk’s output, allowing chunk-level listen-or-speak decisions.
- Streaming chunk flattening: Each chunk spans one second and contains time-aligned audio-visual tokens, a predicted control token, and N_text generated tokens, where N_text may be zero.The model makes one interaction decision per window, supporting proactive responses without an external voice activity detector.
- Context management: A sliding budget of 128 chunks keeps the receptive field near two minutes while bounding inference cost during long interactions.Older chunks are evicted as new chunks arrive, preventing unbounded sequence growth.
- Interaction control: The control token selects among listening, speaking, and other interaction behaviors, determining whether the current window remains silent or emits speech content.A listen token advances perception without text output, while a speak token commits the model to generating spoken content.
- Speech generation: Speech generation separates semantic planning from waveform realization through lightweight speech decoders and streaming waveform synthesis.This design reduces backbone decoding burden and supports continuously produced, low-latency audio playback for full-duplex interaction.
4. Data Construction Pipeline
Gander’s data pipeline combines speech, audio-visual, agentic, and robustness-oriented supervision for continuous interaction. The data cover turn-taking, multimodal perception, asynchronous task coordination, and challenging conversational conditions, with filtering and evaluation applied throughout construction.
- Corpus design: The training corpus targets when to listen or speak, evolving audio-visual context, back-brain coordination, and reliable interaction under real-world conditions.It includes realtime speech, audio-visual, agentic, and robustness-oriented data beyond conventional speech and multimodal understanding.
- Speech interaction data: Speech interaction data comprise approximately 37% of the corpus and supervise dialogue, spoken instruction following, full-duplex interaction, translation, interruption, overlap, and response timing.These data teach the model when to listen, continue speaking, and yield.
- Speech interaction data: Speech dialogues are synthesized from 11.2K scene and topic seeds spanning 45 everyday and task-oriented scenarios.Generated dialogues are constrained to 8–18 turns and at most 96 seconds of aggregate speaking time.
- Speech interaction data: Interaction events explicitly model competitive interruptions and supportive backchannels, with timing, overlap, and hidden continuations represented in the supervision.Backchannel candidates are categorized using a bilingual lexicon organized into 11 intent categories.
- Quality control and robustness: Construction pipelines apply rule-based filtering and LLM judging, while robustness supervision covers irrelevant visual context, no-command settings, interference, and multi-party interaction.Agentic pipelines synthesize user queries, front-cerebellum responses, and back-brain execution trajectories before automatic evaluation and filtering.
- Audio-visual interaction data: Audio-visual interaction data account for approximately 40% of the corpus and cover streaming video question answering, narration, and proactive visual response.Samples undergo quality assessment, filtering, and temporal-alignment refinement.
- Agentic interaction data: Agentic interaction data train coordination among the user, front cerebellum, and back brain across asynchronous, long-horizon task execution.Trajectories interleave user turns with backend states, progress, intermediate results, and tool-grounded reasoning over the task lifecycle.
- Agentic interaction data: Omni-agentic data extend coordination to visually grounded tasks using crawled, existing GUI, and Codex-generated trajectories, totaling approximately 36K examples.These trajectories require continuous tracking of evolving visual states alongside user and back-brain interaction.
5. Experiments
Gander is evaluated as a single end-to-end system for full-duplex interaction, spoken conversation, omni understanding, and agentic tool use. Results show strong turn-taking and competitive spoken and multimodal performance, while revealing boundaries in open-ended generation, perceptual detail, and back-brain execution assessment.
- Evaluation Setup: The evaluation jointly considers tool selection, argument correctness, response quality, exact task success, and interaction timing.The benchmark reports ToolSel, ArgAcc, RespQual, Pass@1, and three interaction metrics across 100 scenarios, with consistent decoding and prompting across evaluations.
- Full-Duplex Interaction: Gander achieves the best turn-taking behavior across 100 full-duplex scenarios while remaining competitive on task accuracy.It takes the floor appropriately in all scenarios and begins speaking prematurely in 8.0% of turns; Pass@1 is 0.400, ToolSel 0.759, and ArgAcc 0.503.
- Full-Duplex Interaction: Gander’s interaction timing avoids the trade-off between suppressing interruptions and missing opportunities to respond, although its 51.6% filler rate is not best.The model conditions turn-taking on an evolving representation shared with response generation, while filler reflects time spent holding the floor during delegated work.
- Full-Duplex Interaction: When driven directly by user transcripts, the back brain reaches Pass@1 0.520 and ToolSel 0.934, indicating that execution quality is not the main end-to-end bottleneck.The end-to-end system must decide when to delegate and is evaluated through synthesized speech and ASR, unlike the text-mediated back-brain condition.
- Spoken Conversation: Gander performs competitively in spoken conversation, ranking strongly on SpokenQA while remaining within a few points of leading systems on VoiceBench.It reaches 75.60 and 59.30 on SpokenQA, places second within the full-duplex group, and remains competitive against turn-based systems across the broader field.
- Spoken Conversation: Interaction training preserves much of the model’s spoken factual-question performance but leaves limitations in extended generation and acoustic coverage.The streaming formulation does not by itself reduce retained language knowledge, while short conversational supervision provides limited support for long-form composition and accented speech.
- Omni Understanding: Gander retains competitive omni understanding and benefits substantially from combining audio and video inputs.It reaches 49.62 on WorldSense and 78.53 on Daily-Omni; joint audio-video input improves over the better single stream by 5.01 and 19.13 points, respectively.
- Omni Understanding: The omni-understanding results expose an asymmetric trade-off, with stronger preservation on Daily-Omni than on WorldSense.Daily-Omni remains within 1.67 points of the MiniCPM-o 4.5 initialization, whereas WorldSense falls by more than six points under interaction training.
6. Conclusion and Future Work
Gander provides an initial unified architecture for omni understanding, realtime interaction, and long-horizon agentic execution, while exposing unresolved challenges for reliable deployment.
- Gander unifies omni understanding, realtime interaction, and long-horizon agentic execution through a Brain–Cerebellum framework and streaming chunk-based paradigm.
- Agent invocation and conversational behavior remain sensitive to training-data distribution, especially in complex omni scenarios.The authors identify data and model scaling as important for robustness and generalization.
- Long-horizon Omni Interaction Agent scenarios remain insufficiently optimized with on-policy distillation or reinforcement learning.Reward design, credit assignment, and optimization stability remain open challenges under joint multimodal interaction and agentic execution.
- The current Brain–Cerebellum design relies primarily on ASR-derived signals for communication between its components.Future work should explore richer bidirectional communication and improved coordination between realtime interaction and long-horizon reasoning.
- Omni agentic interaction poses memory and long-context challenges because it combines extended multimodal histories, tool calls, and evolving task states.Efficient retention and retrieval of task-relevant information remains an important direction.
- Existing benchmarks insufficiently evaluate unified omni interaction agents and Brain–Cerebellum communication and coordination.A unified evaluation framework for omni interaction agents remains largely missing.
- Reliable long-horizon deployment depends on advances in scaling, post-training, Brain–Cerebellum coordination, memory, and evaluation.The conclusion specifically emphasizes sustained realtime interaction and complex task execution.