PaperlayerView exact revision on arXiv

Source-linked AI summary

Realtime-Venus: A full-duplex interaction system with asynchronous delegation

Ant Group

arXiv:2609.13814v3cs.CVeess.AS

TL;DR

Continuous interaction and background computation unfold on different timescales, making coherent task capture and result delivery an important challenge. Realtime-Venus combines full-duplex conversational models with asynchronous task execution; its models lead across several video and audio benchmarks and show strong conversational continuity.

  • Problem

    Coordinating background task capture with result delivery in a conversation that may change during execution is a challenge for coherent interaction.

    Source: §1 · p. 2

  • Method

    Realtime-Venus pairs full-duplex audio or audio-visual conversational frontends with a harness that executes delegated tasks asynchronously and returns results to the ongoing session.

    Source: §1 · p. 2

  • Results

    Realtime-Venus-Omni led online models on six of eight video benchmarks (70.2 on StreamingBench), while Realtime-Venus-Audio led several audio comparisons and showed high continuation under non-interruptive speech.

    Source: §1 · p. 3, §7.3 · p. 19

  • Takeaways & Limitations

    The findings support coordinating immediate conversational responses with longer-running computation as a promising direction for assistants handling tasks beyond frontend capabilities.

    Source: §8 · p. 23

    The evaluations do not isolate the contributions of individual training components; controlled ablations are needed.

    Source: §7.3 · p. 19

Abstract

from arXiv · show

Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requires grounding the conversation in evolving visual context. We present Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction. Each model serves as a complete conversational frontend, integrating continuous perception, conversational control, and native speech generation through a shared causal timeline for user inputs, model outputs, and delegation events. A dual-loop runtime coordinates live interaction with background reasoning and tool execution. Foreground interaction continues while Realtime-Venus-Harness executes tasks asynchronously and returns results for integration into the ongoing dialogue. Both models follow a common post-training recipe combining offline understanding, proactive full-duplex trajectories, and delegation workflows. Among the evaluated online models, Realtime-Venus-Omni achieves the highest scores on six of eight video benchmarks, including StreamingBench (70.2%), OVO-Bench (64.7%), and Daily-Omni (81.3%). Across eight audio understanding and spoken question answering benchmarks, Realtime-Venus-Audio leads the compared models on MMAU (78.0%), MMAU-Pro (63.2%), Llama Questions (83.8%), and Speech CMMLU (67.8%), while matching the best VoiceBench AlpacaEval score of 4.81. On Full-Duplex-Bench v1.5, Realtime-Venus-Audio responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech, respectively, exceeding Gemini 3.1 Live and GPT-4o on all three continuation metrics.

1 Introduction

Realtime-Venus combines proactive full-duplex interaction with asynchronous delegation, coordinating live conversation and background work on a shared causal timeline. Its design pairs separately trained audio-visual and spoken-interaction models with a framework for executing tasks and returning results to the ongoing dialogue.

Introduction

  • Continuous interaction and external computation operate on different timescales, so coherent delegation requires capturing a stable request and evidence record while delivering results in a conversation that may have changed.The frontend can revise its unspoken continuation after a correction, maintain a response through backchannels, or initiate background work when external capabilities are needed.Source: §1 · p. 2
  • Realtime-Venus-Omni leads evaluated online models on six of eight video benchmarks, while Realtime-Venus-Audio scores highest in several audio and spoken question-answering comparisons.Full-duplex evaluations show high continuation rates under non-interruptive speech; tool-use and delegation-decision evaluations distinguish correct routing from successful task completion and identify challenges in executing external work during conversation.Source: §1 · p. 3
  • The two separately trained 9B models integrate continuous perception, conversational control, native speech generation, and private delegation; Omni supports audio-visual interaction and Audio supports spoken dialogue.The authors describe Realtime-Venus-Omni as, to their knowledge, the first full-duplex omni model to support asynchronous backend invocation while maintaining video interaction; with memory augmentation, it supports hour-scale video understanding.Source: §1 · pp. 2–3
  • Realtime-Venus-Harness binds delegated tasks to request-time evidence, executes registered capabilities asynchronously, and returns results to the originating session while the frontend controls the response.Freshness checks determine result eligibility, and playback-aware delivery governs when results re-enter the conversation.Source: §1 · pp. 2–3
  • A unified data pipeline combines scenario planning, speech realization, and temporal alignment to connect conversational events with delegation requests, background results, and response continuations.Its scenarios distinguish backchannels and other-directed speech from interruptions requiring a response to stop, repair, or redirect; the shared post-training recipe combines offline understanding, proactive duplex interaction, and delegation workflows.Source: §1 · p. 3

2 Related Work

Prior work advances multimodal understanding, speech generation, and streaming interaction, but Realtime-Venus frames full-duplex conversation as jointly controlling responses and private delegation on a shared timeline. Its Harness fixes evidence at the request boundary and returns executed results to the originating dialogue.

Omni-modal understanding

  • Prior omni models add multimodal coverage and streaming output, while Realtime-Venus identifies influence on an ongoing response as the full-duplex requirement.Gemini and GPT-4o support cross-modal understanding and native speech interaction; open models add continuous audio–visual input, time-aligned representations, and streaming speech generation.Source: §2 · pp. 3–4

Audio understanding and speech generation

  • Realtime-Venus’s audio frontend aims to retain acoustic and semantic competence while jointly learning conversational control and delegation.Prior approaches range from Q-Former and multitask-pretrained audio understanding to discrete speech representations, interleaved text–audio generation, and retrieval.Source: §2 · p. 4

Proactive and full-duplex interaction

  • Realtime-Venus extends streaming interaction by placing conversational control, response generation, and private delegation within a shared policy and timeline.Prior full-duplex systems use parallel audio streams, dialogue-state prediction, or turn-control architectures; video-focused methods learn when observations warrant a response.Source: §2 · p. 4

Tool use and asynchronous delegation

  • Realtime-Venus studies private delegation for separately trained audio and omni frontends, with Harness executing registered capabilities and returning results to the originating dialogue.Related systems include asynchronous retrieval, pluggable thinking layers, background delegation, asynchronous function calling, and separate tool-script channels.Source: §2 · pp. 4–5

3 System Overview

Realtime-Venus pairs a full-duplex conversational frontend with a harness that executes delegated tasks asynchronously and prepares replies. Audio and audio-visual frontends share this delegation interface while continuing live interaction.

3 System Overview

  • The Audio or Omni frontend handles live interaction and speech output while Realtime-Venus-Harness executes delegated tasks and prepares replies asynchronously.Figure 3 summarizes the runtime architecture.Source: §3 · p. 5, Figure 3 · p. 5

3.1 Dual-loop architecture

  • The latency-sensitive interaction loop continuously processes media and controls listening, speaking, and yielding while the on-demand capability loop executes tasks over potentially longer durations.A private natural-language request creates a work item from context captured at the request boundary; the harness prepares its reply, and the frontend chooses when to speak it during the ongoing interaction.Source: §3.1 · pp. 5–6

3.2 Unified runtime abstraction

  • Both frontends process one-second audio chunks with optional aligned visual input, retained session state, newly admitted replies, and causal runtime feedback.Outputs distinguish user-facing foreground text and speech from delegation text; delegated requests retain the evidence boundary set at request start, and prepared replies can be admitted later for the frontend to schedule for speech.Source: §3.2 · p. 6

4 Model Design

Realtime-Venus pairs two streaming conversational frontends with shared chunk-based control for perception, speech, and asynchronous delegation. Its design combines long-video memory for Omni, event-aware full-duplex interaction, and a unified training approach across understanding, interaction, and delegation data.

4 Model Design

  • Omni and Audio are separate 9B streaming frontends adapted from MiniCPM-o 4.5, each deciding when to respond and generating speech while delegation runs asynchronously.Omni also includes training-free long-video memory for retaining visual context across hour-long sessions.Source: §4 · p. 7

4.1 Model Architecture

  • Omni-Flow encodes aligned video and audio for Omni, while Audio removes the visual branch; a Qwen3-8B backbone conditions S3 speech tokens and flow-matched waveform chunks.Generated text and hidden states condition discrete S3 speech-token prediction; the decoder uses reference audio from the system prompt.Source: §4.1 · pp. 7–8

4.2 Training-Free Long-Video Memory

  • A training-free external memory gates archived frames by motion-compensated prediction cost, retrieves relevant yet novel frames, and reassembles them with adjacent audio and recent context.Local-change protection and a maximum consecutive-dropping constraint reduce the risk of discarding short-lived visual events; frame gating does not affect normal short-term processing.Source: §4.2 · pp. 8–9

4.3 Unified stream serialization

  • One-second chunks align user, assistant, and background streams on a shared clock, allowing perception and foreground speech to continue while delegated tasks run asynchronously.Backend results are inserted as external context at an available chunk boundary; listen, speak, and turn-end tokens control full-duplex turns.Source: §4.3 · pp. 9–10

4.4 Full-duplex interaction

  • The shared semantic control distinguishes pauses, backchannels, and floor-taking interruptions, continuing speech for backchannels but terminating stale responses when users redirect or correct.Interruption trajectories support stop/reject, repair/update, and redirect/follow-up actions, revising unspoken continuation while leaving played audio unchanged.Source: §4.4 · p. 11

4.5 Training data

  • The over-2.8-million-sample corpus covers offline understanding, proactive duplex interaction, and delegation; approximately 70% is video, while the three capabilities comprise approximately 56%, 37%, and 6%.Omni uses both modalities, whereas Audio is trained only on audio; duplex and delegation data include interruptions as ordinary interaction supervision.Source: §4.5 · pp. 11–12

4.6 Training recipe

  • A unified post-training pass mixes streaming and offline formats across modalities, supervises response spans and per-second listen/speak decisions, and updates only the Thinker while freezing acoustic decoding.Each sample’s weighted loss is normalized by its supervision weight; chunked LM-head cross-entropy bounds activation memory for long answers.Source: §4.6 · p. 12

5 Harness Design

Realtime-Venus-Harness connects full-duplex interaction to asynchronous reasoning and tool execution through a shared interface that tracks tasks from capture to result delivery. Its background work uses evidence fixed at the request boundary, while the frontend integrates results into the ongoing conversation.

5 Harness Design

  • The harness uses capture, dispatch, and return stages with lifecycle tracking, while foreground perception and interaction continue independently.A work item preserves task identity and progress through execution and delivery.Source: §5 · p. 13, Figure 7 · p. 13

5.1 Causal task capture

  • Accepted requests capture a bounded evidence snapshot at the request boundary, preventing later observations or acknowledgments from changing the task’s evidence.A private <delegate> span specifies the natural-language objective; timestamp and sequence fences limit evidence to observations available at the opening token, and prior assistant speech requires playback confirmation.Source: §5.1 · p. 13, §5 · p. 13

5.2 Extensible capability execution

  • The router selects a registered capability from its contract using the objective and compact evidence summary, then execution returns a normalized result shared across capabilities.The registry includes multimodal perception and question answering, general structured planning, and extensible domain skills; new skills provide a contract and executor.Source: §5.2 · p. 14

5.3 Conversational result reintegration

  • The harness prepares a polished private reply from the normalized result, while the frontend chooses when to speak it and generates speech through its native Talker.The reply enters the originating session as private conversational context; spoken delivery may occur later.Source: §5.3 · p. 14

5.4 Work lifecycle and delivery tracking

  • Work items progress from QUEUED through RUNNING to COMPLETED or FAILED, and stale results are excluded from reintegration.COMPLETED means the reply is prepared; DELIVERING begins when it is reserved for session admission, and speech-bearing replies become DELIVERED only after frontend generation and playback acknowledgment of every associated Talker chunk.Source: §5.4 · pp. 14–15

6 Full-duplex and Delegation Data Pipeline

The pipeline represents user, assistant, and environmental events alongside delegation on a shared timeline, enabling unified supervision of conversational control and background computation.

6.1 Unified data construction

  • The unified pipeline plans interaction events, realizes overlapping speech and acoustic context, then aligns modalities and system actions for model-specific materialization.Scenario plans cover varied settings and interaction conditions; speech includes disfluencies and secondary speakers, and audio–visual sessions synchronize audio with video. A common clock preserves event spans, overlap, response boundaries, and delegation annotations across audio-only and video data.Source: §6.1 · pp. 15–16

6.2 Coupling duplex and delegation behaviors

  • Each trajectory couples an immediate duplex action with computational handling, using the event’s effect on user intent to coordinate response control and delegation.Backchannels and background speech generally preserve active work, while intent-changing corrections can cancel dependent work and trigger replanning; tool-dependent requests pair immediate feedback with asynchronous execution.Source: §6.2 · p. 16

7 Evaluation Protocol

The evaluation covers video understanding, audio understanding, spoken question answering, full-duplex interaction and tool use, and delegation. Results show strong benchmark performance alongside distinct limitations in overlap handling, tool execution, and delegation decisions.

7 Evaluation Protocol

  • The protocol evaluates Omni on streaming and offline video, both frontends on audio understanding, spoken question answering, and overlap handling, and both frontends on tool use and delegation decisions.Streaming predictions use observations available at query or response time, whereas offline tasks can access the sampled clip before answering.Source: §7 · p. 17, §7.2 · p. 18

7.1 Evaluation datasets and metrics

  • The evaluation covers streaming and offline video, audio understanding, spoken question answering, full-duplex interaction and tool use, and delegation, with benchmark-specific scores including PAUC, accuracy, F1, argument accuracy, Pass@1, and routing measures.Source: §7.1 · pp. 17–18

7.3 Omni results

  • 70.2 on StreamingBench and 64.7 on OVO-Bench help Realtime-Venus-Omni achieve the highest scores among online models on six of eight video benchmarks.It scores 81.3 on Daily-Omni; memory augmentation improves its accuracy by 5.88 points on the 60–90-minute LVOmniBench subset and 4.76 points on the 40–60-minute LongVideoBench subset. It scores below MiniCPM-o 4.5 on ProactiveVideoQA and WorldSense.Source: Table 2 · p. 18, §7.3 · p. 19, Table 3 · p. 19

7.4 Speech results

  • 78.0% on MMAU and 63.2% on MMAU-Pro are Realtime-Venus-Audio’s highest compared-model scores; it ties the best VoiceBench AlpacaEval score at 4.81 and leads Llama Questions at 83.8% and Speech CMMLU at 67.8%.It ranks second on MMAR and Speech TriviaQA; Qwen3-Omni and Fun-Audio-Chat score higher on MMSU. Aggregate scores do not separate speech-interpretation errors from answer-generation errors or establish broader reasoning capability.Source: Table 4 · p. 20, §7.4 · pp. 20–21, Table 5 · p. 20

7.5 Full-duplex results

  • 75% interruption response and continuation rates of 97%, 88%, and 86% under backchannels, speech to others, and background speech characterize Realtime-Venus-Audio on Full-Duplex-Bench v1.5.Realtime-Venus-Omni achieves 86.0% tool selection F1, 53.1% argument accuracy, and 43.0% Pass@1; GPT-Realtime leads all three at 87.6%, 68.0%, and 60.0%, respectively. Strong continuation does not imply strong interruption handling, and the metrics use different success criteria.Source: Table 6 · p. 21, §7.5 · pp. 21–22, Table 7 · p. 22

7.6 Delegate Benchmark

  • 75.93% overall routing accuracy gives Realtime-Venus-Omni a 7.04-point advantage over Realtime-Venus-Audio, while Audio leads delegation recall 92.22% to 68.33% and Omni leads non-delegation specificity 84.44% to 39.44%.Both variants achieve 75.00% reasoning-category routing accuracy. Audio more often delegates routine requests unnecessarily, while Omni more often misses requests requiring delegation; the benchmark tests decisions, not execution or integration of delegated results.Source: Table 8 · p. 23, §7.6 · p. 23

8 Conclusion and Future Work

Realtime-Venus supports coordinating immediate conversation with longer-running computation while keeping interaction active. Future work targets finer streaming, longer context, and more complex asynchronous tasks.

8 Conclusion and Future Work

  • Realtime-Venus demonstrates competitive multimodal understanding and strong conversational continuity under non-interruptive speech, while background reasoning and tool execution proceed alongside active interaction.The frontend continues receiving inputs and managing speech as background work runs.Source: §8 · p. 23
  • Memory augmentation improves hour-scale video understanding across all evaluated duration bins, supporting coordination of immediate responses with longer-running computation.Source: §8 · p. 23
  • Future work will explore finer streaming chunks, longer context, and diverse multi-step tool tasks, including concurrent-operation management and recovery from delayed or failed calls.The goals include capturing brief events, improving response timing, retaining relevant information, and tracking evolving user intent.Source: §8 · p. 23

Appendix · B Training Data Example

The appendix illustrates training examples for proactive duplex interaction and delegate workflows, showing how raw records are rendered in a unit-level format.

B Training Data Example

B.1 Omni-Proactive Example

The Omni proactive example trains event-triggered speech: the assistant listens silently until a requested event occurs, then responds immediately and stays silent otherwise.

B.1 Omni-Proactive Example

  • The assistant receives an instruction at stream start and must remain silent both before and after the requested event.The task requires the response to occur only when the event happens.Source: §B.1 · p. 27
  • The example asks the model to notify the user when a commentator reacts to a wicket.The instruction arrives with image and audio input, followed by listen units before the response.Source: §B.1 · p. 27
  • When the commentator reacts, the model immediately speaks in chunks describing the reaction to the wicket.Its utterance reports that the commentator reacted and quotes the exclamation, “Oh, he’s got him!”Source: §B.1 · pp. 27–28

B.2 Delegate Example

The delegate example demonstrates a full delegation cycle within a continuous video session: the frontend acknowledges a spoken query, dispatches it to the Harness, listens while the backend works, and speaks the returned result in chunks.

B.2 Delegate Example

  • The example shows the full delegate cycle within a continuous video session.The frontend returns to listening while the Harness executes the delegated task asynchronously.Source: §B.2 · pp. 28–29
  • The backend reports that today’s restricted plate endings are 2 and 7, so a Beijing plate ending in 7 is restricted within the Fifth Ring Road from 7:00 to 20:00.The model re-presents the result in chunked spoken form within the same conversational timeline.Source: §B.2 · pp. 28–29
  • The frontend acknowledges a question about Beijing traffic restrictions, commits to checking, then dispatches the lookup to the Harness before closing its turn.It returns to listening while the backend executes.Source: §B.2 · pp. 28–29
Loading 2609.13814v3…