Source-linked AI summary
JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence
Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Haowen Hou, Zheming Liang, Congcong Wang, Yuhang Cao, Shenglong Ye, Shuai Xie, Shuhuan Gu, Haoyang Huang, Qingyi Si, Nan Duan, Jiaqi Wang
TL;DR
Most large models wait for users to prompt them, missing time-critical events that require spontaneous visual interaction. JoyAI-VL-Interaction continuously watches video, decides whether to speak, stay silent, or delegate, and is preferred over Doubao and Gemini across six scenarios.
Problem
Most large models remain turn-based, lacking a mechanism to react to events that occur without a user utterance.
Method
The paper releases an 8B vision-first model trained on time-aligned visual streams to decide per second whether to speak, stay silent, or delegate.
Results
JoyAI-VL-Interaction is preferred in 87.9% of comparisons against Gemini and 77.6% against Doubao.
Takeaways & Limitations
The release provides an open model, training recipe, time-aligned data, and deployable system for genuine streaming interaction.
Takeaways & Limitations
The evaluation remains preliminary, covering six scenarios and 58 human-rated cases against two products.
Abstract
from arXiv · showhide
Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing video into the model, making it genuinely present in the world. All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.
1 Introduction
JoyAI-VL-Interaction introduces a vision-first interaction model that continuously judges when to speak, stay silent, or delegate, rather than waiting for user prompts. It also releases a complete, modular system for real-time video interaction and deployment.
- Motivation and paradigm: JoyAI-VL-Interaction defines interaction as deciding moment to moment whether to respond to an ongoing stream, speak, or remain silent.This criterion distinguishes it from turn-based models that respond only when addressed.
- Contribution 1 — the VL-interaction model: The VL-interaction model makes when to act a learned per-second decision, treating silence as a first-class action alongside speaking and delegating.Speaking and silence support proactivity and time awareness, while delegation enables asynchronous handling of harder problems.
- Contribution 2 — the VL-interaction system: The released system streams webcam or livestream video into the interaction model, which immediately observes and interacts in real time.The system is designed to be genuinely present in the ongoing scene.
- Contribution 2 — the VL-interaction system: The deployable system includes pluggable ASR, TTS, visualization, long-horizon memory, and background bridges to user-selected APIs, models, or agents.The interaction model alone decides whether and when to interact; surrounding components transduce and orchestrate around it.
- Broader significance: The authors frame continuous presence and self-timed interaction as a shift from submitting requests and waiting to watch-and-do collaboration.They position the capability as a foundation for companions, AI glasses, accessibility aids, and other applications in an unpausable physical world.
2 Related Work
Related work largely improves responsiveness while retaining turn-based interaction, whereas newer interaction models move decision-making into the model. JoyAI-VL-Interaction targets a distinct visual, event-driven, deployable setting that combines continual watching with in-model action decisions.
- Turn-based systems: Most existing models remain turn-based, reacting only after a user speaks and lacking mechanisms for events that arrive without user utterances.This limitation is structural rather than merely a matter of response speed.
- Realtime and omni models: Realtime and omni models reduce latency through integrated streaming architectures but primarily optimize conversational turn-taking after the user has spoken.Examples include GPT-Realtime-2 and Qwen3.5-Omni, which supports real-time streaming, turn-taking, and interruption handling.
- Consumer video-call products: Consumer video-call assistants remain turn-based: Doubao caches incoming video frames while no one speaks instead of sending them to the model.These products provide the paper’s most recognizable baseline while relying on conversational-AI pipelines with ASR, VLM, TTS, and turn detection.
- TML interaction models: TML interaction models place interactivity inside the model rather than an external harness, while JoyAI-VL-Interaction deliberately uses a compact ∼8B model.The compact scale is chosen to balance interaction-grade responsiveness, local and low-cost deployment, and capability.
- Streaming video models: Streaming-video research addresses responsiveness, proactivity, real-time inference, or long-horizon memory, but typically studies these properties separately on offline benchmarks.The paper identifies this line as closest to its model contribution while distinguishing it from a deployable interaction model.
- Positioning: Across interaction and release axes, JoyAI-VL-Interaction uniquely places second-by-second, event-driven action decisions inside a vision-first model for watch-and-interact settings.It is distinguished from turn-based systems and TML’s audio–video conversation emphasis by making vision the first-class driver.
3 JoyAI-VL-Interaction Model
JoyAI-VL-Interaction extends a conventional JoyAI-VL model into a continuously streaming, vision-driven agent that decides each second whether to speak, stay silent, or delegate. Its interaction behavior is trained with time-aligned data and a two-loop delegation protocol that keeps the model responsive while background reasoning proceeds asynchronously.
- Base model and architecture: JoyAI-VL-Interaction is built on JoyAI-VL 1.0, whose language model starts from Qwen3-8B and whose visual encoder is the Qwen3-VL ViT.The projection layer is trained from scratch, and interaction behavior is acquired after the base model’s conventional VLM training.
- Base model and architecture: AdaCodec tokenizes streaming video into full ViT reference tokens and compact motion-and-residual P-tokens instead of re-encoding every frame.The resulting input interleaves reference tokens with P-tokens.
- Interaction behavior: At every one-second step, the model watches the visual stream and emits silence, a textual response, or a delegation request.The real-time loop covers speaking and continued watching, while delegation starts an asynchronous loop with a background model.
- Interaction data: The corpus contains more than 4M time-aligned streaming clips spanning proactive alerting, temporal question answering, counting and perception, commentary, and casual chat.Per-second supervision labels both response content and timing, treating silence as a first-class label.
- Delegation training: During delegation, the model gives a brief holding reply, sends a hidden request to the background, and later incorporates the returned result after a randomized delay.The delay trains the model to keep watching, handle new turns, and remain silent while the delegated task is pending.
4 JoyAI-VL-Interaction System
JoyAI-VL-Interaction is a complete, deployable system in which the model alone decides whether to speak or delegate, while surrounding components provide replaceable transduction and orchestration. Its concurrent real-time and asynchronous loops, pluggable modules, hierarchical memory, and open serving stack support continuous, customizable presence.
- System design: The system’s design thesis is “decision in the model, the rest replaceable,” making it composable across domains and deployable with standard infrastructure.ASR, TTS, memory, the background brain, and visualization UI surround the interaction model as replaceable components.
- Real-time and asynchronous loops: The model runs two concurrent conversations: a real-time user loop and an asynchronous background-brain loop joined by the delegate action.Sampled video frames drive one action each second—to speak, stay silent, or delegate—while delegated tasks are handled by the background model.
- Background bridge: A background-agnostic bridge converts delegated foreground queries into a text protocol and can connect the default user-provided large-model API or any user-supplied agent.The scaffold supports agents such as Hermes Agent and OpenClaw while keeping the delegation contract fixed over text requests and results.
- Pluggable modules and UI: Users can replace ASR, TTS, visualization, and perception or memory modules, while the built-in interface supports webcam or RTSP video, configurable sampling, typed or spoken input, and latency monitoring.The system ships ready-to-run open-source speech modules and supports user-supplied modules such as face recognition.
- Memory and open deployment: A pluggable three-tier memory keeps long-running streams bounded while sharing context across the streaming model and background brain, and the complete recipe and serving stack are released openly.The hierarchy uses raw short-term vision tokens, summarized mid-term chunks, aggressively compressed long-term blocks, and dialogue memory; the repository includes orchestration, ASR/TTS, UI, memory, the background bridge, and vLLM serving.
5 Experiments
JoyAI-VL-Interaction is evaluated in six real-world, event-driven scenarios against deployed Doubao and Gemini video-call assistants, with blinded human raters preferring it by wide margins. Its advantage is concentrated in proactive, time-critical interaction, though the comparison is intentionally narrow and the data and evaluation remain early-stage.
- Experimental setup: The evaluation compares JoyAI-VL-Interaction pairwise with deployed Doubao and Gemini assistants across six scenarios spanning real-time operation, proactive response, and long-horizon memory.The scenarios cover monitoring and alerting, counting, translation, time awareness, live commentary and guidance, and long-horizon memory.
- Experimental results: 87.9% of comparisons against Gemini and 77.6% against Doubao preferred JoyAI-VL-Interaction, while losses were 1.7% and 5.2%, respectively.Ties were 10.3% against Gemini and 17.2% against Doubao.
- Experimental results: 100% of monitoring-and-alerting comparisons favored JoyAI-VL-Interaction against both baselines, demonstrating its clearest advantage in vision-driven, time-critical scenarios.In a fall-detection case, it alerted instantly; Doubao reacted four to five seconds later, while Gemini said it could not monitor the scene.
- Experimental results: JoyAI-VL-Interaction was off by only one to two seconds on a twenty-second timing task, whereas Doubao failed to signal and Gemini responded around forty seconds later.The result probes an internal sense of elapsed time rather than a prompted response.
- Emergent capabilities: The model also proactively guided a shopper through changing phone screens despite not being trained for that capability, while neither baseline reacted proactively or in real time.The paper presents this as an emergent capability alongside other untrained behaviors.
- Limitations: The claim is limited to six event-driven scenarios: JoyAI-VL-Interaction is an 8B-scale model evaluated against far larger, mature products, while its data mixture and evaluation remain early-stage.The paper expects the larger products to be stronger in general turn-based ability and plans broader, more systematic evaluation.
6 Conclusion
The paper advances genuine streaming interaction, where models decide when to act rather than waiting to be addressed. It releases an open vision-driven interaction model and deployable ecosystem aimed at making models present, real-time, and time-aware participants.
- The work shifts large models from turn-based behavior toward genuine streaming interaction, with models deciding for themselves when to act.This frames interactivity as an intrinsic model capability rather than behavior simulated by an external harness.
- The release includes time-aligned data, a training recipe, an 8B model, and a complete deployable system for building on open vision-driven interaction.The authors describe this as, to their knowledge, the first open vision-driven interaction model released with these components.
- Interactivity scales along three dimensions: freedom to speak when a moment warrants it, real-time interaction ability, and awareness of elapsed time.Together, these capabilities move a model from a queried tool toward a participant that remains present and watches what happens.
- Streaming scenarios show early deployment advantages across monitoring, live narration, step-by-step guidance, and companionship.The intended experience is unsolicited presence, such as noticing that someone is tired and offering support before they speak.
7 Appendix
The appendix illustrates JoyAI-VL-Interaction’s real-time behavior through monitoring alerts, multi-turn casual chat, and delegation of difficult visual questions to a background model.
- Background model delegation: For a difficult boat-hoist engineering question, the model acknowledges the need to check principles and delegates the explanation to a background model.The delegation request concerns four-point slings, load distribution, and preventing hull deformation in fiberglass vessels.