Source-linked AI summary
JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents
Junseok Ahn, Jaehyun Park, Jeongsoo Choi, Joon Son Chung, Ji-Hoon Kim
TL;DR
Conversational systems have largely focused on text or speech, leaving out visual feedback that contributes to natural interaction. JAVIS jointly streams speech and facial motion from a shared representation and adds a synthetic dialogue corpus; it reports substantially lower initial latency and strong synchronization and expression results.
Problem
Conversational systems have mainly focused on text or speech, while human communication also conveys intent through visual signals and cascaded systems can lose prosodic nuance or dialogue context.
Source: §1 · p. 1Method
JAVIS uses a Joint Refined Head to form a unified audio-visual representation that drives parallel streaming branches for speech and facial motion, alongside the 334-hour JAVIS-Syn corpus.
Source: §2.1 · p. 2, §2.2 · p. 2, §2.3 · p. 3Results
JAVIS reduces initial latency by approximately 200× over a strong cascaded baseline and achieves the highest reported synchronization and facial-expression appropriateness scores.
Source: §3.2 · p. 3, §4 · p. 4, §3.3 · p. 4Takeaways & Limitations
JAVIS combines real-time audio-visual streaming with strong synchronization and facial-expression appropriateness while preserving speech quality.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Human communication is inherently multimodal, conveying intent not only through linguistic and vocal cues but also through visual signals. However, existing conversational systems have mainly focused on text or speech, leaving out the visual feedback that is essential for natural interaction. In this paper, we propose JAVIS, an end-to-end streaming framework for real-time audio-visual conversational agents. Our key idea is to let the voice and the face speak from a single representation. On top of a shared LLM, a Joint Refined Head produces a unified audio-visual representation, which is fed into dedicated streaming branches for speech and face generation. This end-to-end design avoids intermediate bottlenecks, enabling real-time generation while keeping speech and face tightly synchronized. In addition, we present JAVIS-Syn, a 334-hour synthetic corpus designed to address the lack of expressive audio-visual dialogue data, covering emotionally rich multi-turn conversations and assistant-style exchanges. Experiments show that JAVIS achieves approximately 200$\times$ lower initial latency than a strong baseline, while outperforming baselines in lip synchronization and facial expression appropriateness.
1. INTRODUCTION
JAVIS addresses the absence of visual feedback in conversational systems with a real-time framework that jointly generates speech and facial video. Its shared representation is designed to avoid the bottlenecks of cascaded systems, while JAVIS-Syn supplies expressive dialogue data.
1. INTRODUCTION
- Existing conversational AI has largely remained text- or speech-based, despite human communication also conveying intent through facial expressions, head movements, and gaze.The paper identifies the absence of visual feedback as limiting system expressiveness and naturalness.Source: §1 · p. 1
- Cascaded systems use intermediate text or synthesized speech that can discard prosodic nuance or dialogue context needed by downstream face generation.Their sequential dependencies also accumulate latency and propagate errors that degrade audio-visual synchronization.Source: §1 · p. 1
- Prior end-to-end approaches still rely on iterative spatiotemporal video-latent generation, while expressive audio-visual dialogue data remains scarce.The paper identifies real-time streaming of realistic visual responses as an open problem.Source: §1 · p. 1
- JAVIS jointly streams synchronized speech and facial video from a unified audio-visual representation, reducing initial latency by approximately 200× versus a strong baseline.Human evaluation also finds natural responses with accurate lip synchronization and situation-appropriate facial expressions.Source: §1 · p. 1
2. METHOD
JAVIS uses a shared audio-visual representation to drive parallel speech and facial-motion generation. The method also introduces JAVIS-Syn, a synthetic corpus of expressive multi-turn and assistant-style dialogue.
2. METHOD
- Given a spoken query, dialogue history, and reference portrait, JAVIS streams a spoken response and synchronized facial video of the reference identity.Parallel speech and motion branches operate on the Joint Refined Head representation, with generated motion rendered into video using the portrait.Source: §2 · p. 2
2.1. Joint Refined Head
- The Joint Refined Head repurposes a speech refined head to produce five hidden states per chunk as the shared representation for speech-token and facial-motion generation.The shared LLM is frozen while the LoRA-adapted head and Motion Decoder are jointly trained with speech-token and motion losses, encouraging synchronized facial dynamics in the representation.Source: §2.1 · p. 2
2.2. Streaming Audio-Visual Generation
- Five-frame motion chunks span 0.2 seconds and align with speech at 25 Hz, using one-chunk lookahead, prior motion, and a reference facial state.A bidirectional Transformer is trained with conditional flow matching; at inference it samples motion after Ht+1 is available, and its fixed input window keeps per-chunk cost independent of response length.Source: §2.2 · p. 2
2.3. JAVIS-Syn
- JAVIS-Syn addresses gaps in existing corpora with 334 hours of synthetic dialogue featuring expressive facial motion and assistant-style interactions.It draws on emotion-labeled multi-turn dialogue and assistant question–answer sources, rendering utterance videos with emotion prompts and filtering them by lip-sync quality.Source: §2.3 · pp. 2–3
3. EXPERIMENTS
JAVIS is evaluated against cascaded and end-to-end baselines on objective audio-visual, speech, and latency metrics, plus human ratings and component ablations. It leads on synchronization and startup speed, while the ablations show the importance of lookahead, JAVIS-Syn, and dialogue adaptation.
3.1. Experimental Setup
- Evaluation uses 140 VStyle samples unseen during training, with video, lip-sync, speech, and generation-efficiency metrics; training adapts from monologue to dialogue.The second stage uses up to three preceding turns as context and applies loss only to the final assistant response; only the JRH LoRA adapter and Motion Decoder are trained.Source: §3.1 · p. 3
3.2. Objective Evaluation
- LSE-C 9.59 and LSE-D 6.74 give JAVIS the best audio-visual synchronization among the compared systems.Its video quality is comparable to dedicated face-generation models, while UTMOS and SpkSIM improve over the original FAC.Source: §3.2 · pp. 3–4
- 1.27 s initial latency and 0.541 RTF make JAVIS the fastest system, with approximately 200× lower initial latency than FAC + InfiniteTalk while matching its lip synchronization.It also starts 5.3× faster than FAC + FLOAT and has 39% lower initial latency than FLOAT’s streaming configuration.Source: §3.2 · p. 4, Table 3 · p. 4
3.3. Subjective Evaluation
- 3.76 is JAVIS’s best facial expression appropriateness score, while its 4.37 synchronization score is highest and its 3.92 motion naturalness is comparable to the best 4.02.The scores are human ratings reported with 95% confidence intervals.Source: §3.3 · p. 4, Table 4 · p. 4
3.4. Ablation Study
- Removing Ht+1 lowers LSE-C from 9.59 to 8.80 and raises LSE-D from 6.74 to 7.55; removing JAVIS-Syn also degrades lip synchronization and speech quality.Monologue-only training degrades all metrics, including WER 13.91%, LSE-C 6.17, and LSE-D 8.79.Source: §3.4 · p. 4, Table 5 · p. 4
4. CONCLUSION
The conclusion presents JAVIS as a real-time streaming framework that jointly generates speech and facial motion from one representation, alongside a 334-hour synthetic dialogue corpus. The authors report strong synchronization and expression appropriateness with substantially lower initial latency, and identify full-duplex interaction as future work.
4. CONCLUSION
- JAVIS jointly streams speech and facial motion from a single representation, sharing dialogue context across both modalities.JAVIS-Syn is a 334-hour synthetic corpus of emotionally rich multi-turn conversations and assistant-style interactions.Source: §4 · p. 4
- Approximately 200× lower initial latency than a strong cascaded baseline accompanies strong synchronization and facial expression appropriateness, while speech quality is preserved.Source: §4 · p. 4
- Full-duplex interaction with listening behavior is identified as future work.Source: §4 · p. 4