Source-linked AI summary
OmniForcing: Unleashing Real-time Joint Audio-Visual Generation
Yaofeng Su, Yuming Li, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian, Haoyang Huang, Nan Duan
TL;DR
Joint audio-visual diffusion models offer strong generation quality but their bidirectional attention creates latency that limits real-time use. OmniForcing distills such a model into a streaming autoregressive generator using asymmetric causal alignment, stabilization, self-forcing, and rolling-cache inference. It achieves ∼25 FPS streaming on a single GPU while maintaining multimodal synchronization and visual quality on par with the bidirectional teacher.
Problem
Bidirectional attention in joint audio-visual diffusion models causes high latency, while naive causal distillation of dual-stream architectures is unstable under extreme audio-video temporal asymmetry.
Method
OmniForcing combines Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix, Audio Sink Tokens with Identity RoPE, Joint Self-Forcing Distillation, and modality-independent rolling KV-cache inference.
Results
∼25 FPS streaming on a single GPU, with multimodal synchronization and visual quality on par with the bidirectional teacher.
Takeaways & Limitations
OmniForcing provides a real-time streaming autoregressive alternative to offline bidirectional joint audio-visual generation while preserving the teacher’s multimodal fidelity.
Abstract
from arXiv · showhide
Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at $\sim$25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.\textbf{Project Page:} \href{https://omniforcing.com}{https://omniforcing.com}
1 Introduction
OmniForcing distills a bidirectional joint audio-visual model into a streaming autoregressive generator that preserves multimodal fidelity while addressing latency and causal-distillation instability. Its alignment, stabilization, and inference mechanisms support real-time generation.
- Bidirectional full-sequence attention imposes high Time-To-First-Chunk latency, limiting joint audio-visual models in interactive and streaming applications.
- OmniForcing is the first framework presented to distill a heavy bidirectional audio-visual foundation model into a high-fidelity streaming autoregressive generator.
- The framework dynamically interleaves audio and video chunks, enabling ultra-low-latency streaming without sacrificing the teacher’s holistic multimodal distribution.
- Asymmetric Block-Causal Alignment uses a zero-truncation Global Prefix to anchor synchronized one-second blocks at exact boundaries of 3 video frames and 25 audio frames.
- Audio Sink Tokens with Identity RoPE address Softmax collapse caused by multimodal token-density mismatch, while Joint Self-Forcing and rolling KV-cache inference support efficient streaming.
- OmniForcing achieves state-of-the-art streaming generation at ∼25 FPS on a single GPU.
2 Related Work
Joint audio-visual foundation models improve multimodal generation quality but remain difficult to deploy in real time because bidirectional processing is computationally expensive. Existing streaming approaches are either decoupled or unimodal, while causal distillation of dual-stream models introduces severe instability.
- LTX-2 uses an asymmetric dual-stream architecture with 14B video and 5B audio streams coupled through bidirectional cross-attention.
- Bidirectional full-sequence attention makes complexity scale quadratically with sequence length, producing massive TTFC latency that impedes real-time and streaming applications.
- Causal distillation remains unresolved for joint audio-visual generation because 3 FPS video versus 25 FPS audio creates token sparsity, distribution shifts, Softmax collapse, and gradient explosions.
3 OmniForcing
OmniForcing reformulates joint audio-visual generation as synchronized block-causal autoregression and distills a bidirectional teacher through staged denoising, causal adaptation, and self-forcing. Its alignment and masking design addresses the modalities’ unequal token densities while preserving causal information flow.
- 3.1 Problem Formulation and The OmniForcing Pipeline: OmniForcing factorizes joint video-audio generation over K+1 synchronized physical-time blocks conditioned on the text prompt.The blocks are generated autoregressively, with each block conditioned on preceding blocks.
- 3.2 Asymmetric Block-Causal Alignment and Mask Design: A one-second macro-block aligns 3 video latents with 25 audio latents, avoiding fractional remainders caused by the 25:3 temporal-rate ratio.Video and audio therefore advance together at physical-second boundaries despite their unequal latent frequencies.
- 3.2 Asymmetric Block-Causal Alignment and Mask Design: The zero-truncation Global Prefix places the initial video and audio latents in bidirectional block B0, keeping them globally visible as a cross-modal semantic anchor.The prefix follows the VAE stride structure and cannot be squeezed into later one-second blocks.
- 3.2 Asymmetric Block-Causal Alignment and Mask Design: The four-way asymmetric mask permits bidirectional attention within blocks, strictly causal attention across blocks, and global visibility for prefix tokens.The block-index inequalities ensure that future keys remain masked while temporal receptive fields expand synchronously.
- 3.3 Bridging the Gap: Causal Regression and Architectural Stabilizers: The three-stage pipeline first distills few-step bidirectional denoising, then adapts weights through causal ODE regression, and finally uses autoregressive joint Self-Forcing.The final stage addresses exposure bias by training on autoregressively unrolled generation rather than only ground-truth history.
- 3.3 Bridging the Gap: Causal Regression and Architectural Stabilizers: Direct causal adaptation is unstable because sparse audio histories cause Softmax saturation and gradient explosions under the bidirectional-to-causal distribution shift.The instability is especially severe for audio, whose continuous tokens are exposed to extremely short visible histories.
4 Experiments
OmniForcing is evaluated for visual, audio, cross-modal, and streaming performance, alongside stabilization ablations. It delivers fast true streaming while largely preserving the bidirectional teacher’s quality and synchronization.
- Inference Efficiency: ∼25 FPS throughput and ∼0.7 s TTFC enable true streaming, completing a 5-second 480p clip in ∼5.7 s.This represents a ∼35× speedup over the offline LTX-2 teacher’s 197 s runtime.
- Audio-Visual Quality: OmniForcing attains strong joint audio-visual quality, with FVD 137.2, FAD 5.7, best CLIP 0.322, and competitive cross-modal scores.It surpasses all baselines except the teacher on FVD and ranks second on AV-IB and AVHScore.
- Temporal Synchronization: DeSync 0.392 closely tracks the teacher’s 0.384 and substantially outperforms JavisDiT++ at 0.832.The authors attribute the remaining consistency and synchrony gaps to the restricted causal receptive field required for streaming.
- Distillation Fidelity: Under controlled same-prompt VBench evaluation, OmniForcing slightly exceeds the teacher on aesthetic quality, imaging quality, and subject consistency.The reported gains are +0.026, +0.020, and +0.010, respectively.
- Ablation Studies: Audio Sink Tokens with Identity RoPE stabilize Stage II training, while small sink pools trigger NaN gradients from Softmax collapse.Replacing Identity RoPE increases loss from 0.081 to 0.402 at S = 16; QK-Norm and Tanh-Gated Attention also produce inferior stability–quality trade-offs.
5 Conclusion
OmniForcing distills a bidirectional joint audio-visual diffusion model into a real-time streaming autoregressive generator using alignment, stabilization, self-forcing, and rolling-cache components. It achieves ∼25 FPS streaming on a single GPU for interactive and latency-sensitive scenarios.
- Conclusion: OmniForcing distills a bidirectional joint audio-visual diffusion model into a real-time streaming autoregressive generator.The framework addresses temporal asymmetry and gradient instability through Asymmetric Block-Causal Alignment and Audio Sink Tokens with Identity RoPE.
- Conclusion: ∼25 FPS streaming on a single GPU supports deployment of multi-modal foundation models in interactive and latency-sensitive scenarios.The system combines Joint Self-Forcing Distillation with a modality-independent rolling KV-cache.