OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

1 Introduction

• OmniForcing streams at ∼25 FPS with ∼0.7s TTFC, versus LTX-2’s ∼197s offline generation, while maintaining visual and acoustic fidelity on par with the teacher.The framework achieves this streaming generation rate on a single GPU.Source: Figure 1 · p. 2, §1 · p. 3
• Cascaded pipelines sever the joint distribution and delay secondary audio, while adapting video-only causal models to dual streams risks instability from temporal asymmetry.Bidirectional full-sequence attention also raises Time-To-First-Chunk latency, limiting interactive and streaming deployment.Source: §1 · p. 2
• OmniForcing dynamically interleaves audio and video chunks to enable low-latency streaming without sacrificing the holistic multi-modal distribution learned by the bidirectional teacher.It is presented as the first framework to distill a bidirectional audio-visual foundation model into a high-fidelity streaming autoregressive generator.Source: §1 · pp. 2–3
• Asymmetric Block-Causal Alignment uses a zero-truncation Global Prefix to anchor modalities at exact one-second boundaries, aligning 3 video frames with 25 audio frames.The alignment exploits the native stride characteristics of the VAEs to prevent multi-modal synchronization drift.Source: §1 · p. 3
• An unsupervised Audio Attention Sink with Identity RoPE forms position-agnostic global memory, mitigating gradient explosions from sparse causal audio attention and Softmax collapse from token-density mismatch.The sink mechanism is inspired by attention sink phenomena in language and vision models.Source: §1 · p. 3
• Joint Self-Forcing Distillation lets the model self-correct cross-modal errors accumulated during long rollouts, addressing exposure bias.The strategy uses joint self-forcing to combat exposure bias amplified by cross-modal error accumulation.Source: §1 · p. 3
• A Modality-Independent Rolling KV-Cache enables concurrent execution of the audio and video streams on one GPU while reducing per-step context complexity to O(L).The design exploits intra-layer decoupling between the 14B video and 5B audio streams.Source: §1 · p. 3

Joint Audio-Visual and Video Foundation Models

• LTX-2 [7] and Veo 3 [44] generate synchronized, high-fidelity audio and video, but bidirectional full-sequence attention causes quadratic scaling and high Time-To-First-Chunk latency.LTX-2 uses asymmetric 14B video and 5B audio streams coupled through bidirectional cross-attention.Source: §2 · pp. 3–4

Audio-Visual Synthesis and Alignment

• Earlier pipelines generated video before Foley audio or used standalone audio models, while audio-driven video struggled with synchronization and streaming.These decoupled approaches sever joint temporal distributions and hinder fine-grained synchronization and cross-modal temporal reasoning.Source: §2 · p. 4

Diffusion Distillation for Efficiency

• Diffusion distillation methods compress multi-step sampling: DMD minimizes approximate KL divergence, Consistency Models [27] enforce ODE-trajectory self-consistency, and Adversarial Diffusion Distillation uses discriminator-based losses.The passage identifies these methods as a computational foundation for real-time generation.Source: §2 · p. 4

Autoregressive & Streaming Diffusion Models

• StreamingT2V and Pyramid Flow introduced autoregressive diffusion but retained multi-step overhead; CausVid reached ∼9.4 FPS, while Self-Forcing, Causal-Forcing [53], and Rolling-Forcing addressed further streaming bottlenecks.CausVid distilled a bidirectional video teacher into a causal student using asymmetric DMD; Self-Forcing trained on the model's own KV-cache predictions to address exposure bias.Source: §2 · pp. 4–5

Research gap

• Real-time joint audio-visual streaming remains unresolved: causal distillation in dual-stream models can cause token sparsity, conditional shifts, Softmax collapse, and gradient explosions.The passage attributes the instability to audio-video frequency asymmetry, such as 25 FPS versus 3 FPS; OmniForcing is proposed to address this gap.Source: §2 · p. 5

3.1 Problem Formulation and The OmniForcing Pipeline

• OmniForcing factorizes joint video-audio generation into synchronized autoregressive blocks, then distills a pretrained bidirectional dual-stream transformer for streaming generation.The model represents each modality in its own VAE latent space; its pipeline addresses frequency asymmetry, causal-mask instability, and cross-modal error accumulation.Source: §3.1 · p. 5

3.2 Asymmetric Block-Causal Alignment and Mask Design

• One-second physical-time macro-blocks align 3 video latents with 25 audio latents without fractional remainders; a bidirectional global prefix remains visible to future tokens as a cross-modal semantic anchor.Source: §3.2 · pp. 6–7

3.3 Bridging the Gap: Causal Regression and Architectural Stabilizers

• The three-stage pipeline first distills few-step bidirectional denoising, then regresses the student to causal ODE trajectories before joint self-forcing training.Stage I uses Distribution Matching Distillation; Stage II adapts the weights to block-causal masks, whose direct application otherwise causes training collapse from conditional distribution shift.Source: §3.3 · pp. 8–9
• Learnable audio sink tokens in the global prefix expand early tokens’ attention context to prevent Softmax collapse and gradient explosions, with Identity RoPE keeping the sinks position-agnostic.The sinks restore attention entropy and buffer anomalous logit perturbations; the identity constraint removes positional rotational interference.Source: §3.3 · pp. 9–10

3.4 Joint Self-Forcing Distillation and Asymmetric Parallel Inference

• Joint self-forcing unrolls generated blocks against a frozen bidirectional teacher to mitigate exposure bias and cross-modal prediction drift; decoupled streams and a rolling KV cache enable synchronized generation at ∼25 FPS on one GPU.The modality-independent cache reduces per-step context complexity to O(L), while independent modality FFNs support asymmetric tensor parallelism across devices.Source: §3.4 · p. 10

4.1 Experimental Setup

• The evaluation uses JavisBench’s four dimensions plus runtime, TTFC, and FPS, and compares cascaded pipelines, joint models, and bidirectional LTX-2.OmniForcing is built on LTX-2 [7] with 14B video and 5B audio streams; training uses 32 GPUs and three distillation stages, while captions are rewritten for audio-visual coherence.Source: §4 · p. 11, §4.1 · p. 11

4.2 Main Results

• OmniForcing completes a 5-second 480p audio-visual clip in ∼5.7 s, with ∼0.7 s TTFC and ∼25 FPS sustained throughput—a ∼35× speedup over offline LTX-2 teacher (197 s), and the only compared method enabling true streaming generation.Source: §4.2 · p. 11
• On JavisBench, OmniForcing scores FVD 137.2 and FAD 5.7, while the LTX-2 teacher scores FVD 125.4 and FAD 4.6; OmniForcing’s CLIP score of 0.322 is best overall.Its AV-IB 0.269, AVHScore 0.254, and DeSync 0.392 are second overall or closely track the teacher’s DeSync 0.384; the nearest non-teacher AV-IB is 0.198, and JavisDiT++ DeSync is 0.832.Source: §4.2 · pp. 12–13, Table 1 · p. 13
• On same-prompt VBench, OmniForcing exceeds the teacher in aesthetic quality (+0.026), imaging quality (+0.020), and subject consistency (+0.010), while preserving temporal coherence.Qualitative examples show layered voice and ambient sounds, sustained speech, synchronized transient events, and blended narration with sewing-machine sounds.Source: Figure 4 · p. 12, §4.2 · p. 13, Table 2 · p. 14

4.3 Ablation Studies

• Stage II stabilizes with sink pools S ≥4, whereas S ≤2 triggers NaN gradients; at S = 16, Identity RoPE lowers loss to 0.081 from 0.402 with standard positions.QK-Norm loss is 0.232; Tanh-Gated Attention loss plateaus at 1.258 with block artifacts, while its tanh gate suppresses attention patterns despite no NaNs.Source: §4.3 · pp. 13–14, Table 3 · p. 14

• OmniForcing is the first framework to distill a bidirectional joint audio-visual diffusion model into a real-time streaming autoregressive generator.Source: §5 · p. 14
• Asymmetric Block-Causal Alignment, Audio Sink Tokens with Identity RoPE, and Joint Self-Forcing Distillation address temporal asymmetry and gradient instability in causal multi-modal distillation.Source: §5 · pp. 14–15
• ∼25 FPS streaming on a single GPU is achieved with a modality-independent rolling KV-cache.Source: §5 · p. 15