Source-linked AI summary
Encore: Infinite Audio-Video Generation with Adaptive Signal Routing
Shaohua Pan, Junbao Chen, Shengyi He, Jingfeng Xue, Wen Tao, Haocheng Feng, Siming Fan, Dongwei Pan, Yi Yang, Wei He, Hang Zhou
TL;DR
Joint audio-video generation is generally short-form, while long-video generation lacks audio, leaving long synchronized clips with speech and coherent visuals unresolved. Encore combines iterative chunk generation, reference audio-video signals, and adaptive routing of conditioning; it supports long joint and modality-conditioned synthesis, and reports better quality and temporal coherence than existing methods.
Problem
Joint audio-video methods are typically limited to short clips, while long-video methods lack audio, leaving long-form generation to preserve video coherence, audio coherence, and cross-modal synchronization together.
Source: §1 · p. 2Method
Encore generates iteratively with propagated context for local continuity, reference audio-video signals with shifted position embeddings for global consistency, and Adaptive Signal Routing to balance conditioning streams.
Source: §1 · p. 2Results
Encore significantly outperforms existing audio-video generation approaches in quality and temporal coherence; its drift metrics (ΔAS / ΔID) are 0.02 / 0.07.
Source: §1 · p. 3, §4.2 · p. 7Takeaways & Limitations
A single end-to-end-trained model supports long joint audio-video generation and audio-to-video and video-to-audio synthesis at inference.
Source: §1 · p. 2, §5 · p. 8A fixed reference frame makes complex shot transitions involving significant scene or subject changes difficult.
Source: §4.6 · p. 8
Abstract
from arXiv · showhide
Existing audio-video generation methods produce well-synchronized clips but are limited to short durations, while long-video generation methods extend duration through chunk-based iterative synthesis yet lack audio entirely. Generating long audio-video jointly is fundamentally harder than either task alone: each chunk must simultaneously maintain video temporal coherence, audio temporal coherence, and cross-modal synchronization, whose conditioning signals enter the model through different pathways. In this work, we present Encore for long-form synchronized audio-video generation. Our key insight is to factor this challenge into: (1) local continuity handled by iterative generation with explicit cross-chunk context propagation, and (2) global consistency enforced via reference audio-video signals with shifted position embedding. Building on this design, we propose Adaptive Signal Routing (ASR), which introduces learnable attention biases within self-attention and learnable residual scales on cross-attention outputs, enabling the model to adaptively modulate the influence of each conditioning signal. Trained end-to-end for joint audio-video generation, Encore also supports infinite-length audio-to-video and video-to-audio synthesis at inference by conditioning on the ground-truth modality throughout the denoising process. Experiments on our extended VerseBench for long audio-video evaluation demonstrate that Encore significantly outperforms existing methods in both generation quality and temporal coherence. Code and data for this paper are at https://github.com/shaohua-pan/Encore.
1 Introduction
Encore addresses long-form synchronized audio-video generation by combining chunk-wise continuity with global reference anchoring. Adaptive Signal Routing (ASR) adjusts how the model uses heterogeneous conditioning signals, and the paper reports gains in generation quality and temporal coherence.
1 Introduction
- Prior joint audio-video methods produce synchronized but typically few-second clips, while long-video methods generate iteratively without audio; R-FLAV [Ergasti et al. 2025] lacks speech and high-resolution video.Long-form generation must preserve video coherence, audio coherence, and cross-modal synchronization over time.Source: §1 · p. 2
- Chunk-wise generation must propagate enough local context to avoid discontinuities and error accumulation, while balancing continuation, reference, text, and synchronization signals whose importance varies across layers and heads.This motivates adaptively routing conditioning signals rather than treating them uniformly.Source: §1 · p. 2
- Encore divides long-form coherence into local continuity via iterative chunks with propagated context and global consistency via reference audio-video signals with shifted position embeddings.Video chunks use overlapping motion frames and a clean reference frame; audio chunks use preceding audio and a reference clip. The model is trained end-to-end, and at inference it also supports audio-to-video and video-to-audio generation using the ground-truth conditioning modality.Source: §1 · p. 2
- ASR learns per-layer, per-head attention log-biases for anchor and continuation tokens, plus scales on text and synchronization cross-attention outputs.These parameters balance global identity, local continuity, semantic guidance, and cross-modal alignment.Source: §1 · pp. 2–3
- Encore significantly outperforms existing audio-video generation approaches in both generation quality and temporal coherence.The paper presents this comparison as a contribution based on its extended VerseBench.Source: §1 · p. 3
2 Related Work
Related work develops long-video generation and joint audio-video synthesis largely as separate lines. Existing joint systems are usually short-form, while the cited long-form approach is low-resolution and does not model speech.
2.1 Long Video Generation
- Long-video methods progress from hierarchical designs to conditioning and training strategies for stability, including Diffusion Forcing, Self-Forcing, Causal-Forcing, Rolling-Forcing, FramePack, and SVI.These approaches use per-token noise levels, rolling KV caches and student rollouts, distillation, attention sinks, context compression, or error recycling.Source: §2.1 · p. 3
2.2 Joint Audio-Video Generation
- Joint audio-video work spans coupled denoising and stitched experts to unified transformers; OVI, LTX-2, and Harmony explore cross-modal fusion or alignment, while R-FLAV [Ergasti et al. 2025] extends clips but is shown at 64×64 without speech.The paper contrasts this prior work with Encore’s arbitrarily long synchronized generation with speech and high-resolution video.Source: §2.2 · p. 3
3 Method
Encore extends an LTX-2.3 chunk generator for long audio-video synthesis through reference and continuation context, with Adaptive Signal Routing to balance conditioning signals. Its shared model also supports long-duration audio-to-video and video-to-audio inference without mode-specific fine-tuning.
3.1 Preliminary: Latent Flow Matching
- Encore trains a diffusion transformer to predict audio and video velocity targets with a masked flow-matching loss applied only to generated tokens.Clean conditioning tokens are excluded from the loss.Source: §3.1 · p. 3
3.2 Long Audio-Video Training and Inference
- Training separates global reference anchors, preceding-chunk context, and noisy targets; negative RoPE positions distinguish references from temporal predecessors.Reference and continuation tokens remain clean during training, while target tokens are noised.Source: §3.2.1 · pp. 4–5
- Video continuation receives sampled historical prediction errors during training, while target-token loss excludes clean context; audio errors are not recycled.The authors report no noticeable audio error accumulation across chunks.Source: §3.2.1 · p. 5
- At inference, each chunk uses shared references and the preceding chunk’s tail; ground-truth modality conditioning enables long A2V and V2A generation with shared weights.Long descriptions are split into chunk-level captions, with speech transcripts segmented to match natural speaking rates.Source: §3.2.2 · p. 5
3.3 Adaptive Signal Routing
- ASR learns per-layer, per-head biases for anchor and continuation self-attention and scales text and synchronization cross-attention outputs to balance conditioning signals.Output scaling controls cross-attention contributions, while attention-logit biases reweight anchor and continuation token groups.Source: §3.3 · pp. 5–6
4 Experiments
Encore is evaluated on an extended 30-second audio-video benchmark and shows stronger temporal stability, synchronization, and long-duration consistency in reported comparisons. Ablations, a user study, and additional generation examples assess its components and scope; the discussion identifies practical and responsible-use limitations.
4.1 Implementation Details
- The benchmark uses six structured semantic components in 30-second samples and evaluates video, audio, synchronization, and temporal drift.Training used about 270,000 five-second clips; the start-end drift metric compares the first and last 15% of frames, with larger drift indicating more severe temporal degradation.Source: §4.1 · pp. 6–7
4.2 Comparisons
- Encore cuts ΔAS / ΔID drift to 0.02 / 0.07 versus SVI’s 0.06 / 0.22, while reaching LSE-C 2.36 and AV-A 0.88 versus LTX-2.3’s 1.40 and 0.76.Encore ranks best in motion smoothness, lip synchronization, and audio-video alignment, and second on other metrics; across eight 13-minute videos, ID ranges from 0.918 to 0.938, AS from 0.474 to 0.481, PQ from 6.853 to 6.938, and AV-A from 0.793 to 0.928.Source: §4.2 · p. 7
4.3 Evaluations
- Removing error recycling lowers ID to 0.77 from 0.86 and reduces audio quality, while removing ASR causes broad metric degradation.Ablations separately associate anchor routing with identity and motion, context routing with smoothness, semantic routing with audio quality and synchronization, and synchronization routing with alignment while leaving unimodal quality largely intact.Source: §4.3 · p. 8
4.4 User Study
- In a blind study with 10 participants, Encore was preferred over OVI in all comparisons and over LTX-2.3 in 80% of audio-quality and 100% of video-quality and synchronization comparisons.For video quality, Encore was preferred over SVI in 80% and Helios in 100% of comparisons.Source: §4.4 · p. 8
4.5 More Results
- Supplementary examples show generation up to 500 seconds, as well as long audio-to-video and video-to-audio synthesis.The examples are reported to maintain identity, temporal coherence, and cross-modal alignment over extended durations.Source: §4.5 · p. 8
4.6 Discussion
- A fixed reference frame makes complex scene or subject-changing shot transitions difficult, inference is not real-time, and reported metrics do not assess safety, fairness, or copyright compliance.The authors identify dynamic reference switching and distillation-based streaming as potential future directions.Source: §4.6 · p. 8
5 Conclusion
Encore targets synchronized audio-video generation of unlimited duration by separating local continuity from global consistency. It combines iterative chunk generation, reference anchors, and Adaptive Signal Routing, and supports joint and cross-modal generation at inference.
5 Conclusion
- Encore decomposes long-form coherence into local continuity through chunk generation with continuation context and global consistency through reference audio-video anchors with shifted position embeddings.Adaptive Signal Routing dynamically balances anchor, continuation, semantic, and synchronization conditioning streams.Source: §5 · p. 8
- A single model trained end-to-end on short clips supports long joint audio-video, audio-to-video, and video-to-audio generation at inference.Source: §5 · p. 8
- Experiments on extended VerseBench show that Encore significantly outperforms existing methods in identity preservation, audio-video synchronization, and temporal stability over long durations.Source: §5 · p. 8