Source-linked AI summary
Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models
Christian Simon, Masato Ishii, Wei-Yao Wang, Koichi Saito, Akio Hayakawa, Dongseok Shim, Zhi Zhong, Shuyang Cui, Shusuke Takahashi, Takashi Shibuya, Yuki Mitsufuji
TL;DR
Long-form video-to-audio generation remains difficult because models trained on short clips must generalize across longer sequences despite limited long-duration data and transformer positional constraints. The paper introduces MMHNet, combining hierarchical multimodal processing with Non-Causal Mamba, and reports strong long-form benchmark performance across durations, including generation beyond 5 minutes.
Problem
Existing V2A models trained on fixed-length segments struggle to generalize to longer sequences because long-duration data are limited and positional encodings constrain extension.
Method
MMHNet combines multimodal V2A modeling, hierarchical token routing and dynamic chunking, and Non-Causal Mamba for long-form generation.
Results
MMHNet consistently delivers high-quality outputs across durations and outperforms prior methods on long-form V2A benchmarks.
Takeaways & Limitations
The experiments support training on short clips while generating contextually aligned audio for much longer videos, including durations beyond 5 minutes.
Abstract
from arXiv · showhide
Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations.
1. Introduction
The paper frames long-video-to-audio generation as a length-generalization problem: models trained on short, fixed-length segments must generate coherent audio for much longer videos. It introduces MMHNet and evaluates it on long-form benchmarks.
- Motivation: Public long audio-video datasets mostly contain videos of up to 1 minute, limiting evidence for longer-sequence generation.Models trained on fixed-length segments struggle when tested on longer LV2A inputs.
- Motivation: Generating up to 5-minute videos from short training clips is practical but can produce fragmented transitions, misaligned sound events, and degraded quality.These failures reflect limited modeling of long-form video context.
- Motivation: Transformer-based V2A models rely on explicit positional encodings that can hinder generalization to longer outputs.The paper motivates Mamba as an alternative that avoids explicit positional encodings.
- Method: MMHNet combines a multimodal V2A model with hierarchical processing to align visual, textual, and audio modalities across varying token lengths.The framework is designed to handle long video and audio without model adjustment during inference.
- Evaluation: The evaluation uses a long-form V2A benchmark built from UnAV100 and LongVale, with results reported as high quality across different durations.The paper presents this evaluation as evidence for long-form generation capability.
2. Related work
Related work covers visual and textual conditioning strategies, diffusion and autoregressive V2A models, and their limitations on long-form generation. The paper emphasizes degradation, error accumulation, and temporal coherence challenges as duration increases.
- Video-to-Audio generation: V2A methods either inject visual features into pretrained text-to-audio models or train video-to-audio models from scratch.Examples include T2AV and FoleyCrafter for visual-textual integration, and Diff-Foley and Frieren for video-to-audio modeling.
- Transformer limitations: Transformer-based V2A models use positional embeddings for positional awareness, but length mismatch between training and testing can degrade performance.The pilot study examines this issue in MMAudio trained on 8-second data and tested on longer sequences.
- Long-form generation: LoVA often degrades in audio quality and coherence beyond one minute.Its long-form performance is limited despite strengths on shorter content.
- Long-form generation: Autoregressive models can support long-form generation step by step but may accumulate errors, causing drift and reduced fidelity over extended sequences.This creates a different long-duration limitation from the degradation reported for LoVA.
3. Pilot Study
The pilot study examines why transformer-based V2A models struggle when tested on sequences longer than those used for training. It identifies positional embeddings and duration-dependent performance degradation as central issues.
- Observed limitations: Positional embeddings and attention-logit behavior are identified as transformer-based obstacles to length generalization in V2A.The paper states that addressing these obstacles may require substantial inference-time modifications.
- Observed limitations: Increasing duration causes a 3–4 point drop in distribution matching and multimodal alignment for the transformer-based V2A model.The study reports this degradation in the Figure 2 analysis.
4. Proposed Method
MMHNet models long-form multimodal-to-audio generation with flow matching, non-causal Mamba-2, and hierarchical routing. The design removes reliance on positional embeddings, enables bidirectional multimodal fusion, and selectively processes informative tokens.
- Core Network: Mamba-2 variants replace positional-embedding-dependent attention to improve modeling when sequence lengths change during long-audio generation.The method uses non-causal Mamba-2 because video conditions are available offline and multimodal fusion lacks a predefined ordering.
- Flow Matching: MMHNet trains a conditional flow-matching model to generate long-form audio from multimodal conditions in latent space.Inference integrates a learned conditional velocity field from noise to a generated sample, while training uses interpolated points between prior noise and data.
- Multimodal Architecture: The multimodal architecture combines multimodal and single-modality blocks, allowing deeper processing without applying all modalities at every layer.Global conditioning uses adaptive layer normalization, while token-level conditioning captures local audiovisual variation.
- Core Network: Non-causal Mamba-2 enables omnidirectional information flow so global hidden states can combine modalities without constrained scanning orders.The non-causal formulation also avoids cumulative transformation products associated with decay over long sequences.
- Hierarchical Framework: Hierarchical routing reduces redundant token processing by selecting temporally distinct audio or video tokens and highly similar cross-modal tokens.Temporal routing preserves tokens containing distinct temporal information, while multimodal routing forwards tokens with high similarity to the referenced modality; selected tokens are compressed and later upsampled.
5. Experiments
The experiments test whether models trained on short clips can generalize to substantially longer audio-video sequences. Evaluation uses long-form benchmarks and chunk-based metrics spanning distribution matching, quality, semantic consistency, and synchronization.
- Settings: Models are trained on fixed 8-second audio-video clips and then tested on longer sequences to evaluate length generalization.The small and large variants use different numbers of multimodal and single-modal blocks.
- Datasets: UnAV100 contains approximately 2K test videos lasting 10–60 seconds, while LongVale contains around 1K videos lasting 10–500 seconds.Training uses VGGSound 8-second data plus several text-to-audio datasets.
- Baselines: MMHNet is compared with LoVA and MMAudio augmented with duration-based positional-embedding scaling and NTK.The baselines are selected to assess performance in long-video-to-audio scenarios.
- Evaluation: Evaluation covers distribution matching, audio quality, semantic consistency, and temporal synchronization using multiple audio chunks matched to pretrained classifier durations.Chunk-based evaluation reduces errors caused by applying short-audio classifiers directly to long audio.
UnAV100
MMHNet is evaluated against prior methods across multimodal alignment, audio quality, temporal synchronization, and multiple duration settings on UnAV100 and LongVale. It shows stronger long-duration performance, with especially improved alignment and stability across durations.
- MMHNet significantly outperforms existing state-of-the-art methods across a broad spectrum of evaluation metrics.
- 3.9 IB-score points separate MMHNet from HunyuanVideo-Foley on UnAV100.
- 0.23 DeSync-score points separate MMHNet from the second-best method on LongVale.LongVale includes videos up to 7 minutes, testing audio-video alignment under substantially longer durations.
- Transformers vs. Mamba-2: Non-causal Mamba-2 processes long sequences and multimodal tokens more efficiently than causal Mamba-2 in the reported design comparison.
- Hierarchical vs. non-hierarchical methods: Compressed-space hierarchical processing yields better multimodal alignment than processing tokens in the original space.
- Token selection thresholds: A token-selection threshold of 0.5 produces the strongest results across all evaluated metrics.
6. Conclusions
MMHNet addresses length generalization in long-form video-to-audio generation by combining hierarchical modeling with Non-Causal Mamba-2. It trains on short clips while generating high-quality, contextually aligned audio for much longer videos.
- MMHNet combines hierarchical modeling with Non-Causal Mamba-2 for long-form video-to-audio generation.
- The framework targets length generalization by training on short clips and generating audio for much longer videos.
- Hierarchical token routing and dynamic chunking align video, text, and audio while reducing complexity.
- Non-causal modeling supports robust generalization to long sequences.