PaperlayerView exact revision on arXiv

Source-linked AI summary

SoundReactor: Frame-level Online Video-to-Audio Generation

Koichi Saito, Julian Tanke, Christian Simon, Masato Ishii, Kazuki Shimada, Zachary Novack, Zhi Zhong, Akio Hayakawa, Takashi Shibuya, Yuki Mitsufuji

arXiv:2510.02110v1cs.SDcs.LGeess.AS

TL;DR

Offline V2A models cannot directly support frame-level interactive applications because they assume future video frames are available. SoundReactor introduces causal online V2A generation with continuous audio latents, visual frame conditioning, and accelerated diffusion decoding, achieving synchronized stereo audio and low per-frame latency on gameplay videos.

  • Problem

    Existing V2A generation is predominantly offline, limiting frame-level online applications such as live content creation and generative world models.

  • Method

    SoundReactor uses a causal decoder-only multimodal Transformer over continuous audio latents, causal DINOv2 visual conditioning, and diffusion followed by consistency tuning.

  • Results

    SoundReactor generates high-quality, semantically and temporally synchronized full-band stereo audio on AAA gameplay videos, with 26.3ms latency at NFE=1 and 31.5ms at NFE=4.

  • Takeaways & Limitations

    The framework provides a foundational component for interactive multimodal applications including live content creation and sounding world models.

  • Takeaways & Limitations

    The task assumes frame-aligned audio and video tokens, with each audio token paired to the corresponding video frame.

Abstract

from arXiv · show

Prevailing Video-to-Audio (V2A) generation models operate offline, assuming an entire video sequence or chunks of frames are available beforehand. This critically limits their use in interactive applications such as live content creation and emerging generative world models. To address this gap, we introduce the novel task of frame-level online V2A generation, where a model autoregressively generates audio from video without access to future video frames. Furthermore, we propose SoundReactor, which, to the best of our knowledge, is the first simple yet effective framework explicitly tailored for this task. Our design enforces end-to-end causality and targets low per-frame latency with audio-visual synchronization. Our model's backbone is a decoder-only causal transformer over continuous audio latents. For vision conditioning, it leverages grid (patch) features extracted from the smallest variant of the DINOv2 vision encoder, which are aggregated into a single token per frame to maintain end-to-end causality and efficiency. The model is trained through a diffusion pre-training followed by consistency fine-tuning to accelerate the diffusion head decoding. On a benchmark of diverse gameplay videos from AAA titles, our model successfully generates semantically and temporally aligned, high-quality full-band stereo audio, validated by both objective and human evaluations. Furthermore, our model achieves low per-frame waveform-level latency (26.3ms with the head NFE=1, 31.5ms with NFE=4) on 30FPS, 480p videos using a single H100. Demo samples are available at https://koichi-saito-sony.github.io/soundreactor/.

1 INTRODUCTION

SoundReactor addresses frame-level online V2A generation, where audio must be generated autoregressively without future video frames, targeting causal, low-latency, synchronized stereo audio.

  • 1 INTRODUCTION: Frame-level online V2A generation produces audio from video streams without accessing future frames, unlike offline V2A systems that assume complete sequences or frame chunks.The setting is motivated by live content creation and interactive generative worlds.
  • 1 INTRODUCTION: SoundReactor combines an image encoder, waveform encoder, and causal decoder-only multimodal Transformer with a diffusion head.Its visual conditioning uses DINOv2 grid features aggregated into a single token per frame to preserve causality and efficiency.
  • 1 INTRODUCTION: 26.3ms with head NFE=1 and 31.5ms with NFE=4 are achieved for waveform-level per-frame latency on 480p, 30FPS videos using one H100 GPU.The model generates semantically and temporally synchronized, high-quality full-band stereo audio, validated by objective and human evaluations.
  • 1 INTRODUCTION: The paper introduces online V2A as a new paradigm and presents SoundReactor as a framework designed for end-to-end causality, low latency, and audio-visual alignment.The stated application scope includes live content creation and sounding world models.

2 PRELIMINARIES

The preliminaries describe diffusion-based continuous-latent generation, autoregressive conditioning without discrete tokenization, and consistency methods for reducing diffusion sampling cost.

  • 2.1 DIFFUSION MODELS: Diffusion models generate clean samples from a Gaussian prior through iterative denoising, with denoising score matching used for training.The denoiser estimates clean data from noisy samples and their noise levels.
  • 2.2 AUTOREGRESSIVE MODELING WITHOUT DISCRETE TOKENIZATION: MAR predicts continuous output vectors autoregressively and samples each vector conditionally through an iterative diffusion-denoising procedure.The autoregressive model and diffusion head are jointly trained with a diffusion loss.
  • 2.2 AUTOREGRESSIVE MODELING WITHOUT DISCRETE TOKENIZATION: Diffusion sampling can require tens to hundreds of steps per sample, creating substantial per-frame latency for online V2A generation.This motivates methods that reduce sampling steps while preserving quality.
  • 2.3 EASY CONSISTENCY TUNING: ECT progressively tightens a consistency condition to bootstrap a pretrained diffusion model into a consistency model under a smaller training budget.Its training anneals the time difference between noisy samples from large diffusion-model values toward zero consistency-model values.

3 SOUNDREACTOR

SoundReactor models frame-level online V2A with a causal multimodal pipeline that predicts continuous audio latents from current and past video frames. It combines aggregated visual tokens, frame-aligned audio tokens, a decoder-only transformer with diffusion head, and two-stage training and inference for efficient generation.

  • Problem setup: SoundReactor models p(x_i | x_<i, v_≤i), ensuring each audio token uses only past audio and available video frames.The task assumes frame-aligned continuous audio and video tokens and forbids access to future video frames.
  • Token modeling: A pretrained vision encoder produces one aggregated token per RGB frame, while a VAE compresses full-band stereo waveforms into continuous audio tokens.The video pipeline uses DINOv2 patch features with temporal differences; the audio pipeline compresses stereo 48 kHz waveforms.
  • Multimodal transformer: Frame-aligned interleaved audio–visual tokens enter a causal decoder-only multimodal Transformer whose diffusion head generates the audio latents.The transformer uses a LLaMA-style architecture, and the framework is designed for end-to-end causal online generation.
  • Training framework: Training first uses DSM-based diffusion pretraining and then ECT consistency fine-tuning to accelerate diffusion-head decoding and reduce per-frame latency.Stage 2 initializes from Stage 1 and anneals the consistency interval toward zero while fine-tuning both the transformer and diffusion head.
  • Inference: Inference concatenates paired transformer outputs into each latent and decodes it with reverse diffusion or consistency-model sampling, using CFG and KV-cache for efficiency.CFG contrasts visual-conditioned outputs with outputs using a learnable null embedding, while KV-cache supports efficient autoregressive inference.

4 EXPERIMENTS

Experiments evaluate SoundReactor on diverse AAA gameplay videos against an offline autoregressive baseline and across long-context, latency, and ablation settings. SoundReactor outperforms V-AURA on all reported metrics except DeSync, while ECT reduces diffusion-head evaluations substantially and NTK preserves long-sequence timing.

  • 4.3 EVALUATION METRICS: PI slows periodic sounds and harms temporal synchronization, whereas NTK and SWA preserve timing in long-sequence spectrograms.The comparison uses SoundReactor-ECT with NFE=4 and evaluates sequences twice the training context window.
  • 4.2 COMPARING METHODS: SoundReactor methods outperform V-AURA on all reported metrics except DeSync, demonstrating effectiveness for frame-level online V2A generation.The evaluation combines objective metrics and subjective listening tests; V-AURA uses a non-causal video encoder and is not strictly compatible with the online setting.
  • 4.4 RESULTS: ECT matches SoundReactor-Diffusion at NFE=1 and surpasses it at NFE=4, using 14.8× fewer NFEs at NFE=4 and 59× fewer at NFE=1.SoundReactor-Diffusion uses NFE=59, whereas ECT requires only 1–4 diffusion-head NFEs with small performance degradation.
  • 4.4 RESULTS: NTK-aware interpolation avoids significant performance degradation across the first and last 8-second segments when generating 16-second sequences beyond the training window.Position Interpolation degrades throughout the sequence, while Sliding Window Attention degrades in the second half; NTK and SWA are competitive overall.
  • 4.4 RESULTS: Latency is measured per frame on 30FPS, 480p, 16-second clips using a single H100 GPU, with waveform-level latency including video-frame encoding and waveform decoding.The benchmark reports mean and standard deviation over 50 clips for NFE=1–4.
  • 4.5 ABLATION STUDY: ECT’s mapping function and dropout rate substantially affect sample quality, and only CF–0.2 produces high-quality samples at NFE=1.IN–0.2 and CF–0.2 perform comparably at NFE=2 and 4, while dropout tuning remains important for MAR-style models.

5 CONCLUSION

The paper introduces frame-level online V2A generation and SoundReactor, a framework designed for causal, low-latency audio generation from video. Experiments show high-quality full-band stereo audio under end-to-end causality, supporting interactive multimodal applications.

  • 5 CONCLUSION: The paper introduces frame-level online V2A generation and SoundReactor as a framework tailored to this setting.The task and framework target autoregressive audio generation without access to future video frames.
  • 5 CONCLUSION: SoundReactor generates high-quality full-band stereo audio under end-to-end causal constraints while maintaining low per-frame latency.The conclusion attributes these results to experiments and extensive ablations of the framework’s design choices.
  • 5 CONCLUSION: The work is positioned as a foundation for live content creation and sounding world models, motivating further research on interactive multimodal applications.The paper’s related-work discussion situates prior V2A research primarily in offline AR and non-AR settings.

A.2 AR AUDIO GENERATION WITHOUT VECTOR QUANTIZATION

This section describes SoundReactor’s causal autoregressive audio design, which uses continuous latents and frame-level visual conditioning for online V2A generation.

  • Vision token modeling: DINOv2 grid features are selected over [CLS] features because they provide temporal cues needed for audio-visual synchronization.Across 3,830 test clips, average cosine similarity is 0.99 ± 0.0072 for adjacent frames and 0.98 ± 0.010 for every other frame.
  • Frame-level online sampling: Frame-level sampling incrementally processes video and audio tokens, using diffusion or consistency-model sampling before waveform decoding.The procedure supports few-step consistency sampling with NFE=1–4 and incremental decoding when the VAE decoder is causal.
  • Vision token modeling: A single non-causal transformer layer with learnable positional embeddings aggregates each frame’s grid features into a visual conditioning token.This aggregation preserves frame-level causality while reducing the conditioning sequence to one token per frame.
  • Audio token modeling: SoundReactor uses continuous 30-Hz audio latents rather than vector-quantized tokens, supporting autoregressive generation of 48-kHz stereo audio.The stereo VAE produces 64-dimensional continuous latents after temporal downsampling.

B.2 TRAINING DETAILS

The training and evaluation setup combines diffusion pretraining with accelerated consistency fine-tuning, while testing causal context extension and per-frame latency.

  • Stage-1: Diffusion pretraining: Diffusion pretraining standardizes VAE latents and trains the diffusion head with EDM-style scaling for 300K iterations on 8 H100 GPUs.Training takes around 36 hours using bf16 and AdamW.
  • Stage-2: ECT: ECT fine-tuning applies a pseudo-Huber metric and noise-weighted consistency training for 200K iterations to accelerate head sampling.The second stage uses EMA weights and trains for around 20 hours on 8 H100 GPUs.
  • Classifier-free guidance: Classifier-free guidance is applied at the Transformer output rather than inside the diffusion head, reducing diffusion-head sampling overhead.The strategy remains comparable in performance to MAR’s head-level CFG while avoiding CFG computation during diffusion or consistency sampling.
  • Context window extension: For context-window extension, NTK and SWA preserve timing better than PI, whose position compression slows periodic sounds and harms synchronization.The discussion attributes this difference to preservation versus reduction of high-frequency positional structure.
  • Evaluation setup: SoundReactor is evaluated with both objective and human-aligned generation quality under causal frame-level conditions, using longer-context and latency experiments.Latency is measured on 30FPS, 480p clips with a single H100 and the default ECT model.
  • Latency and decoder trade-offs: Causal decoding reduces reconstruction and final generation quality relative to non-causal decoding despite enabling streaming waveform output.For ECT with NFE=4, causal decoding yields MMDOpenL3 = 65.5 and MMDCLAP = 0.682, versus 64.6 and 0.451 with the non-causal decoder.

C.1 ABLATION STUDY ON GENERATOR TRAINING

The generator ablations show that diffusion-head capacity matters under a fixed total parameter budget, while the ECT configuration is compared across fine-tuning scopes.

  • Diffusion head size: Larger diffusion heads consistently improve generation quality when total model capacity is fixed at 320 M parameters.The 10M-head variant fails to produce high-quality samples, motivating accelerated diffusion-head decoding.
  • Diffusion head size: The diffusion head requires substantial capacity for high-dimensional audio tokens, creating a direct decoding-speed cost.An even higher audio-token dimension of 128 yields no valid audio with a 50M head in preliminary trials.
  • ECT fine-tuning scope: Table 8 compares ECT fine-tuning of the entire network with fine-tuning only the diffusion head using IN-0.2 and Center-based MMD.The supplied table caption defines the two fine-tuning scopes and evaluation setting but does not provide cell values.

C.2 ABLATION STUDY ON SAMPLING STRATEGY

Sampling ablations identify the step count and guidance scale needed for valid, high-quality generation, while showing that the proposed CFG placement is computationally preferable.

  • Sampling steps: 30 or more diffusion steps are required for high-quality samples, whereas 10 steps fail and 20 steps only begin producing reasonable samples.The ablation fixes CFG scale at ω = 3.0 and reports NFE=19, 39, and 59 for 10, 20, and 30 steps.
  • CFG scale: Performance is best around CFG scales ω = 2.0–3.0 for both SoundReactor-Diffusion and SoundReactor-ECT.The ECT result holds across all evaluated head NFEs.
  • CFG strategy comparison: The proposed Transformer-level CFG strategy performs comparably to MAR’s head-level CFG while using less computational overhead.It avoids CFG during both diffusion and consistency-model sampling.

C.3 ABLATION STUDY ON VISION CONDITIONING

The vision-conditioning ablation evaluates PCA feature dimensionality and adjacent-frame subtraction, finding that temporal differencing is important for performance.

  • Table 9 evaluates varying PCA-reduced feature dimensionalities alongside the use of adjacent-frame subtraction in SoundReactor-Diffusion.Bold and underlined scores denote the best and second-best results, respectively.
  • Removing adjacent-frame subtraction consistently degrades performance, indicating that temporal differencing is crucial to the vision-conditioning mechanism.
  • Figure 8 examines the effect of CFG scale ω with SoundReactor-ECT across different head NFEs.

C.4.1 EXPERIMENTAL SETUP

The supplementary VGGSound evaluation uses standardized audio and audio-visual metrics on real-world videos, with modified model and inference settings.

  • VGGSound contains 200K+ 10-second YouTube clips across 309 categories, with approximately 180K 8-second training videos used here.The evaluation follows MMAudio’s data split and preprocessing pipeline and uses only audio-visual pairs.
  • The study measures audio quality with FADPaSST and KLPaSST, and audio-visual alignment with ImageBind score and DeSync.Metrics are computed on Center, the average of the left and right audio channels.
  • For VGGSound, the model uses a three-layer non-causal aggregator, a 768-dimensional learnable token, PCA-reduced DINOv2 features, and ω = 4.0 during inference.The DINOv2 feature dimension is reduced to 146 while retaining 80% cumulative explained variance, followed by 30-step deterministic Heun sampling.

C.4.2 RESULTS

On VGGSound, SoundReactor generalizes to real-world videos and outperforms several offline non-autoregressive methods, but remains behind MMAudio and has metric-specific limitations.

  • SoundReactor outperforms Seeing & Hearing, ReWaS, and FoleyCrafter, while a significant gap remains versus MMAudio, the state-of-the-art V2A model.Against V-AURA, it shows similar FADPaSST and DeSync trends but lags on IB-Score and KLPaSST.
  • Larger image or causally pretrained video encoders could help bridge the performance gap, but preserving end-to-end causality and low frame-level latency remains challenging.The paper leaves this architectural improvement as future work.
  • DeSync may favor models using Synchformer features, suggesting a potential methodological bias in this metric.Models using Synchformer features outperform VAE reconstruction on DeSync.
  • Table 10 reports additional VGGSound test-set experiments comparing SoundReactor with offline V2A models.Only SoundReactor models in the table are applicable to frame-level online V2A.
  • Human evaluation is documented with a screenshot accompanying the results section.
Loading 2510.02110v1…