SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet

1. INTRODUCTION

• Foley synthesis requires audio that is semantically and temporally aligned with video, a need with broad relevance to creative industries.Text-to-audio systems can produce realistic, text-aligned sounds, but their use in foley synthesis remains limited.Source: §1 · p. 1
• Joint audio-visual generative modeling can learn temporal alignment, but training such models can be expensive; pretrained audio-video latent alignment has yielded poor temporal alignment.The paper identifies steering pretrained audio models as an attractive alternative to training an audio generator from scratch.Source: §1 · p. 1
• Earlier ControlNet foley methods inject handcrafted temporal conditions, while from-scratch systems use high-dimensional features from pretrained video encoders.The paper frames the performance gap between these approaches as motivation for improving pretrained-model-based foley synthesis.Source: §1 · p. 1
• SpecMaskFoley outperforms prior ControlNet models and strong from-scratch AR, MaskGIT, and flow-matching baselines on a widely used foley benchmark.It steers pretrained SpecMaskGIT through a single ControlNet branch using a frequency-aware temporal feature aligner.Source: §1 · p. 1

2. RELATED WORKS

• Jointly modeling synchronized audio and video can produce aligned latent spaces, but training multiple modalities from scratch is non-trivial.Adapting or aligning pretrained audio and video latent spaces has been explored, but prior work reports poor temporal alignment.Source: §2 · pp. 1–2
• Prior ControlNet foley methods inject handcrafted conditions such as energy curves, onset timestamps, or binarized CLIP stamps; FoleyCrafter also uses parallel cross-attention adaptors.These approaches differ from from-scratch systems that leverage deep visual features.Source: §2 · p. 2
• From-scratch systems use pretrained visual encoders with varied audio generators, including MaskGIT, autoregressive Transformers, and flow matching.Examples include VATT, V-AURA, Frieren, and MMAudio, with the latter two using 1-D VAE-GAN latents.Source: §2 · p. 2
• More effective use of pretrained audio generators is proposed as a way to narrow the observed performance gap between ControlNet-based and from-scratch foley models.The paper presents that gap as motivation for its approach, not as evidence that existing pretrained-model methods already match from-scratch systems.Source: §2 · p. 2

3. SPECMASKFOLEY

• SpecMaskFoley combines pretrained SpecMaskGIT, a ControlNet branch, and an FT-Aligner that adapts deep video features to the time-frequency ControlNet.SpecMaskGIT was selected for competitive text-to-audio performance, efficient inference, and resource-efficient experimentation.Source: §3 · p. 2, §3.1 · p. 2

3.2. Deep Video Features

• The method uses 25 Hz Synchformer features for high-frame-rate video information and 8 Hz CLIP features for semantic information.CLIP features are averaged across frames and added to the Synchformer sequence to form temporal features.Source: §3.2 · p. 2

3.3. ControlNet and FT-Aligner

• The Transformer ControlNet branch starts from pretrained weights and connects to the backbone through zero-initialized linear layers.Training uses cross-entropy loss.Source: §3.3 · p. 2
• FT-Aligner addresses the mismatch between 1-D temporal video features and SpecMaskGIT’s 2-D time-frequency token grid by repeating each temporal feature across frequency.It downsamples projected features to [1, T, D], then repeats them to produce [F, T, D]; the authors report that careful alignment facilitates convergence and is necessary for training.Source: §3.3 · pp. 2–3

3.4. Multi Classifier-Free Guidance

• Multi-CFG extends single-condition guidance to combine video-only and joint text-and-video conditioning with unconditional logits.The method randomly replaces CLAP conditioning with an unconditional mask for 90% of training steps because text prompts alone were found less important.Source: §3.4 · p. 3

4.1. Datasets

• VGGSound is the study’s only audio-visual training dataset, with around 500 hours of clips and around 180K 10-second clips in the training split.The study follows MMAudio’s data split and preprocessing but keeps videos at 10 seconds to match the audio backbone’s pretraining clip length.Source: §4.1 · p. 3

4.2. Implementation Details

• SpecMaskGIT was pretrained on AudioSet, and its audio input uses 10-second, 22.05 kHz clips transformed into 848-frame, 80-bin Mel-spectrograms and then a [F = 5, T = 53] token map.The ControlNet branch is initialized from the backbone’s first 12 of 24 Transformer blocks; FT-Aligner maps video features to the token-map shape.Source: §4.2 · p. 3

4.3. Metrics

• Evaluation measures audio synthesis quality with FD, KL, and FAD, semantic matching with ImageBind cosine similarity, and synchronization with DeSync in seconds.FAD uses the 16 kHz VGGish classifier; FD and KL use PaSST, and PANN is excluded from metric computation.Source: §4.3 · p. 3

5.1. VGGSound Benchmarking

• SpecMaskFoley outperforms previous ControlNet-based methods in audio quality, audio-video semantic matching, synchronization, and inference speed.The authors attribute its advantage to the SpecMaskGIT backbone, deep video features, and their use with ControlNet and FT-Aligner.Source: §5.1 · p. 3, Table 1 · p. 4
• SpecMaskFoley substantially outperforms VATT on most metrics, exceeds V-AURA in audio quality and inference speed while matching its synchronization, and performs similarly to Frieren.Against Frieren, it is more advantageous on FAD, KL, IB similarity, and DeSync; its 126M trainable parameters enable single-GPU training.Source: §5.1 · pp. 3–4
• SpecMaskFoley substantially narrows the observed performance gap between ControlNet-based and from-scratch foley methods.It remains competitive with strong from-scratch baselines, including Frieren and MMAudio.Source: §5.1 · p. 4

5.2. Ablation Studies and Discussions

• Adding ControlNet improves the backbone’s audio-video semantic matching and temporal synchronization without significantly degrading audio quality.Backbone-only prompting with test-set CLAP audio features yields FAD 0.91 and KL 0.95, but semantic matching and temporal alignment remain weak without injected video features.Source: §5.2 · p. 4
• Multi-CFG improves all metrics over standard CFG with only a slightly increased, still affordable inference cost.Source: §5.2 · p. 4, Table 2 · p. 4
• FAD and DeSync scores saturate after 12 inference steps, versus the 16-step optimum reported for SpecMaskGIT; four-step synthesis already outperforms VATT and FoleyCrafter on FAD and DeSync.At six steps, FAD becomes close to ReWaS; the few-step capability requires no distillation.Source: §5.2 · p. 4, Figure 3 · p. 4
• The 2-D VAE represents a 10-second clip with 53 temporal frames, versus more than 300 for the 1-D VAEs in Frieren and MMAudio; this was not a DeSync bottleneck on VGGSound but may limit future 2-D methods.Transplanting SpecMaskGIT and SpecMaskFoley to 1-D is left for future work.Source: §5.2 · p. 4

• SpecMaskFoley could outperform strong from-scratch baselines on a common foley synthesis benchmark.Source: §6 · p. 4
• The method steers pretrained SpecMaskGIT toward video-synchronized foley synthesis by directly processing deep video features with ControlNet.Source: §6 · p. 4
• A frequency-aware temporal feature aligner resolves the mismatch between temporal video features and SpecMaskGIT's time-frequency nature, simplifying conditioning.Source: §6 · p. 4
• Future work includes transferring the approach to 1-D audio MaskGIT models and extending audio ControlNet toward spatial audio.Source: §6 · p. 4