Vision Transformers are Parameter-Efficient Audio-Visual Learners

1. Introduction

• Modality-specific systems require substantial training resources and separate backbones, while late fusion limits access to cross-modal cues in early layers.The introduction cites more than 2,000 V100 hours for large-scale audio training, more than 5,000 for video training, and over 48 GB of GPU memory for MBT-Large.Source: §1 · p. 1
• The study tests whether image-pretrained ViTs can also encode useful audio representations, since audio can be represented as a 2D spectrogram and vision architectures have processed such audio images.The authors contrast this hypothesis with prior methods that use vision architectures for large-scale audio representation learning.Source: §1 · p. 2
• LAVISH adapts frozen image-pretrained ViTs by adding trainable modules at every layer; latent tokens compress one modality before cross-attention with the other, avoiding standard cross-attention’s quadratic cost.The scheme is bidirectional, transferring information from audio to visual tokens and vice versa.Source: §1 · p. 2
• LAVISH achieves competitive or better results than modality-specific audio-visual approaches on event localization, segmentation, and question answering, with fewer tunable parameters and no costly audio pretraining or external audio encoder.The authors also report that LAVISH is more effective and efficient than standard adapter schemes.Source: §1 · p. 2

2. Related Work

• Audio-visual methods commonly use modality-specific audio and visual models with ad-hoc fusion, while LAVISH studies image-pretrained frozen ViTs without large-scale audio-visual pretraining.The related-work discussion spans event localization, video parsing, sound localization, segmentation, and audio-visual question answering.Source: §2 · pp. 2–3
• Parameter-efficient methods add parameters, update sparse weights, or use low-rank factorization; standard adapters are lightweight but generally designed for unimodal settings.The paper distinguishes recent multimodal parameter-efficient approaches by their reliance on costly large-scale multimodal pretraining.Source: §2 · pp. 2–3
• LAVISH uses separate audio and visual latent tokens to compress each modality, fuse across modalities, and pass the fused representation through a lightweight adapter.The overview describes bidirectional information transfer through adapters inserted into each frozen ViT layer.Source: Figure 2 · p. 3

3.1. Audio-Visual Input Embeddings

• Each RGB frame is patch-tokenized, while a multi-second spectrogram centered on that frame is projected into audio tokens, with its channels expanded to match the ViT input.The visual and audio inputs are represented as separate token embeddings.Source: §3.1 · p. 3

3.2. Adding LAVISH Adapters to a Frozen ViT

• Each frozen ViT layer retains its MHA and MLP while adding four separately parameterized adapters across the visual and audio streams for bidirectional information exchange.The adapters can transfer information from audio to visual tokens and from visual to audio tokens.Source: §3.2 · p. 3

3.3. LAVISH Adapter

• The dual-pathway LAVISH adapter combines latent-token compression, audio-visual fusion, and a lightweight bottleneck module to incorporate cross-modal cues into new feature representations.The bottleneck uses a learnable down-projection, nonlinear activation, and up-projection.Source: §3.3 · pp. 4–5

3. Technical Approach

• For the downstream task configurations in Figure 3, the audio and visual backbones are frozen and share parameters, while adopted baseline modules remain trainable.The figure covers audio-visual event localization, segmentation, and question answering.Source: Figure 3 · p. 5

3.3. LAVISH Adapter

• LAVISH uses separate latent-token sets to compress each modality before cross-modal fusion, avoiding standard cross-attention's quadratic cost for streams with more than 2K tokens.In most cases, it uses m = 2 latent tokens, far fewer than the token count of the Swin transformer.Source: §3.3 · pp. 4–5

• On AVE, event localization uses 4,143 ten-second videos across 28 categories; frozen transformer features are concatenated per segment for linear prediction and evaluated by segment accuracy.Source: §4.1 · p. 5
• On AVSBench-S4, the model replaces AVS’s pretrained U-Net and audio feature extractor with a frozen transformer plus LAVISH adapters, evaluated by mean Intersection-over-Union.Source: §4.1 · pp. 5–6
• LAVISH achieves better event-localization accuracy than all prior methods without external audio pretraining, while using relatively few trainable parameters.Source: Table 1 · p. 6
• On MUSIC-AVQA, LAVISH replaces the baseline’s pretrained visual and audio encoders, with answer-prediction accuracy measured over 9,288 videos and 45,867 question-answer pairs.Source: §4.1 · p. 6

5.1. Audio-Visual Event Localization

• 81.1% vs. 80.4% accuracy: LAVISH surpasses CMBS with the same Swin-V2-L backbone, using 10.1M vs. 14.1M trainable parameters without audio pretraining.At the large scale, LAVISH uses 14.5M trainable parameters and reaches 78.1% accuracy, versus MBT-L's 393M parameters and 77.8%; MBT-L runs out of memory on a 48GB A6000.Source: §5.1 · p. 6

5.2. Audio-Visual Segmentation

• 80.1% vs. 78.7% mIoU: LAVISH surpasses the prior AVS method without an external audio encoder or large-scale audio pretraining.With a shared Swin-V2-L backbone, AVS reaches 80.4% vs. LAVISH's 80.1%, using 249.7M vs. 37.2M trainable parameters.Source: §5.2 · p. 7, Table 2 · p. 7

5.3. Audio-Visual Question Answering

• LAVISH achieves the best accuracy across audio, visual, and audio-visual question categories, including audio-only questions, without a pretrained audio encoder or large-scale audio pretraining.Reported gains over prior approaches are +3.09%, +3.37%, and +7.54%; gains are larger for audio-visual questions (+2.57%) than visual (+1.73%) or audio-based (+1.69%) questions.Source: §5.3 · p. 7, Table 3 · p. 7

5.4. Action Recognition on UCF101.

• 92.6% vs. 86.8%, 86.8%, and 89.3%: LAVISH outperforms XDC, AVTS, and GDT on UCF101 without audio-visual pretraining.The compared XDC, AVTS, and GDT methods used large-scale audio-visual pretraining on Kinetics-400.Source: §5.4 · p. 7, Table 4 · p. 8

5.5. Ablation Studies

• Bidirectional fusion improves over no cross-modal connections and one-way A2V or V2A variants, while LAVISH also outperforms AVISH variants without latent tokens.On AVE, bidirectional LAVISH gains +3.2% over the no-connection baseline and +2.3% and +2.4% over the unidirectional baselines; it gains +1.3% over bidirectional AVISH.Source: §5.5 · p. 8, Table 5 · p. 8
• 11 vs. 217 GFLOPs: LAVISH's modules cost 20× less than AVISH's and save about 20% GPU memory for training; two latent tokens yield the best AVE accuracy at 81.1%.AVISH performs cross-attention between every pair of audio and visual tokens; using more latent tokens linearly increases computation without improving AVE results, while the larger AVS and AVQA datasets have an optimum of 16 tokens.Source: §5.5 · pp. 8–9, Figure 4 · p. 9
• LAVISH reaches 81.1% vs. LoRA's 79.1% accuracy using 10.1M vs. 17.7M trainable parameters, and outpaces CMBS at 1.40 vs. 0.72 samples/sec.Compacter and Adapter use 8.9M and 3.7M trainable parameters but reach 79.1% and 78.4% accuracy, respectively; adding VGGish features yields a 1.3% boost.Source: §5.5 · pp. 9–10, Table 9 · p. 10, Table 8 · p. 10
• Audio-visual variants outperform visual-only baselines by 5.8% top-1 accuracy on AVE, 8% mIoU on AVS, and 13.9% top-1 accuracy on AVQA.The AVE, AVS, and AVQA results compare joint audio-visual modeling against variants using only visual information.Source: §5.5 · p. 9, Table 7 · p. 9

6. Conclusions

• Without audio pretraining, LAVISH outperforms state-of-the-art approaches on audio-visual event localization, segmentation, and question answering.Source: §6 · p. 10
• Compared with prior approaches, LAVISH requires significantly fewer trainable parameters for audio-visual adaptation.The authors describe this as enabling efficient adaptation.Source: §6 · p. 10
• Future work will examine transfer to audio-only and visual-language tasks, and transfer of pretrained audio models to audio-visual data.Source: §6 · p. 10

A. Implementation Details

• All experiments sample visual frames at 1 fps and use a frozen Swin-V2-Large pretrained at 192 × 192 resolution.Source: A. Implementation Details · p. 10
• Event localization uses 2 latent tokens, while segmentation and question answering use 16, and action recognition uses 24.The downsampling factor is 8 for event localization and 4 for the other three tasks; each uses 2 group convolutions.Source: A. Implementation Details · p. 10
• All experiments use Adam, with task-specific learning rates for the LAVISH adapter and, where applicable, prediction or grounding modules.The adapter learning rates are 5e−6 for event localization, 1e−4 for segmentation, 8e−5 for question answering, and 3e−5 for action recognition.Source: A. Implementation Details · p. 10
• Audio inputs use Kaldi fbank spectrograms with 192 triangular mel-frequency bins and a 5.2-millisecond frame shift, expanded from one channel to three.The channel expansion matches the SwinV2 linear patch projection dimensions.Source: A. Implementation Details · p. 11