X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation

• Large multimodal datasets are costly to store, curate, and repeatedly train on, motivating data-centric methods and curation [Schwartz et al., 2020, Strubell et al., 2019].Source: §1 · p. 1
• X-AVDD consistently outperforms DM and AVDD on VGGSound across SPC settings [Kushwaha et al., 2024], and improves over AVDD and multimodal SRe2L across benchmarks.Source: Figure 1 · p. 1, §1 · p. 2
• Audio-visual distillation must preserve cross-modal associations alongside modality-specific information, while AVDD is memory-intensive and struggles with strong semantic correspondence.Source: §1 · p. 2
• X-AVDD explicitly models inter-modal relationships with bidirectional, multi-head cross-attention and separates cross-modal alignment from joint optimization using a pretrained teacher.Source: §1 · p. 2
• X-AVDD reduces synthesis time on a Tesla V100 GPU from 10 hours to 1.3 hours.Source: §1 · p. 2

2.1 Dataset Distillation

• Dataset distillation synthesizes compact labeled examples to preserve the training signal of a larger dataset; methods include bi-level optimization [Wang et al., 2018] and distribution matching.Source: §2.1 · p. 3

2.2 Attention in DD

• Unlike unimodal spatial-attention matching in DataDAM [Sajedi et al., 2023], X-AVDD uses bidirectional multi-head cross-attention to model audio-visual temporal and semantic dependencies during synthesis.Source: §2.2 · p. 3

2.3 Multimodal Dataset Distillation (MDD)

• Audio-visual distillation remains under-explored, and modality collapse describes synthetic pairs losing semantic alignment; X-AVDD addresses this with decoupled alignment.Source: §2.3 · p. 3

2.4 Audio-Visual Event Recognition

• Audio-visual event recognition relies on aligned representations, with AVE [Tian et al., 2018] and VGGSound [Chen et al., 2020] supporting work on architectures such as AudioCLIP [Guzhov et al., 2022].Source: §2.4 · p. 3

2.5 Limitations of AVDD

• AVDD faces OOM failures as SPC or model complexity increases; X-AVDD decouples alignment from synthesis to support class-wise optimization with lower peak VRAM and synthesis time.Source: §2.5 · p. 3

3 Methodology

• The distillation target is a compact synthetic dataset whose trained model aims for generalization performance comparable to a model trained on the full dataset.Each sample pairs audio represented by a spectrogram with a visual input represented by the video's central frame and a ground-truth label.Source: §3 · p. 4
• X-AVDD decouples synthetic-data distillation from teacher training and uses bidirectional cross-attention to extract complementary information from both modalities.The authors state that this decoupling allows distillation to be parallelized and not depend on access to the original large dataset.Source: §3 · p. 4
• The method pre-trains a teacher with bidirectional cross-attention, computes and stores real audio and visual embeddings, then iteratively learns the synthetic dataset.The iterative stage matches cross-attended embeddings from real and synthetic samples.Source: §3 · p. 4
• Euclidean Distribution Matching (EDM) encourages synthetic embeddings to cluster near real ones by minimizing pairwise Euclidean distances between attended embeddings.The distillation process is initialized with randomly sampled audio-frame pairs from real data.Source: §3 · pp. 4–5
• For synthetic embeddings, the audio-guided image branch uses synthetic images as queries with real audio as keys and values, while the image-guided audio branch reverses the modalities.The two branches implement cross-modal information exchange in opposite directions.Source: §3 · p. 5
• The synthesis loss combines classification cross-entropy with weighted audio-to-visual and visual-to-audio cross-modal losses.The loss weights are λce, λav, and λva.Source: §3 · p. 5

4.1 Results for ConvNet, ResNet-18 and ViT architectures

• Across ConvNet, ResNet-18, and ViT settings, X-AVDD significantly outperforms AVDD across most SPC settings, with benefits emerging at SPC ≥10 for ConvNet and a particularly significant VGGSound jump.Table 1 reports test accuracies for models trained from scratch; comparisons are against DM and AVDD for ConvNet and multimodal SRe2L for ResNet-18 and ViT.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Results for AudioCLIP architecture

• With AudioCLIP, X-AVDD's distilled data achieves comparable generalization accuracy to the full-data setting while taking a fraction of its training time.Table 2 reports test accuracy on VGGSound-10K and AVE and gives the condensed-data ratio.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Cross architecture analysis

• X-AVDD’s ResNet-18-generated synthetic dataset achieves higher generalization accuracy than the SRe2L adaptation on VGG-11 and ViT target architectures, using ensemble fusion.The synthetic dataset was generated using ResNet-18 for both audio and video modalities.Source: Table 3 · p. 7

4.4 Transfer learning with frozen backbone of SOTA image and audio encoder models

• With 50 samples per class, X-AVDD data generated using ResNet-18 achieves performance comparable to full-data training when used with frozen state-of-the-art backbones.The transfer-learning evaluation compares datasets distilled using ConvNet and ResNet-18.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 Memory comparison

• Batch-wise image generation lets X-AVDD use less memory than AVDD, supporting scaling to higher SPC settings and larger architectures.The GPU-hour comparison uses a single 32 GB Tesla V100 GPU at SPC=20 on ConvNet.Source: §4.5 · p. 8, Table 5 · p. 8

4.6.2 Directionality ablation

• Either cross-modal EDM direction alone underperforms the full bidirectional objective across SPC regimes, indicating complementary constraints.The ablation toggles the video→audio and audio→video loss weights on VGGSound-10K.Source: §4.6.2 · p. 8, Table 6 · p. 9

4.6.3 Improvement Over Random

• X-AVDD achieves significant improvement over random selection across multiple datasets, measured by the difference between synthetic-data and random-subset test accuracy.Table 7 reports IOR for each dataset and SPC setting.Source: §4.6.3 · p. 9, Table 7 · p. 9

4.6.1 Impact of α on test accuracy

• Combining hard and soft labels improves test accuracy for both SRe2L and X-AVDD; the α study also reports higher generalization accuracy when both label types are combined.The α-weighted loss is a convex combination of hard-label cross-entropy and soft-label KL divergence.Source: §4.6.1 · p. 8, §4.6.4 · p. 9, Table 8 · p. 9

5 Conclusion

• X-AVDD uses bidirectional cross-attention to jointly synthesize audio-visual data, preserving semantic alignment while reducing memory and computational overhead.The method distills through cross-attended embeddings rather than independently matched modality features.Source: §5 · p. 9
• Experiments report strong performance across four audio-visual benchmarks, cross-architecture comparisons, and finetuning of frozen state-of-the-art backbones.The cross-architecture evaluation uses VGG-11 and ViT.Source: §5 · p. 9
• The paper focuses on audio-visual classification; temporal localization and audio-visual retrieval remain future work.Source: §5 · p. 9

A Table of notation

• Table 10 collects notation used throughout the paper.Source: A Table of notation · p. 13, Table 10 · p. 13

C Computing Infrastructure Information

• Table 12 provides details of the computing infrastructure used for the experiments.Source: C Computing Infrastructure Information · p. 13, Table 12 · p. 14

B Diversity Comparison

• X-AVDD outperforms the multimodal SRe2L baseline across all three diversity metrics.The metrics are Vendi Score, intra-class cosine similarity, and coverage.Source: B Diversity Comparison · p. 13, Table 11 · p. 14

• SRe2L proceeds through Squeeze, Recover, and Relabel stages [Yin et al., 2023].Squeeze stores BatchNorm statistics from a model trained on the original data; Recover uses them to synthesize data, and Relabel assigns synthetic data soft labels.Source: D Implementation of multimodal SRe2L · p. 14
• The audio-visual adaptation trains a teacher on the full training set and uses stored BatchNorm statistics during synthesis.Source: D Implementation of multimodal SRe2L · p. 14
• Real samples initialize Recover instead of independent Gaussian noise, and the same initialization samples are used for SRe2L and X-AVDD.The paper identifies random noise’s lack of semantic information as a bottleneck and notes other methods also use real-sample initialization [Cazenavette et al., 2022, Du et al., 2024].Source: D Implementation of multimodal SRe2L · p. 14

• AVE has 4,143 ten-second videos across 28 event categories, and its annotation-aligned one-second windows yield 27,726 / 3,288 / 3,305 train/val/test samples [Tian et al., 2018].Source: E Dataset details · p. 14
• MUSIC-21’s processed solo subset uses 146,908 / 7,103 / 42,440 train/val/test samples [Zhao et al., 2019].The dataset contains musical solo and duet videos; the original paper reports 1,065 training videos and 300 test videos, processed into 5,861 shots.Source: E Dataset details · pp. 14–15
• VGGSound’s AVDD-provided preprocessed split contains approximately 165k / 13k train/test samples across 309 classes [Chen et al., 2020, Kushwaha et al., 2024].VGGSound is a large-scale dataset of short YouTube clips curated for audio-visual correspondence.Source: E Dataset details · p. 15
• The VGGSound-10K subset has 8,808 / 444 train/test samples across 10 classes using AVDD’s preprocessing and split.Source: E Dataset details · p. 15

• Table 13 lists hyperparameters for pre-training the teacher and the audio-guided video and video-guided audio networks.Source: F Hyperparameter details · p. 15, Table 13 · p. 15
• Table 14 tabulates the hyperparameters used during synthesis.Source: F Hyperparameter details · p. 15

G Timing comparison

• X-AVDD's synthesis stage is minuscule versus AVDD's; teacher training and embedding storage are one-time, with only synthesis recurring across SPC settings.The timing comparison was run on a single 32 GB Tesla V100.Source: G Timing comparison · p. 15, Table 15 · p. 15

H Modality gap on VGGSound-10K

• Across VGGSound-10K classes, X-AVDD achieves a consistently lower modality gap than AVDD-generated synthetic data.Figure 5 compares the modality gap in real data and synthetic data from both methods.Source: H Modality gap on VGGSound-10K · p. 15, Figure 5 · p. 16

I Impact of temperature on soft label generation

• For VGGSound-10K with ResNet-18 at SPC=10 and SPC=20, test accuracy rises with temperature to a maximum and then falls.The paper identifies temperature τ as important for smoothing soft labels Müller et al. [2019].Source: Figure 6 · p. 16, I Impact of temperature on soft label… · p. 16

J Results with ‘concat’ fusion method

• X-AVDD consistently outperforms the SRe2L baseline across four datasets and various SPC settings with concat fusion.Table 16 reports test-set accuracy for the concat fusion method.Source: J Results with ‘concat’ fusion method · p. 17, Table 16 · p. 17