PaperlayerView exact revision on arXiv

Source-linked AI summary

ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning

Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, Yale Song

arXiv:2101.10803v3cs.CV

TL;DR

Online videos offer audio-visual self-supervision but often contain irrelevant or overdubbed audio, while manually curated alternatives are labor intensive. ACAV100M automatically selects subsets maximizing audio-visual mutual information using scalable clustering-based estimation, achieving higher precision than ranking baselines and supporting a 100M-video dataset.

  • Problem

    Online videos often contain noisy audio-visual signals, whereas manually verified datasets are labor intensive to construct, limiting scalable self-supervised video learning.

    Source: §1 · pp. 1–2

  • Method

    The paper uses information-theoretic subset optimization to maximize mutual information between audio and visual channels, estimating it with multi-layer clustering and generic off-the-shelf feature extractors.

    Source: §1 · p. 2

  • Results

    The clustering approach achieves higher precision than ranking baselines on the evaluated datasets, with multi-layer variants outperforming single-layer methods and broader clustering connections improving performance.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    ACAV100M provides an automatically curated, open-domain dataset of 100M videos for audio-visual representation learning.

    Source: §1 · p. 2, §6 · p. 8

    The pipeline depends on feature extractors that capture useful information across low- and high-level aspects, but an ideal extractor is difficult to obtain and impractical to learn at this scale.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

The natural association between visual observations and their corresponding sound provides powerful self-supervisory signals for learning video representations, which makes the ever-growing amount of online videos an attractive source of training data. However, large portions of online videos contain irrelevant audio-visual signals because of edited/overdubbed audio, and models trained on such uncurated videos have shown to learn suboptimal representations. Therefore, existing approaches rely almost exclusively on datasets with predetermined taxonomies of semantic concepts, where there is a high chance of audio-visual correspondence. Unfortunately, constructing such datasets require labor intensive manual annotation and/or verification, which severely limits the utility of online videos for large-scale learning. In this work, we present an automatic dataset curation approach based on subset optimization where the objective is to maximize the mutual information between audio and visual channels in videos. We demonstrate that our approach finds videos with high audio-visual correspondence and show that self-supervised models trained on our data achieve competitive performances compared to models trained on existing manually curated datasets. The most significant benefit of our approach is scalability: We release ACAV100M that contains 100 million videos with high audio-visual correspondence, ideal for self-supervised video representation learning.

1. Introduction

The paper addresses scalable curation of audio-visual clips without manual annotation by selecting videos with high mutual information between audio and visual channels. It produces ACAV100M and shows that curated data supports competitive self-supervised representation learning.

1. Introduction

  • Existing audio-visual self-supervision relies on manually created or domain-specific datasets because online videos often contain irrelevant or mismatched audio and visual signals.Manual labeling is impractical for full-length, diverse video collections.Source: §1 · p. 1
  • The method automatically reduces arbitrary-length video collections to short clips by maximizing mutual information between audio and visual channels.This curation step supplies datasets for existing or future self-supervised algorithms rather than defining the learning algorithm itself.Source: Figure 1 · p. 1, §1 · p. 2
  • Off-the-shelf models provide multi-layer audio and visual features for clustering-based mutual-information estimation, avoiding dependence on classifiers tied to restricted concept sets.The approach uses generic feature extraction across multiple layers.Source: §1 · p. 2
  • SGD-based K-means and mini-batch greedy optimization make subset selection scalable to hundreds of millions of clips.The method avoids memory-heavy clustering and approximately solves the subset maximization objective.Source: §1 · p. 2
  • 100 million 10-second clips are curated from 140 million full-length videos into ACAV100M with high audio-visual correspondence.The processed source totals 1,030 years, while the resulting clips total 31 years.Source: §1 · p. 2
  • Models pretrained on the curated datasets perform competitively or better than models trained on similarly sized manually curated baselines across UCF101, ESC-50, and Kinetics-Sounds.The comparisons use Kinetics-Sounds, VGG-Sound, and AudioSet at approximately 20K-, 200K-, and 2M-scale, respectively.Source: §1 · p. 2

2. Related Work

Prior audio-visual datasets commonly depend on manual labeling, domain restrictions, or keyword-based collection with filtering and review. This work instead applies mutual information to automatic, large-scale subset selection for audio-visual dataset curation.

2. Related Work

  • Existing audio-visual datasets include manually labeled, domain-specific, and unrestricted unlabeled collections.Examples include AudioSet, AVE, AVA datasets, Greatest Hits, FAIR-Play, YouTube-ASMR-300K, and Flickr-SoundNet.Source: §2 · p. 2
  • Large datasets such as AudioSet, Moments in Time, HowTo100M, WVT, and Chen et al. rely on keyword search, metadata, classifiers, manual review, or combinations of these steps.The cited construction pipelines use different forms of filtering and verification to improve correspondence.Source: §2 · p. 3
  • The paper targets the cost and impracticality of manually collecting and labeling the large training sets needed for diverse, unrestricted audio-visual inputs.Its proposed alternative is an automatic and scalable data curation pipeline.Source: §2 · p. 3
  • Unlike prior subset-selection criteria, the paper uses mutual information between audio and visual features to measure within-video correspondence and guide large-scale curation.For clustering-based estimation, greedy optimization is presented as a practical solution.Source: §2 · p. 3

3. Data Collection Pipeline

The pipeline converts unconstrained web videos into short clips, estimates audio-visual mutual information, and selects a scalable subset with strong correspondence. It combines metadata filtering, pretrained feature extraction, clustering-based MI estimation, and approximate greedy optimization.

3. Data Collection Pipeline

  • The pipeline comprises raw-video acquisition and metadata filtering, clip segmentation and feature extraction, MI estimation, and MI-maximizing subset selection.Source: §3 · p. 3

3.1. Obtaining Candidate Videos

  • Candidate videos are gathered across diverse topics, locations, events, categories, cultures, and languages rather than from a narrowly curated keyword set.Metadata filtering excludes videos outside the 30–600-second duration range, selected categories, and languages beyond the cumulative 0.9 ratio.Source: §3.1 · p. 3
  • The collection comprises 140 million full-length videos totaling 1,030 years (median duration: 198 seconds); downloading them at 360p still requires 1.8 petabytes of storage.Handling such large-scale data requires a carefully designed data pipeline.Source: §3.1 · p. 3

3.2. Segmentation & Feature Extraction

  • Up to three nonredundant 10-second clips are extracted per full-length video, yielding about 300 million clips for correspondence measurement.Shot boundaries and MPEG-7 video signatures guide local-search selection of clips with low pairwise similarity.Source: §3.2 · p. 3
  • Pretrained SlowFast and VGGish models provide visual and audio features because learning optimal extractors end-to-end is impractical at this scale.The approach uses pretrained models rather than an oracle representation capable of covering low-level and high-level information.Source: §3.2 · pp. 3–4

3.3.1 NCE-based MI Estimation

  • NCE estimates MI in continuous feature space per clip, whereas clustering estimates MI in discrete vector-quantized space for a clip set and is more robust to real-world noise.NCE uses projection heads, cosine similarity, and a temperature of 0.1, then ranks clips by similarity for top-N selection.Source: §3.3 · p. 4, §3.3.1 · p. 4, §3.3.2 · p. 4

3.3.2 Clustering-based MI Estimation

  • The clustering estimator averages MI across pairings of audio and visual clusterings from five convolutional blocks, spanning low-level textures to high-level object parts.The combination pairing scheme includes within- and across-modality layer pairs, with 10C2 equal to 45 pairings.Source: §3.3.2 · p. 4
  • Batch greedy selection reduces subset-optimization complexity from quadratic in 300 million candidates to O(N × |B|) by searching each iteration within a sampled batch.This approximates the usual greedy solution while making selection scalable to the full clip population.Source: §3.3.2 · pp. 4–5

4. Evaluation on Correspondence Retrieval

The evaluation tests correspondence retrieval across synthetic and real-world audio-visual datasets, then examines how clustering, greedy selection, and centroid choices affect performance. The clustering approach performs best on Kinetics-Sounds, while multi-layer clustering, combination pairing, and scalable approximations improve or preserve retrieval quality.

4.1. Experimental Setting

  • The evaluation uses synthetic natural and arbitrary class correspondences alongside sample-level audio-visual correspondence in Kinetics-Sounds.Each dataset is split into equal-sized train and test partitions, with five runs reported for each dataset.Source: §4 · p. 5, §4.1 · pp. 5–6

4. Evaluation on Correspondence Retrieval

  • The clustering approach excels on Kinetics-Sounds, whereas contrastive estimation performs well on datasets with natural class correspondence.Source: §4.2 · p. 6

4.2. Ablation Results & Discussion

  • Multi-layer clustering substantially outperforms every single-layer variant, with precision increasing when higher-layer clustering results are included.Source: §4.2 · p. 6
  • Combination pairing improves performance as it connects more clustering results across and within audio and visual extractors.The method averages MI across all possible layer combinations, and the reported trend indicates that consensus across layers contributes to correspondence clarity.Source: §4.2 · p. 6, Table 2 · p. 6
  • 88.705 ± 0.004 for SGD K-means versus 88.732 ± 0.005 for EM shows negligible performance degradation with substantially lower memory requirements.Source: §4.2 · p. 6
  • 98.970 for greedy versus 98.020 for batch greedy with (b, s) = (160, 5) shows a marginal performance gap, while performance drops sharply when s/b exceeds 0.25.The batch greedy method remains robust when the selection-to-batch ratio does not exceed the roughly 25% easy-positive ratio.Source: §4.2 · p. 7, Figure 2 · p. 7
  • Underclustering with k = {8, 16} gives high early precision, whereas overclustering with k = {64, 128} produces a slower later-stage drop despite similar final performance.The sensitivity analysis uses Kinetics-Sounds, whose ground-truth number of categories is c = 32.Source: §4.2 · p. 7, Figure 3 · p. 7

5. Large-Scale Evaluation

Across 20K, 200K, and 2M scales, ACAV datasets match or slightly exceed manually curated baselines on downstream tasks, while human evaluation confirms stronger audiovisual correspondence than random sampling.

5.1. Linear Evaluation on Downstream Tasks

  • Models pretrained on ACAV datasets achieve similar or slightly better performance than Kinetics-Sounds, VGG-Sound, and AudioSet at 20K, 200K, and 2M scales.The evaluation covers video classification on UCF101, audio classification on ESC-50, and audio-visual classification on Kinetics-Sounds.Source: §5.1 · p. 8, Figure 4 · p. 8
  • The clustering-based MI estimator generally outperforms the contrastive estimator, indicating robustness to noisy real-world audiovisual correspondences.Source: §5.1 · p. 8
  • Performance improves substantially for the 10M and 100M ACAV datasets, supporting the value of large-scale training without manual annotation or verification.Source: §5 · p. 7, §5.1 · p. 8

5.2. Human Evaluation

  • Human raters identify audiovisual correspondence in 69% of ACAV clips versus 44% of random clips, with ACAV rated slightly higher than AudioSet.The study used 400 clips from four datasets and three votes per video; interrater reliability exceeded the reported threshold for every dataset.Source: §5.2 · p. 8, Table 3 · p. 8
  • ACAV remains below VGG-Sound in human-rated correspondence while finding correspondence without VGG-Sound’s manual verification.Source: §5.2 · p. 8

6. Conclusion

The paper contributes an automatic, scalable MI-based pipeline for curating audiovisual data and releases ACAV100M, a 100M-clip open-domain dataset.

6. Conclusion

  • The work proposes an automatic and scalable data-collection pipeline for audiovisual representation learning.Source: §6 · p. 8
  • MI-based subset selection retrieves audiovisual correspondence in both artificial and practical settings.Source: §6 · p. 8
  • ACAV100M provides 100M curated clips for large-scale self-supervised video representation learning.Source: §6 · p. 8

A. On the Diversity of Concepts in Sampled Clips

Curated subsets preserve broad visual concept diversity while concentrating audio content toward clips with stronger audiovisual correspondence.

A.1. Histogram of Cluster IDs

  • Curated subsets largely follow the population’s visual cluster distribution, indicating visual concept diversity comparable to the full video collection.Source: §A.1 · p. 9
  • Audio clusters become concentrated after selection because curation favors clips with high audiovisual correspondence and filters clips with background noise or mismatched sound.Source: §A.1 · p. 9

A.2. Qualitative Analysis of Audio-Visual Clustering Results

Qualitative inspection shows diverse audiovisual concepts in ACAV100M, while the multi-layer clustering estimator remains robust unless layer weights become excessively steep.

A.2. Qualitative Analysis of Audio-Visual Clustering Results

  • The 2M curated dataset contains diverse audio concepts, including voices, object sounds, interviews, and cooking, alongside visual concepts such as animals, fire, makeup, and guitar playing.Source: §A.2 · p. 9, Figure 6 · p. 10, Figure 7 · p. 11
  • Larger clusters contain clearer, more isolated sound sources, whereas smaller clusters more often contain multiple sound sources or background noise.Source: §A.2 · p. 9
  • ACAV captures concepts not offered by some existing datasets, including phone-microphone voice recordings, crowd-accompanied performances, and shoe-unboxing videos.Source: §A.2 · p. 9

B. Weighted Summation of Layer Scores (Section 4.2)

C. Details of Linear Evaluation on Downstream Tasks (Section 5.1)

C.1. Experimental Settings

The experiments define the linear-evaluation setup and examine how clustering and greedy subset selection affect curation. They find that roughly 500 centroids perform best, while over-clustering fragments similar concepts and harms selection quality.

C.1. Experimental Settings

  • Pretraining uses 3D ResNet-50 and ResNet-50 backbones with 128-dimensional projection heads and a contrastive noise-contrastive objective.Models are trained end-to-end for 50 epochs using AdamW with warm-up and linear learning-rate decay.Source: §C.1 · p. 12
  • Linear evaluation freezes pretrained feature extractors and trains classifiers using visual features for UCF101, audio features for ESC-50, and concatenated audiovisual features for Kinetics-Sounds.Dropout is applied before the classifier, with task-specific training epochs and batch sizes.Source: §C.1 · p. 12

C.2. Impact of the Number of Centroids

  • About 500 clusters yield the best downstream performance, whereas C = 2000 consistently gives low accuracy across datasets and subset sizes.The authors attribute this to over-clustering, which scatters homogeneous concepts into small clusters and complicates clustering-based MI estimation.Source: §C.2 · p. 12

D.1. Greedy Algorithm

  • The exact greedy algorithm has O(N^2) time complexity because it reevaluates the clustering-based MI estimator over all remaining candidates at every iteration.The batch greedy approximation instead searches only within a randomly selected batch.Source: §D.1 · p. 12

D.2. Batch Greedy Subset Selection

This section evaluates batch greedy subset selection across batch sizes and selection ratios. Selection-ratio choice matters substantially more than batch size, with robustness degrading once the ratio exceeds 25%.

D.2. Batch Greedy Subset Selection

  • Table 5 reports top-1/5 linear-evaluation accuracies for video, audio, and audiovisual classification across the three downstream datasets.UCF101 and ESC-50 results are averaged across their official splits.Source: Table 5 · p. 13
  • Figure 8 reports top-1 accuracy on UCF101, ESC-50, and Kinetics-Sounds, grouped by the number of cluster centroids with 99% confidence intervals.The evaluation averages runs over the official UCF101 and ESC-50 splits.Source: Figure 8 · p. 13
  • 93.9%, 94.3% and 94.6% precision are obtained for b = 40, 80, 160 at the shared ratio s/b = 12.5%.The performance gap between batch sizes is small under this fixed ratio.Source: §D.2 · p. 13
  • Precision drops sharply when s/b exceeds 25%, regardless of batch size.The authors relate this boundary to the 25% easy-positive ratio in the balanced positive-negative construction.Source: §D.2 · p. 13

E. Details of Automatic Dataset Curation

The curation details cover dataset scales, NCE-based and clustering-based MI estimation, and the greedy procedures used to construct subsets. The clustering pipeline varies centroid counts, while the implementation fixes batch-greedy parameters for scalability.

E. Details of Automatic Dataset Curation

  • Datasets are constructed at 20K, 200K, and 2M scales, with an additional 10M version generated using the clustering-based approach.The 10M dataset is produced only with C = 500 for computational reasons.Source: E. Details of Automatic Dataset Curation · p. 13, §E.2 · p. 14
  • The greedy algorithm incrementally adds the candidate maximizing F until the target subset size M is reached.F is the clustering-based MI estimator, and the output is a subset X with |X| = M.Source: E. Details of Automatic Dataset Curation · p. 13
  • Batch greedy repeats subset updates by sampling a batch and selecting s elements from it until the target subset size is reached.Its inputs include the initial dataset, clustering-based MI estimator, target size M, batch size b, and selection size s.Source: E. Details of Automatic Dataset Curation · p. 14

E.1. NCE-Based MI Estimation

  • The NCE-based estimator trains projection heads on 100M sampled clips for three epochs using 1,024-example batches.The clustering-based estimator trains SGD K-means centroids for 100 epochs and uses b = 10,000, s = 500, and s/b = 0.05 in batch greedy.Source: §E.1 · p. 14, §E.2 · p. 14

E.2. Clustering-Based MI Estimation

  • Clustering-based estimation evaluates C ∈ {100, 200, 500, 1000, 2000} for each dataset scale, except the 10M dataset, which uses C = 500.The clustering implementation uses a 100K mini-batch for SGD K-means.Source: §E.2 · p. 14

F. Human Evaluation Interface (Section 5.2)

The human-evaluation interface presents annotators with one 10-second clip and collects a binary judgment of audio-visual correspondence. Instructions define correspondence by whether the sound source is visible or inferable from visual context.

F. Human Evaluation Interface (Section 5.2)

Related papers

Loading 2101.10803v3…