PaperlayerView exact revision on arXiv

Source-linked AI summary

Audiovisual SlowFast Networks for Video Recognition

Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, Christoph Feichtenhofer

arXiv:2001.08740v2cs.CV

TL;DR

Video understanding often ignores audio or combines modalities only late, despite audio helping actions that are subtle or difficult to distinguish visually. AVSlowFast hierarchically integrates Slow and Fast visual pathways with a Faster Audio pathway, using DropPathway and audiovisual synchronization for joint learning. The approach reports state-of-the-art performance across six datasets, with consistent gains at relatively small computational cost.

  • Problem

    Most video-analysis models ignore audio or use late fusion, while audio may be irrelevant, uninformative, or learn at a different speed from visual pathways.

  • Method

    AVSlowFast hierarchically fuses a Faster Audio pathway with Slow and Fast visual pathways, using DropPathway and multilayer audiovisual synchronization for joint learning.

  • Results

    The model achieves state-of-the-art performance across six video classification, detection, and self-supervised learning datasets, with consistent improvement from integrating audio.

  • Takeaways & Limitations

    AVSlowFast provides a unified audiovisual backbone for video action classification, detection, and self-supervised representation learning.

Abstract

from arXiv · show

We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception. AVSlowFast has Slow and Fast visual pathways that are deeply integrated with a Faster Audio pathway to model vision and sound in a unified representation. We fuse audio and visual features at multiple layers, enabling audio to contribute to the formation of hierarchical audiovisual concepts. To overcome training difficulties that arise from different learning dynamics for audio and visual modalities, we introduce DropPathway, which randomly drops the Audio pathway during training as an effective regularization technique. Inspired by prior studies in neuroscience, we perform hierarchical audiovisual synchronization to learn joint audiovisual features. We report state-of-the-art results on six video action classification and detection datasets, perform detailed ablation studies, and show the generalization of AVSlowFast to learn self-supervised audiovisual features. Code will be made available at: https://github.com/facebookresearch/SlowFast.

1. Introduction

AVSlowFast addresses limited audiovisual integration in video understanding by hierarchically combining audio and visual pathways. It introduces training and synchronization strategies, and reports consistent gains across recognition, detection, and self-supervised settings.

  • Motivation: Audio can aid obvious, visually subtle, and closely related actions where visual information is insufficient or ambiguous.Examples include playing saxophone, whistling, and distinguishing closing from slamming a door.
  • Challenges: Audiovisual learning is challenged by irrelevant or uninformative audio and mismatched learning speeds between audio and visual pathways.Audio may be background music unrelated to the action, while audio and visual pathways can train at different rates.
  • Approach: AVSlowFast hierarchically fuses a Faster Audio pathway with Slow and Fast visual pathways to learn integrated audiovisual features rather than relying on late fusion.The Audio pathway can be lightweight, using less than 20% of computation, but requires careful design and training.
  • Results: +2.9/+4.3/+2.3 top-1 accuracy is reported for verb/noun/action recognition on EPIC-Kitchens at 20% of overall compute.The paper also reports consistent improvement across standard datasets and evaluates self-supervised audiovisual representation learning.
  • Approach: DropPathway randomly drops the Audio pathway during training to regularize joint learning despite incompatible modality dynamics.The technique tunes the pace of learning and supports hierarchical fusion across modalities.
  • Approach: Audiovisual synchronization is applied at multiple layers to learn features that generalize across modalities.The design is inspired by proposed audiovisual mirror neurons responding to auditory or visual evidence of an action.

2. Related Work

Related work spans visual video-recognition architectures, late-fusion audiovisual activity recognition, broader multimodal learning, and self-supervised or task-specific audiovisual applications.

  • Video recognition: Video recognition has explored two-stream networks, 3D ConvNets, and SlowFast pathways to capture spatial and temporal information.These approaches process RGB and optical flow, extend 2D networks spatiotemporally, or use different temporal frequencies.
  • Audiovisual activity recognition: Audiovisual activity recognition has largely used late fusion, processing audio and visual inputs separately before combining features or class scores.The passage describes concatenating output features or averaging final class scores across modalities.
  • Multimodal learning: Multimodal learning also includes combinations such as vision-language, vision-locomotion, and vision with physiological data.
  • Other audiovisual tasks: Audiovisual research covers self-supervised representation learning, speech recognition, lip reading, biometric matching, localization, source separation, and question answering.

3. Audiovisual SlowFast Networks

AVSlowFast extends SlowFast with a higher-rate Audio pathway and multimodal lateral connections, using hierarchical fusion designs and training strategies for differing modality dynamics.

  • Architecture: AVSlowFast fuses audio and visual features at multiple stages so audio contributes to visual concepts from intermediate features to high-level semantics.The architecture adds an Audio pathway to Slow and Fast visual pathways based on neuroscience-inspired multimodal fusion.
  • SlowFast pathways: The Slow pathway samples sparsely for semantic content, while the Fast pathway samples more densely and preserves temporal resolution for motion.The Fast pathway uses αF times the frame rate and βF times the Slow pathway channels, with αF > 1 and βF < 1.
  • Audio pathway: The Audio pathway uses log-mel-spectrograms and an even finer temporal structure than the Slow and Fast pathways.Its temporal ratio αA can be much larger than αF, such as 32 versus 8.
  • Instantiations: The illustrated AVSlowFast instantiation uses αF = 8, αA = 32, βF = 1/8, βA = 1/2, τ = 16, and a ResNet-50 backbone.
  • Lateral connections: Audio-visual lateral connections begin at intermediate levels because very low-level visual features may lack particular sound signatures.
  • Fusion connections: The A→FS fusion design relaxes temporal alignment requirements by injecting audio into the temporally downsampled SlowFast output.A→F→S instead fuses audio into Fast before Slow, while Audiovisual Nonlocal uses audio to select important visual features.
  • Joint audiovisual training: Audio-only training reaches overfitting after about 70 epochs, approximately one-third of the visual model’s training duration.This learning-speed mismatch motivates training strategies for joint audiovisual learning.

4. Experiments: Action Classification

AVSlowFast consistently improves strong visual baselines across action-classification datasets, with gains varying by dataset and class while adding relatively little computation. Ablations indicate that hierarchical fusion, DropPathway, and audiovisual synchronization are important design choices.

  • EPIC-Kitchens: AVSlowFast improves SlowFast by +2.9 / +4.3 / +2.3 top-1 accuracy for verb / noun / action on EPIC-Kitchens.
  • Kinetics: AVSlowFast improves Kinetics top-1 accuracy by 1.4% with R50 and 0.9% with R101 at modest computational cost.The audio stream adds approximately 10%–20% of overall computation, while increasing depth from R50 to R101 increases computation by 194%.
  • Kinetics: On Kinetics-Sounds, AVSlowFast gains +3.2%/+2.3% over SlowFast for R50/R101, respectively, on 34 classes manifested visually and aurally.
  • Charades: Audio improves Charades recognition by +1.2% over a strong SlowFast baseline, and AVSlowFast achieves state-of-the-art performance with Kinetics-400 pretraining.
  • Ablation Studies: Accuracy rises from 75.4% to 77.0% as fusion expands from one pool5 connection to three res3,4 plus pool5 connections, but declines with an additional res2 connection.The results support fusing intermediate-level features through the top of the network rather than starting at the earliest level.
  • Ablation Studies: DropPathway is necessary for joint training: without it, accuracy falls to 75.2% versus 75.6% for visual-only models, while the full gain reaches 77.0%.Drop rates of 0.5 or 0.8 are required to slow the faster-learning Audio pathway.

5. Experiments: AVA Action Detection

AVSlowFast applies audiovisual modeling to AVA action detection, where actions must be both localized and recognized, and reports consistent gains over visual-only SlowFast with low computational overhead.

  • AVA evaluates spatiotemporal localization and recognition over 60 action classes using frame-level mAP at IoU 0.5.
  • The detector feeds sampled RGB frames through SlowFast pathways alongside a corresponding log-mel-spectrogram through the Audio pathway.
  • ∼1.2% margin: AVSlowFast outperforms SlowFast with only a 2% FLOPs increase for the R50 backbone.
  • AVSlowFast achieves state-of-the-art AVA performance under Kinetics-400 pretraining, with gains varying substantially across action classes.

6. Experiments: Self-supervised Learning

The paper tests whether AVSlowFast generalizes to self-supervised audiovisual representation learning using existing pretext tasks rather than introducing a new one.

  • AVSlowFast uses audiovisual synchronization and image rotation prediction as self-supervised pretraining tasks, then evaluates frozen representations on UCF101 and HMDB51.
  • +23.4 and +12.7 top-1 accuracy: the smallest AVSlowFast R50 exceeds previous-best CBT on UCF101 and HMDB51, respectively.
  • The experiment is intended to assess architecture generalization with off-the-shelf self-supervised tasks.

7. Conclusion

The paper presents AVSlowFast as an integrated audiovisual architecture and reports state-of-the-art results across video classification, detection, and self-supervised learning tasks on six datasets.

  • AVSlowFast is presented as a unified audiovisual backbone for video understanding.
  • State-of-the-art performance is reported on six datasets spanning video action classification, detection, and self-supervised learning tasks.

A.1. Results: Self-supervised Learning

Additional self-supervised experiments evaluate AVSlowFast with linear classification, task ablations, compute-sensitive temporal sampling, and full-layer fine-tuning.

  • The linear protocol freezes the pretrained network and trains only a classifier after AVS and rotation-prediction pretraining on Kinetics-400.
  • +23.4% for UCF101 and +14.6% for HMDB51: AVSlowFast features outperform CBT under the linear classification comparison.
  • On UCF101, combining AVS and rotation prediction performs best, while HMDB51 variants perform similarly and audio appears less important.
  • Performance increases with higher temporal-resolution clips and larger FLOPs even when model parameters remain unchanged.
  • Fine-tuning all layers is also evaluated, although the authors consider it suboptimal for isolating self-supervised representation quality.

A.2. Results: Audio-only Classification

The Audio-only AVSlowFast model is evaluated against existing audio models on Kinetics datasets, with results reported alongside AVA per-class average precision comparisons.

  • +3.3% top-1 accuracy on Kinetics-600 and +3.2% on Kinetics-400 are reported over existing audio methods.
  • 27.8 mAP for AVSlowFast exceeds 26.3 mAP for its SlowFast counterpart on AVA.
  • Table A.3 reports results for Audio-only models, including a VGG* result taken from an iTXN ActivityNet challenge submission.

A.3. Results: Classification & Detection Analysis

AVSlowFast improves classification over strong visual baselines in the reported analyses, with gains concentrated in sound-informative classes and losses where sound is absent or noisy.

  • 77.0% versus 75.6% for 4×16, R50 on Kinetics shows higher AVSlowFast classification accuracy than SlowFast.
  • Audio benefits Kinetics classes including dancing macarena (+24.5%), whistling (+24.0%), beatboxing (+20.4%), salsa dancing (+19.1%), and singing (+16.0%).
  • On AVA, the largest absolute gains occur for swim, dance, shoot, hit (an object), and cut.
  • Push (an object) gains 3.2× and throw gains 2.0× in relative terms on AVA.
  • Performance losses arise for visually oriented or noisy classes such as watch (e.g., TV), read, eat, and work on a computer.The authors suggest explicitly modeling foreground and background sound as a future direction.

A.8. Details: Self-supervised Evaluation

The self-supervised evaluation trains audiovisual models on Kinetics-400 and applies curriculum learning for audiovisual synchronization, alongside linear-classification protocols on UCF and HMDB.

  • Self-supervised pretraining uses Kinetics-400 for 120k iterations across 16 machines with per-machine batch size 64.
  • The self-supervised learning rate is decayed 10× at 80k, 100k, and 110k iterations after a 10k-iteration warm-up.
  • Audiovisual synchronization uses curriculum learning that begins with easy negatives.
  • Linear-classification experiments on UCF and HMDB train for 320k iterations with a half-period cosine decay schedule.
  • Training all layers uses 80k iterations for UCF and 30k for HMDB with batch sizes of 16.

B. Details: Kinetics-Sound dataset

The dataset details specify 32 classes retained in the current Kinetics-400 version rather than the original 34 classes selected from an earlier dataset version.

  • The evaluation uses 32 classes that remain in the current Kinetics-400 dataset.
  • The original selection contained 34 classes based on an earlier version of Kinetics, with some classes subsequently removed.
Loading 2001.08740v2…