PaperlayerView exact revision on arXiv

Source-linked AI summary

Listen to Look: Action Recognition by Previewing Audio

Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, Lorenzo Torresani

arXiv:1912.04487v3cs.CVcs.LGcs.SDeess.AS

TL;DR

Dense processing of long untrimmed videos is costly because visual content contains short- and long-term redundancy. The paper previews video with audio, distills clip descriptors from image-audio pairs, and uses attention-based skimming to select useful moments. Across four datasets, it reports state-of-the-art action recognition with a favorable accuracy-speed trade-off using few selected frames or clips.

  • Problem

    Long untrimmed videos make dense clip-level action recognition computationally impractical despite substantial short- and long-term temporal redundancy.

    Source: §1 · p. 1

  • Method

    IMGAUD2VID distills clip-level video descriptors from a single frame and accompanying audio, while IMGAUD-SKIMMING uses attention-based LSTM selection for efficient video-level recognition.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    The framework achieves state-of-the-art action-recognition results across four datasets with a favorable accuracy-speed trade-off using few selected frames or clips.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Audio can serve as a fast preview for reducing both clip-level and video-level redundancy in long untrimmed-video action recognition.

    Source: §1 · p. 2

Abstract

from arXiv · show

In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical. We propose a framework for efficient action recognition in untrimmed video that uses audio as a preview mechanism to eliminate both short-term and long-term visual redundancies. First, we devise an ImgAud2Vid framework that hallucinates clip-level features by distilling from lighter modalities---a single frame and its accompanying audio---reducing short-term temporal redundancy for efficient clip-level recognition. Second, building on ImgAud2Vid, we further propose ImgAud-Skimming, an attention-based long short-term memory network that iteratively selects useful moments in untrimmed videos, reducing long-term temporal redundancy for efficient video-level recognition. Extensive experiments on four action recognition datasets demonstrate that our method achieves the state-of-the-art in terms of both recognition accuracy and speed.

1. Introduction

Long untrimmed videos make dense clip-level action recognition computationally impractical because they contain substantial short- and long-term temporal redundancy. The paper uses audio with single frames to preview video content, distill clip features, and select key moments efficiently.

  • Long untrimmed videos make dense clip-level classification computationally impractical for videos lasting minutes or hours.Source: §1 · p. 1
  • Video contains redundancy both across repeated long-term events and within clips, where adjacent frames often show similar appearance despite motion.Source: §1 · pp. 1–2
  • IMGAUD2VID distills an expensive video descriptor from one frame and accompanying audio, reducing clip-level redundancy.The frame captures appearance, while audio contributes dynamic information.Source: §1 · p. 2
  • IMGAUD-SKIMMING uses attention-based LSTM scanning to select key moments and skip irrelevant or uninformative segments for video-level recognition.Source: §1 · p. 2
  • The framework targets accurate and efficient recognition across Kinetics-Sounds, Mini-Sports1M, ActivityNet, and UCF-101, achieving state-of-the-art results with few selected frames or clips.Source: §1 · p. 2

2. Related Work

Prior action-recognition work emphasizes powerful spatiotemporal models, while efficient recognition in long untrimmed videos remains less addressed. This paper instead uses audio-aided selection and lightweight representations to reduce computation without requiring strong temporal supervision or complex reinforcement learning.

Action Recognition

  • Recent action-recognition research models spatiotemporal information with deep networks, recurrent structures, ranking, pooling, and long-term temporal relations.Source: §2 · p. 2

Efficient Action Recognition

  • Many prior approaches prioritize recognition accuracy without accounting for computation, whereas this work targets efficient recognition in long untrimmed videos.Source: §2 · p. 2

Action Proposal and Localization

  • Action-proposal methods resemble preview-based selection but often cost several orders of magnitude more because they generate and process many clip proposals.Source: §2 · p. 2

Selection of Frames or Clips

  • Unlike frame-selection methods using strong temporal annotations or reinforcement learning, the proposed skimming mechanism uses weakly labeled video and avoids complex policy gradients.Source: §2 · p. 3

Video Summarization

  • The method differs from video summarization by selecting features useful for activity recognition rather than keyframes intended to convey a video's gist to viewers.Source: §2 · p. 3

3. Approach

The approach uses audio with a single frame to replace expensive clip processing, then applies attention-based recurrent selection to reduce redundancy across untrimmed videos.

Approach

  • The framework targets two inefficiencies: visual similarity within short clips and repeated, mostly uninformative clips across long videos.It addresses both clip-level and video-level temporal redundancy through audio-based previews.Source: §3.1 · p. 3, §3 · p. 3

Clip-Level Preview

  • IMGAUD2VID distills a video-clip classifier into an image-audio model that estimates clip descriptors from one frame and its accompanying audio.The student uses a visual stream for the starting frame and an audio stream for the spectrogram, while the teacher processes the full clip.Source: §3.1 · p. 3, Figure 2 · p. 4, §3.2 · p. 4

Video-Level Preview

  • IMGAUD-SKIMMING uses an attention-based LSTM to select a small subset of T image-audio pairs from N video pairs, with T ≪ N.The LSTM processes indexed image and audio features with recurrent state, while query vectors attend to feature sequences to select useful moments.Source: §3.1 · p. 3, §3.3 · p. 4

Clip-Level Preview

  • The student is trained against the teacher’s soft class predictions and clip descriptors using a combined KL-divergence and L1 objective.The teacher remains fixed while the image, audio, and fusion components are trained, followed by target-dataset fine-tuning.Source: §3.2 · p. 4

Video-Level Preview

  • A differentiable soft-indexing mechanism replaces nondifferentiable frame indexing, avoiding the policy-gradient approximations used in prior work.The selected indexed features are aggregated and passed through the shared fusion network for subsequent action prediction.Source: §3.3 · p. 4
  • The querying mechanism transforms indexing features into keys, compares them with modality-specific query vectors, and normalizes scores into attention weights.Separate image and audio queries can emphasize different moments before their information is fused for the next recurrent step.Source: §3.3 · p. 4

4. Experiments

Experiments across four datasets evaluate efficient clip-level and video-level action recognition. The framework achieves competitive or improved accuracy while reducing computational cost through image-audio distillation and iterative moment selection.

Experimental setup

  • Four datasets evaluate clip-level recognition on Kinetics-Sounds, UCF-101, and ActivityNet, and video-level recognition on ActivityNet and Mini-Sports1M.Kinetics-Sounds and UCF-101 contain short trimmed videos, while ActivityNet and Mini-Sports1M provide long untrimmed-video settings.Source: §4 · pp. 5–6

Clip-level action recognition

  • IMGAUD2VID achieves accuracy comparable to the R(2+1)D-18 clip-based teacher model at much lower computational cost.Image-only and audio-only distillation produce lower accuracy, indicating that the two modalities provide complementary information.Source: §4.1 · p. 6
  • Dynamic scenes with informative audio, including grinding meat, jumpstyle dancing, and welding, are among the clips where IMGAUD2VID best matches the ground-truth video descriptor.The comparison uses L1 distance between hallucinated and ground-truth video descriptors.Source: §4.1 · p. 6

Video-level action recognition

  • IMGAUD-SKIMMING selects a small subset of moments from long videos, while sparse indexing uses about one-fifth of the computation cost of dense baselines.The method is evaluated against random, uniform, fixed-position, dense, SCSAMPLER, LSTM, and non-recurrent baselines.Source: Table 1 · p. 7, §4.2 · p. 7

Efficiency analysis

  • The first three recurrent skimming steps provide sufficient recognition cues, allowing early stopping with negligible accuracy loss.This observation is reported on ActivityNet, with a similar observation noted for Mini-Sports1M.Source: §4.2 · pp. 6–7

Video-level action recognition

  • The method consistently outperforms existing approaches and balances speed and accuracy, including comparable performance to dense R(2+1)D-152 recognition while being 20× faster with image-audio indexing.Using R(2+1)D-152 features for both indexing and recognition is 10× faster than the dense approach.Source: §4.2 · p. 8

5. Conclusion

The paper achieves accurate and efficient action recognition in long untrimmed videos by using audio as a preview, combining lightweight clip recognition with iterative selection of useful image-audio pairs.

  • IMGAUD2VID replaces expensive clip-based models with a lightweight image-audio model for clip-level action recognition.Source: §5 · p. 8
  • IMGAUD-SKIMMING iteratively selects useful image-audio pairs to support efficient video-level action recognition.Source: §5 · p. 8
  • The approach achieves state-of-the-art video action recognition using few selected frames or clips while balancing speed and accuracy.Source: §5 · p. 8

Related papers

Loading 1912.04487v3…