Source-linked AI summary
Listen to Look: Action Recognition by Previewing Audio
Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, Lorenzo Torresani
TL;DR
Dense processing of long untrimmed videos is costly because visual content contains short- and long-term redundancy. The paper previews video with audio, distills clip descriptors from image-audio pairs, and uses attention-based skimming to select useful moments. Across four datasets, it reports state-of-the-art action recognition with a favorable accuracy-speed trade-off using few selected frames or clips.
Problem
Long untrimmed videos make dense clip-level action recognition computationally impractical despite substantial short- and long-term temporal redundancy.
Method
IMGAUD2VID distills clip-level video descriptors from a single frame and accompanying audio, while IMGAUD-SKIMMING uses attention-based LSTM selection for efficient video-level recognition.
Results
The framework achieves state-of-the-art action-recognition results across four datasets with a favorable accuracy-speed trade-off using few selected frames or clips.
Takeaways & Limitations
Audio can serve as a fast preview for reducing both clip-level and video-level redundancy in long untrimmed-video action recognition.
Abstract
from arXiv · showhide
In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical. We propose a framework for efficient action recognition in untrimmed video that uses audio as a preview mechanism to eliminate both short-term and long-term visual redundancies. First, we devise an ImgAud2Vid framework that hallucinates clip-level features by distilling from lighter modalities---a single frame and its accompanying audio---reducing short-term temporal redundancy for efficient clip-level recognition. Second, building on ImgAud2Vid, we further propose ImgAud-Skimming, an attention-based long short-term memory network that iteratively selects useful moments in untrimmed videos, reducing long-term temporal redundancy for efficient video-level recognition. Extensive experiments on four action recognition datasets demonstrate that our method achieves the state-of-the-art in terms of both recognition accuracy and speed.
1. Introduction
Long untrimmed videos make dense clip-level action recognition computationally impractical because they contain substantial short- and long-term temporal redundancy. The paper uses audio with single frames to preview video content, distill clip features, and select key moments efficiently.
- Long untrimmed videos make dense clip-level classification computationally impractical for videos lasting minutes or hours.
- Video contains redundancy both across repeated long-term events and within clips, where adjacent frames often show similar appearance despite motion.
- IMGAUD2VID distills an expensive video descriptor from one frame and accompanying audio, reducing clip-level redundancy.The frame captures appearance, while audio contributes dynamic information.
- IMGAUD-SKIMMING uses attention-based LSTM scanning to select key moments and skip irrelevant or uninformative segments for video-level recognition.
- The framework targets accurate and efficient recognition across Kinetics-Sounds, Mini-Sports1M, ActivityNet, and UCF-101, achieving state-of-the-art results with few selected frames or clips.
2. Related Work
Prior action-recognition work emphasizes powerful spatiotemporal models, while efficient recognition in long untrimmed videos remains less addressed. This paper instead uses audio-aided selection and lightweight representations to reduce computation without requiring strong temporal supervision or complex reinforcement learning.
- Action Recognition: Recent action-recognition research models spatiotemporal information with deep networks, recurrent structures, ranking, pooling, and long-term temporal relations.
- Efficient Action Recognition: Many prior approaches prioritize recognition accuracy without accounting for computation, whereas this work targets efficient recognition in long untrimmed videos.
- Action Proposal and Localization: Action-proposal methods resemble preview-based selection but often cost several orders of magnitude more because they generate and process many clip proposals.
- Selection of Frames or Clips: Unlike frame-selection methods using strong temporal annotations or reinforcement learning, the proposed skimming mechanism uses weakly labeled video and avoids complex policy gradients.
- Video Summarization: The method differs from video summarization by selecting features useful for activity recognition rather than keyframes intended to convey a video's gist to viewers.
3. Approach
The approach uses audio with a single frame to replace expensive clip processing, then applies attention-based recurrent selection to reduce redundancy across untrimmed videos.
- Approach: The framework targets two inefficiencies: visual similarity within short clips and repeated, mostly uninformative clips across long videos.It addresses both clip-level and video-level temporal redundancy through audio-based previews.
- Clip-Level Preview: IMGAUD2VID distills a video-clip classifier into an image-audio model that estimates clip descriptors from one frame and its accompanying audio.The student uses a visual stream for the starting frame and an audio stream for the spectrogram, while the teacher processes the full clip.
- Video-Level Preview: IMGAUD-SKIMMING uses an attention-based LSTM to select a small subset of T image-audio pairs from N video pairs, with T ≪ N.The LSTM processes indexed image and audio features with recurrent state, while query vectors attend to feature sequences to select useful moments.
- Clip-Level Preview: The student is trained against the teacher’s soft class predictions and clip descriptors using a combined KL-divergence and L1 objective.The teacher remains fixed while the image, audio, and fusion components are trained, followed by target-dataset fine-tuning.
- Video-Level Preview: A differentiable soft-indexing mechanism replaces nondifferentiable frame indexing, avoiding the policy-gradient approximations used in prior work.The selected indexed features are aggregated and passed through the shared fusion network for subsequent action prediction.
- Video-Level Preview: The querying mechanism transforms indexing features into keys, compares them with modality-specific query vectors, and normalizes scores into attention weights.Separate image and audio queries can emphasize different moments before their information is fused for the next recurrent step.
4. Experiments
Experiments across four datasets evaluate efficient clip-level and video-level action recognition. The framework achieves competitive or improved accuracy while reducing computational cost through image-audio distillation and iterative moment selection.
- Experimental setup: Four datasets evaluate clip-level recognition on Kinetics-Sounds, UCF-101, and ActivityNet, and video-level recognition on ActivityNet and Mini-Sports1M.Kinetics-Sounds and UCF-101 contain short trimmed videos, while ActivityNet and Mini-Sports1M provide long untrimmed-video settings.
- Clip-level action recognition: IMGAUD2VID achieves accuracy comparable to the R(2+1)D-18 clip-based teacher model at much lower computational cost.Image-only and audio-only distillation produce lower accuracy, indicating that the two modalities provide complementary information.
- Clip-level action recognition: Dynamic scenes with informative audio, including grinding meat, jumpstyle dancing, and welding, are among the clips where IMGAUD2VID best matches the ground-truth video descriptor.The comparison uses L1 distance between hallucinated and ground-truth video descriptors.
- Video-level action recognition: IMGAUD-SKIMMING selects a small subset of moments from long videos, while sparse indexing uses about one-fifth of the computation cost of dense baselines.The method is evaluated against random, uniform, fixed-position, dense, SCSAMPLER, LSTM, and non-recurrent baselines.
- Efficiency analysis: The first three recurrent skimming steps provide sufficient recognition cues, allowing early stopping with negligible accuracy loss.This observation is reported on ActivityNet, with a similar observation noted for Mini-Sports1M.
- Video-level action recognition: The method consistently outperforms existing approaches and balances speed and accuracy, including comparable performance to dense R(2+1)D-152 recognition while being 20× faster with image-audio indexing.Using R(2+1)D-152 features for both indexing and recognition is 10× faster than the dense approach.
5. Conclusion
The paper achieves accurate and efficient action recognition in long untrimmed videos by using audio as a preview, combining lightweight clip recognition with iterative selection of useful image-audio pairs.
- IMGAUD2VID replaces expensive clip-based models with a lightweight image-audio model for clip-level action recognition.
- IMGAUD-SKIMMING iteratively selects useful image-audio pairs to support efficient video-level action recognition.
- The approach achieves state-of-the-art video action recognition using few selected frames or clips while balancing speed and accuracy.