Source-linked AI summary
CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016
Yuanjun Xiong, Limin Wang, Zhe Wang, Bowen Zhang, Hang Song, Wei Li, Dahua Lin, Yu Qiao, Luc Van Gool, Xiaoou Tang
TL;DR
Untrimmed-video action recognition must handle long videos containing substantial irrelevant content, unlike conventional short-clip recognition. The paper extends temporal segment networks with deep visual models, advanced aggregation, and complementary audio CNNs, achieving mAP 93.23% on the ActivityNet testing set.
Problem
Untrimmed-video action recognition is needed for real-world video but is difficult because a significant fraction of each video may be irrelevant to the action.
Method
The method combines temporal-segment visual modeling with ResNet or Inception V3 predictors, top-k or attention-weighted aggregation, and CNN-based spectrogram audio modeling.
Results
mAP 93.23% on the testing set was achieved by the combined visual and acoustic models.
Takeaways & Limitations
The combined visual-acoustic system achieved high ActivityNet classification accuracy while evaluating only 1 frame per second, around 4% of the input frames.
Abstract
from arXiv · showhide
This paper presents the method that underlies our submission to the untrimmed video classification task of ActivityNet Challenge 2016. We follow the basic pipeline of temporal segment networks and further raise the performance via a number of other techniques. Specifically, we use the latest deep model architecture, e.g., ResNet and Inception V3, and introduce new aggregation schemes (top-k and attention-weighted pooling). Additionally, we incorporate the audio as a complementary channel, extracting relevant information via a CNN applied to the spectrograms. With these techniques, we derive an ensemble of deep models, which, together, attains a high classification accuracy (mAP $93.23\%$) on the testing set and secured the first place in the challenge.
1. Introduction
The paper addresses untrimmed-video action recognition, where long videos reduce annotation effort but contain substantial irrelevant content. It builds an integrated visual-acoustic approach for ActivityNet and reports high testing accuracy using benchmark training data plus ImageNet initialization.
- Motivation: Untrimmed-video recognition is more relevant to real-world demands than short-clip analysis but includes a significant fraction of irrelevant frames.Short-clip preparation is laborious, whereas untrimmed videos reduce manual annotation effort while creating a recognition challenge.
- Contribution: The approach extends temporal segment networks with newer deep architectures, aggregation techniques, and an Audio CNN for complementary spectrogram features.The integrated system combines long-range temporal modeling with visual and acoustic analysis.
- Result: mAP 93.23% on the testing set was achieved by combining visual and acoustic models.The reported training and testing procedure used ActivityNet data, with CNNs pretrained on ILSVRC12 only for initialization.
2. Our Approach
The approach combines snippet-based visual modeling with acoustic modeling for video-level classification. It uses deep two-stream predictors, advanced score aggregation, and audio representations based on MFCCs and multiscale spectrogram CNNs.
- Our Approach: Video classification combines visual and acoustic components, with visual analysis providing the primary signal and audio supplying complementary information.The final fusion combines the two modeling systems for ActivityNet classification.
- Visual Analysis System: Visual analysis samples snippets, applies very deep two-stream CNNs, and aggregates snippet predictions using strategies including top-k and attention-weighted pooling.The predictor follows temporal segment networks and uses architectures such as ResNet and Inception V3.
- Visual Analysis System: The visual predictor is trained with multiple snippets and augmentation techniques, while advanced aggregation strategies produce complementary models for the final ensemble.Scale jittering and stronger dropout are applied during training, and aggregation can be used in both training and testing.
- Acoustic Analysis System: The acoustic system combines MFCC-based SVM features with CNNs trained directly on grayscale spectrograms.The Audio CNN stacks spectrograms with varying window sizes to model multiple time scales.
- Fusion: Fusion models combine visual and acoustic scores, with the final testing ensemble including all components trained on the union of training and validation data.Table 5 evaluates visual-plus-audio fusion and reports the final ensemble on ActivityNet v1.3.
3. Experiments
The experiments evaluate ActivityNet performance across training strategies, network architectures, component contributions, and audio–visual fusion. The final ensemble achieves 93.2% mAP on the testing set, while a single visual model reaches 91.2%.
- The models are trained on ActivityNet v1.3, using 10,024 training videos across 200 activity classes and evaluating validation performance on 4,926 videos.
- The experiments compare temporal segment networks using BN-Inception, Inception V3, and ResNet architectures.
- The study separately evaluates training strategies, appearance and motion CNN contributions, and the two acoustic-system components.These comparisons are organized across Tables 2, 3, and 4.
- 93.2% mAP is achieved by the final ensemble on the ActivityNet testing set.The ensemble fuses the visual and audio analysis systems.
- 91.2% mAP is achieved by a single combination of one appearance CNN and one motion CNN.The paper identifies this single-model setting as potentially better suited to industrial applications.
4. Conclusions
The paper concludes that combining visual and acoustic analysis produces strong untrimmed-video classification results. It also reports high accuracy while evaluating only 1 frame per second, or about 4% of input frames, supporting practical deployment.
- Combining visual and acoustic analysis achieves strong results for classifying temporally untrimmed videos.
- 1 frame per second, equivalent to about 4% of input frames, is sufficient for the reported high accuracy.The paper identifies this low evaluation rate as important for industrial applications.