Source-linked AI summary
AdaScan: Adaptive Scan Pooling in Deep Convolutional Neural Networks for Human Action Recognition in Videos
Amlan Kar, Nishant Rai, Karan Sikka, Gaurav Sharma
TL;DR
Video action recognition needs pooling methods that handle irrelevant frames and noisy real-world videos. AdaScan dynamically weights and pools informative frames during a single scan, consistently outperforming pooling baselines and achieving near state-of-the-art results with complementary representations.
Problem
Existing frame-pooling methods treat all frames equally and are not robust to irrelevant action frames, while video indexing requires fast, noise-robust action recognition.
Method
AdaScan augments a deep CNN with adaptive pooling that predicts each frame’s importance and recursively performs weighted mean pooling to produce a video descriptor.
Results
AdaScan consistently outperforms relevant pooling baselines across two benchmark datasets and achieves near state-of-the-art results when combined with complementary video representations.
Takeaways & Limitations
Adaptive temporal pooling can retain discriminative frames while discarding non-informative ones, improving action-recognition performance across RGB and optical-flow networks.
Takeaways & Limitations
The evaluation uses supervised classification with videos represented as RGB images or stacks of neighboring optical-flow images.
Abstract
from arXiv · showhide
We propose a novel method for temporally pooling frames in a video for the task of human action recognition. The method is motivated by the observation that there are only a small number of frames which, together, contain sufficient information to discriminate an action class present in a video, from the rest. The proposed method learns to pool such discriminative and informative frames, while discarding a majority of the non-informative frames in a single temporal scan of the video. Our algorithm does so by continuously predicting the discriminative importance of each video frame and subsequently pooling them in a deep learning framework. We show the effectiveness of our proposed pooling method on standard benchmarks where it consistently improves on baseline pooling methods, with both RGB and optical flow based Convolutional networks. Further, in combination with complementary video representations, we show results that are competitive with respect to the state-of-the-art results on two challenging and publicly available benchmark datasets.
1. Introduction
AdaScan addresses noisy video action recognition by learning to identify and pool informative frames rather than treating every frame equally. It is designed as an end-to-end, inductive method and is reported to outperform pooling baselines across two benchmark datasets.
- Internet video growth makes fast, trainable, and noise-robust human action recognition important for searching and indexing.
- Mean and max pooling treat all frames equally, so irrelevant frames can reduce action-recognition performance.
- MIL alternates between spotting relevant frames and relearning the model, but can overfit and does not model joint contributions from multiple frames.
- AdaScan identifies informative frames, pools them while discarding others, and trains jointly with image representations for discriminative classification.
- Across two public benchmarks, AdaScan consistently outperforms relevant pooling baselines and reaches state-of-the-art performance with complementary video representations.
2. Related Work
Prior video-action methods used orderless handcrafted representations, latent temporal models, recurrent networks, or scoring-based frame selection. AdaScan instead predicts frame importance directly and pools frames in a single scan without an inference problem.
- Bag-of-Words methods encode local spatio-temporal features, including HoG and HoF, into dictionary-based histograms.
- Latent structured SVM approaches model activities with temporal structure by inferring discriminative frames or segments while learning model parameters.
- Li et al. score and pool discriminative frames, whereas AdaScan predicts frame importance explicitly and pools frames in a single scan without solving an inference problem.
- Two-stream CNNs use RGB and stacked optical-flow inputs, while later approaches add temporal pooling or LSTMs to capture video dynamics.
- Some earlier pooling approaches require a non-trivial ranking formulation for pooling each vector.
3. Proposed Approach
AdaScan augments a deep CNN with adaptive pooling that scans video frames, predicts their discriminative importance, and aggregates selected information for action classification. Its entropy-regularized, end-to-end design favors informative, non-redundant frames while retaining online interpretability.
- Architecture: AdaScan combines feature extraction, adaptive pooling, and label prediction in a sequential deep CNN architecture.The feature extractor produces a vector for each frame, adaptive pooling aggregates selected frame features, and the final module predicts the label.
- Adaptive Pooling: During a temporal scan, AdaScan predicts each frame’s importance from its current features and the pooled video representation so far.Importance is intended to be high for class-relevant information and low for redundant or uninformative frames.
- Adaptive Pooling: Unlike one-hot frame weighting in MIL, AdaScan can focus on multiple frames and outputs frame importances online.This allows the method to represent actions requiring information from more than one frame.
- Adaptive Pooling: The pooling update is a weighted mean that aggregates prior pooled features with the next frame according to its predicted importance.The resulting pooled vector uses the sequence of predicted importance scores as frame weights.
- Adaptive Pooling: Feeding the difference between current pooled features and the next frame helps the module reject redundant frames and improves generalization.The authors associate this residual with focusing on unseen features when deciding whether to pool a frame.
- Loss Function and Learning: The model is trained with cross-entropy classification loss plus an entropy regularizer that encourages a peaky distribution over frame importances.The trade-off parameter balances sparse frame selection against minimizing classification loss; excessive sparsity can select too few frames, while weak regularization can select more frames and possibly overfit.
4. Experimental Results
Experiments evaluate AdaScan on UCF101 and HMDB51 using spatial and temporal networks, baseline pooling methods, regularization, complementary features, and qualitative frame selection. AdaScan generally improves over mean pooling, achieves competitive state-of-the-art performance with complementary features, and learns sparse frame importance patterns.
- Experimental setup: AdaScan is evaluated on two challenging public action-classification datasets using spatial and temporal two-stream networks.The experimental section describes UCF101 and HMDB51 evaluations with RGB-based spatial and optical-flow-based temporal networks.
- Comparison with pooling methods: 79.1 and 81.7 are AdaScan’s spatial and temporal accuracies on UCF101 Split 1, exceeding mean pooling’s 78.0 and 80.8.MIL and max pooling perform worse in the same comparison, with 76.7 and 77.2 for spatial and 79.1 and 80.3 for temporal networks.
- Comparison with pooling methods: AdaScan improves over mean pooling in 11 of 12 dataset-network cases, with UCF101 gains from 77.6 to 78.6 and 82.4 to 83.4.HMDB51 gains are smaller: 40.9 to 41.4 for the spatial network and 48.6 to 49.2 for the temporal network.
- Effect of regularization strength: λ controls the trade-off between frame pruning and model fitting, enabling sharply increased sparsity from 1e6 to 1e7 with only a small performance drop.Beyond 1e7, sparsity continues increasing while performance declines, with changes reduced by gradient clipping.
- Comparison with state of the art: With complementary features, AdaScan reaches 91.3 and 61.0 with iDT and 93.2 and 66.9 with C3D on UCF101 and HMDB51.The paper reports these results as competitive with existing state-of-the-art results.
- Qualitative results: Qualitative visualizations show higher importance for action-relevant frames and multiple selected temporal segments in basketball, tennis-swing, and floor-gymnastics videos.In the punch example, failed punch attempts receive a moderate importance score of 0.2.
5. Conclusion
AdaScan adaptively pools video frames for human action recognition and outperforms baseline pooling methods. With complementary features, it achieves near-state-of-the-art results.
- AdaScan dynamically pools different frames for each video using predicted importance weights.The method performs a single temporal scan and pools frames online.
- The entropy-based regularizer controls pooling sparsity, helping control model overfitting.
- AdaScan outperforms max-pooling and mean-pooling baselines on UCF101 and HMDB51.
- AdaScan also performs better than Multiple Instance Learning-based deep networks and improves over previous LSTM-based deep networks.
- With complementary features, AdaScan achieves near-state-of-the-art results.