Source-linked AI summary
Exploiting Image-trained CNN Architectures for Unconstrained Video Classification
Shengxin Zha, Florian Luisier, Walter Andrews, Nitish Srivastava, Ruslan Salakhutdinov
TL;DR
The paper asks how far image-trained CNNs can support unconstrained video classification despite limited motion modeling. It systematically evaluates CNN-based video representations and combines them with motion features, obtaining strong benchmark performance and state-of-the-art results on the reported datasets.
Problem
Image-classification CNNs do not directly exploit video motion, while the benefit of image-trained CNNs for video classification remained unclear.
Method
The paper develops a step-by-step video-classification procedure using off-the-shelf ImageNet-trained CNN features, evaluating layers, sampling, pooling, normalization, classifiers, and fusion.
Results
The image-trained CNN approach outperforms competitive non-CNN approaches on TRECVID MED’14, while CNN-motion fusion reaches 89.62% accuracy on UCF-101.
Takeaways & Limitations
Image-trained CNNs can provide a strong 2D baseline for video classification, and adding motion features can further improve performance.
Takeaways & Limitations
The ImageNet-trained CNN faces domain mismatch with heterogeneous, lower-quality video, and the paper expects gains from fine-tuning or spatiotemporal networks.
Abstract
from arXiv · showhide
We conduct an in-depth exploration of different strategies for doing event detection in videos using convolutional neural networks (CNNs) trained for image classification. We study different ways of performing spatial and temporal pooling, feature normalization, choice of CNN layers as well as choice of classifiers. Making judicious choices along these dimensions led to a very significant increase in performance over more naive approaches that have been used till now. We evaluate our approach on the challenging TRECVID MED'14 dataset with two popular CNN architectures pretrained on ImageNet. On this MED'14 dataset, our methods, based entirely on image-trained CNN features, can outperform several state-of-the-art non-CNN models. Our proposed late fusion of CNN- and motion-based features can further increase the mean average precision (mAP) on MED'14 from 34.95% to 38.74%. The fusion approach achieves the state-of-the-art classification performance on the challenging UCF-101 dataset.
1. Introduction
The paper investigates whether image-trained CNNs can support effective video classification despite their lack of explicit motion modeling. It proposes a carefully evaluated pipeline and combines CNN appearance features with motion features for stronger performance.
- Video classification requires methods that handle the large volume of consumer and web-shared videos.
- Directly applying image-classification pipelines to video fails to exploit motion information.
- The proposed approach systematically evaluates CNN layers, frame sampling, spatial and temporal pooling, normalization, and classifiers.
- CNN features can outperform strong static and motion-based approaches in accuracy and speed on TRECVID MED’14.
- Late fusion of CNN and motion-based features improves classification and achieves state-of-the-art performance on TRECVID MED’14 and UCF-101.
- The results motivate using image-trained CNNs as a strong 2D CNN baseline before training spatiotemporal architectures.
2. Deep Convolutional Neural Networks
The paper describes CNNs as layered spatial feature extractors and selects VGG as its pretrained architecture because of its reported single-model image-classification performance. The approach is intended to remain adaptable to other CNN architectures.
- CNN hidden units apply convolution or pooling and nonlinearities to spatial patches, producing feature maps with translation invariance.
- The system adopts a publicly available pretrained VGG model because of its superior postcompetition single-model performance over GoogLeNet and Krizhevsky’s model.
- The proposed video-classification approach is generic with respect to CNN architecture and can be adapted to other architectures.
3. Video Classification Pipeline
The video-classification pipeline converts sampled video frames and spatial regions into video-level CNN features, then applies normalization and SVM classification. It compares multiple pooling, normalization, and classifier choices.
- Each pipeline component is evaluated as part of the proposed video-classification system.
- The output layer provides 1,000-dimensional ImageNet posterior scores, while hidden layers serve as high-level image representations.
- The method samples 50 to 120 frames per clip and rescales multiple frame patches to 224 × 224 CNN inputs.
- Spatial and temporal pooling compare average and max pooling across spatial partitions, overlapping patches, and objectness-guided regions.
- Feature normalization compares ℓ1, ℓ2, and root normalization for video-level CNN feature vectors.
- The classifiers include linear SVMs and nonlinear SVMs with Gaussian RBF and exponential χ2 kernels, optionally preceded by PCA.
4. Modality Fusion
The paper compares CNN features with Fisher-vector representations built from handcrafted descriptors and evaluates weighted score fusion. Fusion is designed to combine complementary CNN and motion information.
- Fisher vectors encode zero-, first-, and second-order statistics of low-level descriptor distributions.
- The Fisher-vector pipeline learns a GMM, computes score-function gradients, concatenates them, and applies power and global ℓ2 normalization.
- The motion-based IDT representation concatenates HOG, HOF, and MBH descriptors extracted along estimated motion trajectories.
- Weighted average fusion converts SVM margins into posterior probabilities and optimizes class-score weights using cross-validation.
5. Evaluation in Event Detection
Evaluation on heterogeneous, untrimmed TRECVID MED’14 videos shows that carefully selected CNN layers, pooling, normalization, classifiers, dimensionality reduction, and fusion materially affect event-detection performance. CNN features outperform strong hand-engineered and competing CNN approaches, while motion fusion further improves results.
- Video Dataset and Performance Metric: TRECVID MED’14 contains 4,992 unlabeled background videos, 2,991 positive and near-miss training videos, and 23,953 test videos across 20 events.The dataset is heterogeneous, temporally untrimmed, and varies in resolution, quality, camera motion, and illumination; performance is measured by mAP.
- CNN Architectures and Layers: The deeper CNN architecture consistently performs better, and both hidden layers outperform the output layer under matched settings.The evaluation compares CNN architectures and layers within the same normalization and spatiotemporal-pooling configurations.
- Pooling, Spatial Pyramids and Objectness: Max pooling consistently outperforms average pooling, while SP8 provides up to 6% mAP gain over no spatial pooling at higher dimensionality and computational cost.Objectness-guided pooling reaches nearly SP8 performance with one-fourth the feature dimensionality when using a kernel SVM.
- Normalization: ℓ2 normalization is essential for hidden-layer features, whereas ℓ1 normalization performs poorly and root normalization reduces hidden-layer performance.Root normalization on the output layer performs essentially like ℓ2 normalization.
- Classifier: Kernel SVMs consistently outperform linear SVMs; RBF is best for hidden layers, while χ2 and RBF perform essentially the same for the output layer.The classifier comparison follows the MED’14 training rules, with one SVM trained per event while excluding other events’ positive and near-miss examples.
- PCA: PCA reduces the top feature from 32,768 dimensions to 4,096, 2,048, or 1,024 with slightly lower mAP, while still matching or outperforming features without spatial pyramids.The result supports spatial pyramids as useful information carriers while allowing dimensionality reduction without loss of important information.
- Fusion Performance: Late fusion consistently improves CNN-based features when combined with motion-based Fisher vectors, indicating benefits from integrating motion information with CNN representations.Static-plus-motion FV fusion also substantially improves over motion-only FV, while CNN-feature fusion alone provides little gain.
- Comparison with the State-of-the-Art: CNN features significantly outperform D-SIFT+FV, IDT+FV, MIFS, and a competing CNN approach on MED’14, despite low-resolution, compressed, and motion-blurred video frames.The proposed approach achieves a new state of the art through carefully designed CNN processing and fusion with motion-based features.
6. Evaluation in Action Recognition
On UCF-101, the image-trained CNN approach was compared across three splits with motion-based and neural-network methods. Late fusion with motion features achieved the strongest reported performance.
- Fusion Performance: 86.5% mean accuracy was obtained by the motion-based IDT+FV approach across three UCF-101 splits.Because UCF-101 videos are temporally trimmed and centered more on motion, IDT+FV outperformed the image-based CNN approach before fusion.
- Comparison with the State-of-the-Art: The image-based CNN approach yielded 6.3%, 6.0% and 13.9% higher performance than three neural-network baselines without dataset-specific fine-tuning.The comparisons were against the spatial stream ConvNet, single-frame model, and slow-fusion spatiotemporal ConvNet.
- Comparison with the State-of-the-Art: Late fusion of CNN-hidden6 or CNN-hidden7 with IDT+FV outperformed the two-stream CNN and LSTM approaches using image and optical-flow information.Table 9 reports comparisons in mean accuracy over three UCF-101 splits.
7. Computational Cost
CNN feature extraction was measured as real-time on a CPU and required less testing time than applying a Fisher-vector event model to the MED’14 videos.
- Feature Extraction: CNN feature extraction required 0.4 times the video playback time, so the features could be extracted in real time.D-SIFT Fisher-vector extraction also required 0.4 times playback time, whereas IDT required about five times playback time.
- Testing Time: 15s were required to apply a CNN-trained event model to 23,953 TRECVID MED’14 videos, compared with around 30s for a Fisher-vector model.The comparison was performed on the testing side.
8. Conclusion
The paper presents a step-by-step procedure for exploiting image-trained CNNs in video classification and finds that careful design choices can outperform motion-based alternatives. Late fusion adds motion information and improves results, while domain mismatch leaves scope for further gains.
- Conclusion: The procedure identifies CNN architecture, layer choice, spatiotemporal pooling, normalization, and classifier choice as the most sensitive factors.The authors support these design choices with experiments on video classification.
- Conclusion: Image-trained CNNs outperformed competitive motion- and spatiotemporal-based non-CNN approaches on the challenging TRECVID MED’14 dataset.The result held despite the mismatch between image-trained CNNs and the video domain.
- Conclusion: Late fusion of image-trained CNN features and motion-based IDT-FV features brought substantial gains on MED’14 and outperformed other vision-based approaches.On UCF-101, the image-trained CNN approach was comparable with the state of the art, while late fusion outperformed it.
- Limitations and Future Gains: Fine-tuning on the video dataset and learning motion with a spatiotemporal deep neural network could provide additional gains.The image-trained CNN was used as a blackbox feature extractor, and ImageNet images differ from the heterogeneous video data in quality and capture conditions.