Source-linked AI summary
Action Recognition with Image Based CNN Features
Mahdyar Ravanbakhsh, Hossein Mousavi, Mohammad Rastegari, Vittorio Murino, Larry S. Davis
TL;DR
Video action recognition needs temporal representations, while image-trained CNN features alone do not capture action structure sufficiently. The paper builds a hierarchical representation over CNN feature changes, uses binary coding to select key-frames, and reports superior results to state-of-the-art methods across several datasets.
Problem
Image-trained CNN fc7 features alone do not adequately represent the temporal variation and sub-actions needed for video action recognition.
Method
The method tracks changes in image-trained CNN features, organizes them in a coarse-to-fine hierarchy, and uses binary coding to select key-frames and informative snippets.
Results
The method achieves superior action-recognition performance compared with state-of-the-art methods across several action datasets.
Takeaways & Limitations
Image-trained CNN features can support video action recognition when augmented with temporal hierarchy and binary key-frame selection.
Abstract
from arXiv · showhide
Most of human actions consist of complex temporal compositions of more simple actions. Action recognition tasks usually relies on complex handcrafted structures as features to represent the human action model. Convolutional Neural Nets (CNN) have shown to be a powerful tool that eliminate the need for designing handcrafted features. Usually, the output of the last layer in CNN (a layer before the classification layer -known as fc7) is used as a generic feature for images. In this paper, we show that fc7 features, per se, can not get a good performance for the task of action recognition, when the network is trained only on images. We present a feature structure on top of fc7 features, which can capture the temporal variation in a video. To represent the temporal components, which is needed to capture motion information, we introduced a hierarchical structure. The hierarchical model enables to capture sub-actions from a complex action. At the higher levels of the hierarchy, it represents a coarse capture of action sequence and lower levels represent fine action elements. Furthermore, we introduce a method for extracting key-frames using binary coding of each frame in a video, which helps to improve the performance of our hierarchical model. We experimented our method on several action datasets and show that our method achieves superior results compared to other state-of-the-arts methods.
1. Introduction
The paper addresses video action recognition by extending image-trained CNN features with temporal structure, replacing handcrafted descriptors while avoiding the cost of training CNNs on videos.
- The complete pipeline computes frame CNN features, maps them to binary codes, builds a hierarchical representation, and classifies actions from temporal-word histograms.Figure 1 summarizes the stages from frame processing through classification.
- It tracks changes in fc7 features across time and organizes them hierarchically so higher levels capture coarse motion while lower levels capture fine sub-actions.The hierarchy decomposes complex actions into simpler temporal components.
- The approach is evaluated against classical handcrafted-feature methods and deep video models across several action-recognition datasets.The introduction identifies both families of prior approaches and reports experiments across standard datasets.
- The method represents complex actions with CNN features from image-trained networks, replacing handcrafted local descriptors with higher-level semantic representations.The paper frames this as a way to use image-based CNN training rather than video training.
- Binary coding automatically detects key-frames and selects informative video snippets for the hierarchical representation.The paper presents key-frame extraction as a way to focus on meaningful sub-videos.
2. CNN Feature Pyramid Architecture
The method represents video actions using image-trained CNN features while capturing temporal variation through hierarchical CNN flows and binary key-frame selection.
- 2.1. Spatial and Temporal: The pipeline uses AlexNet fc7 features for spatial information and tracks temporal changes in CNN feature space instead of using optical flow.The full pipeline comprises spatial and temporal feature extraction, pyramid construction, video representation, and classification.
- 2.4. Bag of snippets: The method represents snippets as temporal words and aggregates them with a bag-of-words histogram, using a χ2-kernel SVM for classification.The implementation constructs a 4000-element codebook for each pyramid level and assigns descriptors by Euclidean distance.
- 2.2. Pyramid of CNN Flows: A hierarchical CNN-flow pyramid captures coarse motion at upper levels and progressively finer sub-actions at lower levels.Each node computes the difference between the last and first key-frame of a sub-snippet; stacking the flows yields the snippet representation.
- 2.2. Pyramid of CNN Flows: The resulting high-dimensional pyramid representation is reduced with PCA at each hierarchy level before further video representation.Each CNN flow has 4096 dimensions, motivating level-wise dimensionality reduction.
- 2.3. Snippet Selection: Binary key-frame selection hashes frame features and selects frames when consecutive binary codes change, defining snippets between successive key-frames.The method uses binary codes whose changes reflect potentially meaningful semantic movement; ITQ produces 16-bit codes and uses Hamming-distance changes for selection.
- 2.4. Bag of snippets: Experiments compare the approach across action datasets, including UCF Sport, KTH, and UCF 11, with per-class accuracy tables for UCF Sport and KTH.The supplied passages identify the dataset tables and the experimental comparison scope but do not provide their accuracy values.
3. Experiments
Experiments evaluate the method on KTH, UCF-11, and UCF-Sport using appearance, overlapping-window, and binary key-frame snippets. Results report improved recognition over state-of-the-art methods, with parameter studies showing benefits from shorter snippets, larger binary codes, and deeper pyramids.
- 3.3. Experimental Results: The method improves action-recognition accuracy over state-of-the-art results across KTH, UCF-Sport, and UCF-11.Table 1 compares the proposed appearance baseline, overlapping-window approach, and binary-based snippet selection technique with state-of-the-art methods.
- 3.3.1 Results on KTH Dataset: On KTH, accuracy is best with binary size 16 and four pyramid levels.The study varies binary sizes 8, 10, 16, 20, and 32, and pyramid depths from one to four levels.
- 3.3.1 Results on KTH Dataset: On KTH, increasing overlapping-window length decreases recognition accuracy, while more pyramid levels improve performance by capturing finer motion.The tested snippet lengths are 20, 30, 40, and 50 frames; longer pyramids capture smaller sub-actions.
- 3.3.2 Results on UCF Sport Dataset: On UCF-Sport, the best accuracy uses 20-frame snippets and four pyramid levels.UCF-Sport experiments test overlapping-window lengths of 20, 25, 30, and 35 frames because its videos are short.
- 3.3.2 Results on UCF Sport Dataset: For UCF-Sport, larger binary vectors improve recognition, whereas longer snippets reduce recognition performance.Figure 8 examines binary-vector size, overlapping-window length, and pyramid levels.
- 3.3.3 Results on UCF11: UCF-11 evaluation uses 25-fold leave-one-out cross-validation with binary size 16, 20-frame snippets, and four pyramid levels.Results for the proposed snippet, binary, and baseline methods are summarized in Table 1, with class-level details in Figure 5.
4. Conclusion
The paper introduces a video representation that combines image-trained CNN features with hierarchical temporal modeling and binary key-frame selection. It reports better action-recognition performance than state-of-the-art methods across several datasets.
- 4. Conclusion: The proposed hierarchical CNN representation outperforms state-of-the-art action-recognition methods on several datasets.The underlying CNN is trained on image data rather than videos.
- 4. Conclusion: The representation models motion through CNN flow, defined as subtraction between CNN features across time.A hierarchical structure captures coarse motion at higher levels and fine motion at lower levels.
- 4. Conclusion: Binary key-frame selection identifies snippets between key frames that carry considerable information for the video representation.The key-frame technique is presented as a novel component of the proposed representation.