Source-linked AI summary
Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
Limin Wang, Yu Qiao, Xiaoou Tang
TL;DR
Action recognition in realistic videos needs robust visual representations despite substantial variation and challenging video quality. The paper proposes TDD, which combines deep convolutional feature learning with trajectory-constrained pooling of improved-trajectory-aligned responses. TDDs outperform prior hand-crafted and deep-learned features on HMDB51 and UCF101, achieving state-of-the-art comparisons.
Problem
Action recognition remains difficult on realistic videos because visual representations must handle substantial variation, clutter, viewpoint changes, and motion differences.
Method
TDD combines deep-learned multi-scale convolutional feature maps with trajectory-constrained sampling and pooling based on improved trajectories.
Results
TDDs outperform previous methods on HMDB51 and UCF101, with best results exceeding other methods by 4.8% and 3.5%, respectively.
Takeaways & Limitations
TDD integrates the advantages of hand-crafted trajectories and deep-learned features and achieves superior performance on two action-recognition datasets.
Takeaways & Limitations
Temporal TDD is only comparable to the temporal network because its extraction uses original rather than warped optical flow; warped flow is left for future work.
Abstract
from arXiv · showhide
Visual features are of vital importance for human action understanding in videos. This paper presents a new video representation, called trajectory-pooled deep-convolutional descriptor (TDD), which shares the merits of both hand-crafted features and deep-learned features. Specifically, we utilize deep architectures to learn discriminative convolutional feature maps, and conduct trajectory-constrained pooling to aggregate these convolutional features into effective descriptors. To enhance the robustness of TDDs, we design two normalization methods to transform convolutional feature maps, namely spatiotemporal normalization and channel normalization. The advantages of our features come from (i) TDDs are automatically learned and contain high discriminative capacity compared with those hand-crafted features; (ii) TDDs take account of the intrinsic characteristics of temporal dimension and introduce the strategies of trajectory-constrained sampling and pooling for aggregating deep-learned features. We conduct experiments on two challenging datasets: HMDB51 and UCF101. Experimental results show that TDDs outperform previous hand-crafted features and deep-learned features. Our method also achieves superior performance to the state of the art on these datasets (HMDB51 65.9%, UCF101 91.5%).
1. Introduction
Action recognition remains difficult because realistic videos exhibit substantial variation, while existing representations separately emphasize hand-crafted motion descriptors or deep-learned semantic features. TDD combines these strengths by pooling deep convolutional responses along improved trajectories and achieves strong benchmark performance.
- Realistic action videos are difficult because background clutter, viewpoint changes, motion variation, high dimensionality, and low resolution complicate robust recognition.
- Hand-crafted representations use local descriptors and trajectory alignment, while deep-learned methods learn semantic representations with discriminatively trained neural networks.
- TDD combines improved trajectories and two-stream ConvNets by pooling multi-scale convolutional feature maps along trajectories.
- The extraction pipeline trains two-stream ConvNets, obtains convolutional maps, detects improved trajectories, pools responses around trajectory-centered spatiotemporal tubes, and aggregates local TDDs with Fisher vectors.
2. Related Works
Prior work includes hand-crafted local features and deep architectures for action recognition. TDD addresses limitations of semantic capacity and temporal modeling by combining trajectory-constrained pooling with deep-learned convolutional features.
- Hand-crafted features: Hand-crafted local features represent spatiotemporal regions with descriptors such as HOG, HOF, HOG3D, ESURF, and MBH, often using point trajectories for alignment.
- Hand-crafted features: A limitation of local features is their lack of semantics and discriminative capacity, motivating mid-level and high-level representations based on heuristic mining.
- Deep-learned features: Deep video models include GRBMs, 3D ConvNets, Deep ConvNets, and Two-Stream ConvNets for learning representations from video data.
- TDD: TDD uses trajectory-constrained sampling and pooling to incorporate temporal characteristics that earlier deep models did not adequately consider.
3. Improved Trajectories Revisited
The paper revisits improved trajectories as the motion-aligned foundation for TDD extraction. Its implementation tracks points at one spatial scale, accounts for camera motion, and uses the resulting trajectories for later pooling.
- TDD uses improved trajectories because they perform well, while the descriptor design remains independent of the particular trajectory-extraction method.
- Dense trajectories sample points across eight spatial scales, remove homogeneous regions, and track the remaining points using median filtering of dense optical flow.
- Trajectory updates use median-filtered optical flow, while static or abruptly displaced trajectories are discarded and trajectory length is limited to 15 frames.
- Improved trajectories estimate camera motion with feature matching and RANSAC homography, rectify frames, and recompute warped optical flow.
- The implementation tracks points only at the original spatial scale, then extracts multi-scale TDDs around those trajectories for faster processing.
- A video yields K trajectories, each represented as a sequence of P tracked points used for trajectory-constrained sampling and pooling.
4. Deep Convolutional Descriptors
TDDs use two-stream ConvNets to learn multi-scale convolutional feature maps, then normalize and pool them along improved trajectories to form descriptors combining deep and trajectory-based representations.
- Convolutional networks: TDD extraction uses two-stream ConvNets to obtain multi-scale spatial and temporal convolutional feature maps.Spatial nets capture static appearance, while temporal nets process stacked optical-flow fields to describe motion.
- Convolutional feature maps: After training, the ConvNets serve as generic feature extractors, producing spatial and temporal feature maps for each video.The implementation removes layers after the target extraction layer and pads inputs so trajectory positions map directly to feature-map coordinates.
- Convolutional networks: The networks are adapted for action recognition with reduced conv4 filters and a lower-dimensional full7 layer.The architecture uses 512 rather than 1024 conv4 filters and 2048 rather than 4096 full7 dimensions.
- Trajectory-pooled descriptors: TDDs are local trajectory-aligned descriptors computed by pooling convolutional responses within a 3D volume around each trajectory.The volume spans N × N pixels and P frames, while spatial and temporal networks provide appearance and motion information, respectively.
- Feature map normalization: Normalization suppresses activation burstiness using spatiotemporal or channel normalization before trajectory pooling.Spatiotemporal normalization equalizes each channel across the video, whereas channel normalization equalizes feature values across channels at each pixel.
- Trajectory-pooled descriptors: The resulting TDD combines the merits of improved dense trajectories and two-stream ConvNets.It uses deep-learned convolutional representations together with trajectory-constrained sampling and pooling.
5. Experiments
Experiments on HMDB51 and UCF101 evaluate TDD design choices, component performance, and comparisons with prior representations. TDDs benefit from trajectory-constrained pooling, normalization, and fusion with iDTs, achieving strong performance against existing methods.
- Experimental setup: Experiments evaluate TDDs on the HMDB51 and UCF101 datasets using three training/testing splits.HMDB51 results average accuracy across three splits; UCF101 follows the THUMOS13 evaluation scheme.
- Implementation details: 84.7% accuracy is achieved by the implemented two-stream ConvNets, with spatial and temporal nets reaching 71.2% and 80.1%, respectively.The implemented two-stream result is close to the 85.6% reported for the reference two-stream ConvNets.
- Exploration experiments: PCA dimension 64 performs best in the explored 32–256 range, while higher dimensions may degrade performance.The exploration uses conv4 spatial-net descriptors with spatiotemporal normalization on HMDB51.
- Exploration experiments: Normalization improves performance; spatiotemporal normalization performs best, and combining it with channel normalization further improves results.The combined Fisher-vector representation is used for TDDs in the remainder of the evaluation.
- Exploration experiments: Conv4 and conv5 are the strongest spatial-net layers, while conv3 and conv4 are the strongest temporal-net layers on HMDB51.The comparison covers outputs from convolutional, activation, and pooling layers across spatial and temporal nets.
- Evaluation of TDDs: Temporal convolutional descriptors are better than or comparable to HOF and MBH, but their improvement is less evident than for spatial descriptors.The authors attribute this difference to HOF and MBH being computed from warped optical flow and propose warped flow for TDD extraction as future work.
- Evaluation of TDDs: Combining TDDs with iDTs through early Fisher-vector fusion further boosts recognition performance.The improvement indicates that TDDs are complementary to low-level local features.
- Evaluation of TDDs: TDDs outperform previous methods on both datasets, with best results exceeding other methods by 4.8% on HMDB51 and 3.5% on UCF101.Combined TDDs also outperform two-stream ConvNets by around 4% and 2% on the two datasets, respectively.
6. Conclusions
The paper proposes TDDs, which combine deep-learned convolutional features with trajectory-constrained sampling and pooling. TDDs achieve superior action-recognition performance on two datasets compared with state-of-the-art methods.
- TDDs integrate the advantages of hand-crafted and deep-learned features.
- Deep architectures learn discriminative convolutional feature maps, which trajectory-constrained sampling and pooling aggregate into TDDs.
- TDDs achieve superior performance on two action-recognition datasets compared with state-of-the-art methods.