Source-linked AI summary
Sequential Deep Trajectory Descriptor for Action Recognition with Three-stream CNN
Yemin Shi, Yonghong Tian, Yaowei Wang, Tiejun Huang
TL;DR
Existing motion descriptors are limited in representing long-term motion, which is important for action recognition. The paper introduces sDTD by converting dense trajectories into two-dimensional sequential images and learning them with CNN-RNN processing, then adds an sDTD stream to a three-stream framework. The resulting method achieves state-of-the-art performance on KTH and UCF101 and outperforms most existing methods on HMDB51.
Problem
Existing features and descriptors cannot effectively capture motion information, especially long-term motion that is important for action recognition.
Method
The method projects dense trajectories into two-dimensional sequential Trajectory Texture images, learns them with a CNN-RNN network, and combines the resulting sDTD stream with spatial and temporal streams.
Results
The method achieves state-of-the-art performance on KTH and UCF101 and outperforms most existing methods on HMDB51.
Takeaways & Limitations
The three-stream framework jointly captures static spatial features, short-term motion, and long-term motion for video action recognition.
Abstract
from arXiv · showhide
Learning the spatial-temporal representation of motion information is crucial to human action recognition. Nevertheless, most of the existing features or descriptors cannot capture motion information effectively, especially for long-term motion. To address this problem, this paper proposes a long-term motion descriptor called sequential Deep Trajectory Descriptor (sDTD). Specifically, we project dense trajectories into two-dimensional planes, and subsequently a CNN-RNN network is employed to learn an effective representation for long-term motion. Unlike the popular two-stream ConvNets, the sDTD stream is introduced into a three-stream framework so as to identify actions from a video sequence. Consequently, this three-stream framework can simultaneously capture static spatial features, short-term motion and long-term motion in the video. Extensive experiments were conducted on three challenging datasets: KTH, HMDB51 and UCF101. Experimental results show that our method achieves state-of-the-art performance on the KTH and UCF101 datasets, and is comparable to the state-of-the-art methods on the HMDB51 dataset.
I. INTRODUCTION
The paper targets the difficulty of representing long-term motion for action recognition by introducing sDTD and integrating it into a three-stream framework. Experiments across KTH, HMDB51, and UCF101 report strong performance, including state-of-the-art results on KTH and UCF101.
- Long-term motion remains difficult to model effectively, although motion information is crucial for recognizing actions across consecutive video frames.
- The proposed sDTD projects dense trajectories into two-dimensional Trajectory Texture images and learns their sequential representation with CNN-RNN processing.
- The three-stream framework combines spatial, temporal, and sDTD streams to capture static appearance, short-term motion, and long-term motion, respectively.
- The method achieves state-of-the-art performance on KTH and UCF101 and outperforms most existing methods on HMDB51.
- The paper extends prior work by using sequential Trajectory Texture images, three-channel inputs, GoogLeNet, LSTMs, and evaluations on additional datasets.
II. SEQUENTIAL DEEP TRAJECTORY DESCRIPTOR
The sDTD pipeline extracts simplified dense trajectories, converts them into sequential Trajectory Texture images, and learns motion descriptors with a CNN-LSTM network. Its trajectory processing compensates camera motion under a planar-scene assumption and filters unsuitable trajectories.
- II. SEQUENTIAL DEEP TRAJECTORY DESCRIPTOR: sDTD extracts simplified dense trajectories from multiple video frames and converts them into sequential Trajectory Texture images before deep descriptor learning.
- II. SEQUENTIAL DEEP TRAJECTORY DESCRIPTOR: The CNN learns spatial features from the trajectory images, while the LSTM models their temporal features.
- A. The Simplified Dense Trajectories: Dense trajectories are sampled across spatial scales and tracked through a median-filtered dense optical-flow field.
- A. The Simplified Dense Trajectories: Trajectories are limited to a maximum length and removed when their displacement is too small or too large to reduce drifting and unsuitable motion.
- A. The Simplified Dense Trajectories: Unlike earlier dense-trajectory methods, the paper represents trajectories using absolute coordinates rather than extracting HOG, HOF, and MBH features.
- A. The Simplified Dense Trajectories: Camera-motion compensation assumes planar scenes, so complex camera motion is not considered and performance may be limited for 3D motion scenes.
B. The Sequential Trajectory Texture Image
The method converts dense trajectories into Trajectory Texture images so CNNs can efficiently learn motion representations while reducing information loss from overlapping trajectories.
- Dense trajectories represent movement and are projected from video segments into two-dimensional Trajectory Texture images.The trajectories are extracted from multiple frames and converted onto image planes.
- Overlapping dense trajectories cause overwriting and can reduce recognition accuracy during image generation.The paper identifies overwriting as information loss when trajectories overlap heavily.
- The method tracks overwritten pixels and starts a new Texture image when the overwrite count exceeds threshold P.S_j^n indicates whether pixels are nonzero, while O_j^n counts pixels overwritten after the nth frame.
- Trajectory Texture images encode trajectory information in a form that can be processed by CNNs.The images are reformulated as three-channel inputs compatible with existing ImageNet models.
C. The Sequential Deep Convolutional Trajectory Descriptor
The sDTD descriptor uses CNN features and LSTMs to model sequential trajectory textures and temporal dependencies, producing frame-level predictions from prior frames.
- The sDTD network learns from sequences of Trajectory Texture images using a CNN-RNN architecture.The implementation tested VGG-2048 and GoogLeNet, with GoogLeNet performing better in most cases.
- LSTM units recursively process sequence inputs through input, forget, memory-cell, and output states.The paper defines the gates and hidden-state computation used in its simplified LSTM model.
- CNN outputs are fed into multilayer LSTMs so the joint model can fuse spatial and temporal features.The network contains inputs, CNNs, RNNs, and predictions, with each frame predicted using previous frames.
III. ACTION RECOGNITION WITH SDTD
The three-stream framework separates static appearance, short-term motion, and long-term motion, then combines their predictions through late fusion.
- The framework contains spatial, temporal, and sDTD streams corresponding to static appearance, short-term motion, and long-term motion.The temporal subsystem is divided into short-term and long-term modules.
- The spatial stream uses single frames, while the temporal stream uses single optical-flow fields to describe short-term motion.This differs from two-stream ConvNets that stack multiple optical-flow fields as temporal input.
- CNN outputs are fed into LSTMs, producing a prediction for each frame before predictions are fused.The joint CNN-LSTM model is trained jointly in the described architecture.
- The sDTD stream represents long-term motion differently from the temporal stream despite both using optical-flow information.The sDTD stream samples motion states sparsely compared with dense optical flow.
- Late fusion combines nine predictions from the three GoogLeNet branches to obtain the final result.Each of the three streams provides three branch predictions.
IV. EXPERIMENTS
The experiments describe dataset evaluation, implementation details, and comparisons between sDTD and state-of-the-art methods.
- The experiments introduce the datasets and evaluation schemes, describe implementation details, and compare sDTD with state-of-the-art methods.
A. Datasets
The study evaluates sDTD on KTH, HMDB51, and UCF101 using dataset-specific training/testing protocols, including transfer from UCF101 to smaller datasets.
- Experiments use three public action-recognition datasets: KTH, HMDB51, and UCF101.
- KTH contains 2,391 sequences across six actions and 25 subjects, with 16 subjects for training and nine for testing.
- HMDB51 contains 6,766 realistic video clips from 51 action categories; the first split uses 70 training and 30 testing clips per class.
- UCF101 contains 13,320 clips from 101 action classes, and evaluation uses its first training/testing split.
- ImageNet pretraining is used, while UCF101 initially trains the three-stream model before transfer to KTH and HMDB51.
B. Implementation details
The implementation combines CNN and recurrent processing across three streams, using staged training for sDTD and late fusion of stream predictions.
- GoogLeNet and VGG-2048 are both tested as network backbones.
- The sDTD CNN is trained before adding the RNN because reduced sample counts may cause overfitting during joint training.
- CNN-RNN training reads 16 frames, flows, or sDTDs per video sample and uses batch size 16, totaling 256 inputs.
- The three-stream framework uses spatial, temporal, and sDTD branches with inputs, CNNs, RNNs, and predictions.
- The spatial, temporal, and sDTD learning rates use staged reductions, with CNN-RNN training stopping at 30K iterations.
C. Exploration experiments
Exploration experiments examine recurrent structure, backbone choice, stream complementarity, efficiency, and dataset-level behavior of sDTD.
- Benefits from LSTMs: CNN-RNN improves over pure CNN baselines: ST-streamG is 4.5% better than ST-ConvNetG, while DTDV and sDTDV reach 70.7% and 70.9%.
- Network structure: GoogLeNet is selected over VGG-2048 because sDTDG is about one percentage point better individually and retains a 0.3 advantage after fusion.
- Complementary properties of three streams: On UCF101, the ST-stream reaches 90.0%, spatial+sDTD reaches 89.7%, temporal+sDTD reaches 82.5%, and all three streams reach 92.2%.
- Evaluation across datasets: sDTD improves the ST-stream by 3.1% on KTH, 5.3% on HMDB51, and 2.1% on UCF101.
- Efficiency: Each video segment produces an average of nine trajectory texture images on UCF101, enabling sDTD to process 3.24 videos per second.
- Confusion analysis: HMDB51 remains challenging because some categories are easily misclassified, although sDTD performs well on most categories.
- Comparison with iDT: The sDTD stream outperforms iDT by around 7.4% on UCF101.
E. Comparison to the state of the art
sDTD outperforms the compared methods on KTH and UCF101 and most methods on HMDB51. It also exceeds the TDD+iDT+FV framework by 0.7% on UCF101, while confusion matrices are provided for HMDB51 and UCF101.
- Confusion matrices report sDTD performance on HMDB51 and UCF101.
- 0.7% improvement over TDD+iDT+FV is reported on UCF101.
V. CONCLUSIONS
The paper concludes that sDTD represents long-term motion through projected dense trajectories and a CNN-RNN network within a three-stream action-recognition framework. It reports strong benchmark performance and identifies broader long-term-dependence methods and larger datasets for future evaluation.
- sDTD projects dense trajectories into two-dimensional planes and uses a CNN-RNN network to learn long-term motion representation.
- A three-stream framework is employed to identify actions from video sequences.
- The method achieves state-of-the-art performance on KTH and UCF101 and outperforms most existing methods on HMDB51.
- Future work will test attention, Neural Turing Machines, and Memory Networks for long-term dependence and evaluate sDTD on larger datasets.