Source-linked AI summary
Deep Temporal Linear Encoding Networks
Ali Diba, Vivek Sharma, Luc Van Gool
TL;DR
Existing ConvNet approaches rarely encode features from entire videos, often limiting temporal processing to shorter sequences or separate spatial and temporal streams. The paper introduces TLE, an end-to-end CNN layer that aggregates and encodes whole-video spatial and temporal information. TLE outperforms current state-of-the-art methods on HMDB51 and UCF101, including 95.6% accuracy on UCF101 and 71.1% on HMDB51 for TLE:Bilinear.
Problem
Existing ConvNet approaches rarely encode features from entire videos and typically process shorter sequences, limiting whole-video temporal representation.
Method
TLE aggregates feature maps from frames or clips across an entire video and encodes them into a compact feature vector through end-to-end learning.
Results
TLE outperforms current state-of-the-art methods on HMDB51 and UCF101; TLE:Bilinear reaches 95.6% accuracy on UCF101 and 71.1% on HMDB51.
Takeaways & Limitations
TLE provides a compact global video representation that models feature interactions expressively without undesired information loss.
Takeaways & Limitations
The paper focuses on two-stream architectures and identifies further architectural alternatives for spatial and temporal segment aggregation as future work.
Abstract
from arXiv · showhide
The CNN-encoding of features from entire videos for the representation of human actions has rarely been addressed. Instead, CNN work has focused on approaches to fuse spatial and temporal networks, but these were typically limited to processing shorter sequences. We present a new video representation, called temporal linear encoding (TLE) and embedded inside of CNNs as a new layer, which captures the appearance and motion throughout entire videos. It encodes this aggregated information into a robust video feature representation, via end-to-end learning. Advantages of TLEs are: (a) they encode the entire video into a compact feature representation, learning the semantics and a discriminative feature space; (b) they are applicable to all kinds of networks like 2D and 3D CNNs for video classification; and (c) they model feature interactions in a more expressive way and without loss of information. We conduct experiments on two challenging human action datasets: HMDB51 and UCF101. The experiments show that TLE outperforms current state-of-the-art methods on both datasets.
1. Introduction
Human action recognition remains difficult because videos contain camera motion, changing viewpoints, and long-range temporal structure. Existing ConvNet approaches either omit motion or process only sparsely sampled shorter clips, motivating TLE for whole-video aggregation.
- Human action recognition remains challenging because camera motion and changing viewpoints complicate video understanding.
- Two-stream ConvNets integrate spatial and temporal information and rely on fusion techniques such as trajectory-constrained, 3D, and consensus pooling.
- One-stream spatial ConvNets use individual frames and lack motion modeling, while temporal networks rely on dense optical flow and incur high costs on longer videos.
- TLE aggregates frames or clips over longer time ranges into a compact, robust global feature representation through end-to-end learning.
2. Related Work
Prior action-recognition methods include hand-crafted spatio-temporal features and ConvNet architectures, while encoding methods provide compact alternatives to high-dimensional representations. TLE extends this direction by encoding interactions among segments across entire videos for video-level prediction.
- Hand-crafted features such as HOG3D, SIFT3D, HOF, ESURF, MBH, and IDT provide effective representations but have computational and semantic limitations.
- One-stream and two-stream ConvNets sought to exploit temporal information, but frame-based models could not capture motion and two-stream methods focused on fusion.
- Prior encoding methods include BoW, Fisher vector, sparse encoding, VLAD, and bilinear models, with compact bilinear pooling reducing dimensionality while retaining performance.
- TLE captures expressive interactions among segments across entire videos and encodes them into a compact representation for video-level prediction.
3. Approach
TLE aggregates feature maps from sparsely sampled frames or clips across an entire video, then encodes the aggregate into a compact video representation for prediction. The approach supports multiple aggregation and encoding choices, with element-wise multiplication selected and bilinear encoding modeling channel interactions.
- Deep Temporal Linear Encoding: TLE aggregates CNN feature maps from multiple frames or clips across longer time ranges instead of separately scoring short segments.The layer produces a single encoded feature space for the video using end-to-end learning.
- Deep Temporal Linear Encoding: The layer accepts K feature maps of size h×w×c, applies an aggregation operator, and outputs an encoded feature vector y∈R^d.The encoded dimension d is determined by the selected encoding method.
- 3D ConvNet Application: For 3D ConvNets, shared-weight ConvNets process video clips whose feature maps are aggregated and encoded into a video-level prediction.The figure describes the same TLE pipeline applied to clip-based 3D ConvNets.
- Temporal Aggregation: Element-wise multiplication of segment feature maps yielded the best results among the investigated temporal aggregation functions.The other considered choices included element-wise average and maximum aggregation.
- Feature Encoding: The aggregated feature map is passed to an encoding method in which every channel interacts with every other channel.The paper investigates bilinear models and other encoding or pooling methods.
- Feature Encoding: Bilinear encoding captures feature interactions through outer products, while Tensor Sketch projects the resulting high-dimensional space to a lower-dimensional representation.The projection avoids computing the outer product directly and significantly reduces model parameters.
- Training and Comparison: Compared with fully-connected pooling, bilinear models use fewer parameters and perform better, apart from computational efficiency.The paper also states that the model parameters are learned through end-to-end back-propagation.
4. Evaluation
The evaluation tests TLE with two-stream and C3D ConvNets on HMDB51 and UCF101, comparing aggregation functions, architectures, and state-of-the-art methods. TLE performs best with element-wise multiplication and bilinear models, encoding long-range appearance and motion across entire videos.
- Experimental setup: Experiments use HMDB51 and UCF101, evaluating TLE with two-stream and C3D ConvNets across the datasets’ three official splits.HMDB51 contains 51 action categories and 6,766 clips; UCF101 contains 101 action classes and 13,320 clips.
- Aggregation functions: Element-wise multiplication performs best among the evaluated aggregation functions for both two-stream and C3D ConvNets.The authors select it as the default aggregation function.
- ConvNet architectures: BN-Inception achieves higher TLE performance than AlexNet and VGG-16, exceeding VGG-16 by 5.4/2.3% for spatial ConvNets and 2.3/4.9% for temporal ConvNets on UCF101/HMDB51.The reported gains are ordered as UCF101/HMDB51.
- Two-stream ConvNets: 95.6% and 71.1% accuracy are achieved by TLE:Bilinear on UCF101 and HMDB51, respectively, outperforming the compared two-stream and traditional methods.TLE:Bilinear is reported as the best-performing method in this comparison.
- C3D ConvNets: 86.3% and 60.3% accuracy are achieved by TLE:Bilinear on UCF101 and HMDB51, respectively, improving over original C3D ConvNets by 4/3.5%.The gains are ordered as UCF101/HMDB51.
- C3D ConvNets: TLE:Bilinear encodes dynamic appearance and motion using multiple long-range temporal cues unavailable to the original C3D ConvNets.The authors associate this encoding with the model’s stronger performance than other methods.
5. Scene Context Embedding
The scene-context experiment adds a Places365-pretrained segment to TLE so the model can combine action and scene information. This context-augmented method outperforms the three-segment spatial TLE baseline, indicating complementary information between the streams.
- Method: The experiment adds a fourth ConvNet segment pretrained on Places365 to a three-segment TLE architecture.The additional segment supervises scene-related representations and uses extra data.
- Motivation: The context stream incorporates scenery and object information that spatial ConvNets trained on individual frames may misclassify.The authors treat this contextual information as potentially useful for action recognition.
- Results: The context-augmented method outperforms three-segment TLE on spatial ConvNets, showing that the two information streams are complementary.The comparison is reported for VGG-16 on split1 of HMDB51 and UCF101.
6. Conclusion
TLE aggregates information from entire videos within ConvNet architectures and produces global features through end-to-end learning. The authors report improved performance and efficiency, while identifying further architectural alternatives for segment aggregation as future work.
- TLE aggregates frame- or clip-level information from entire videos inside ConvNet architectures to produce a global feature representation through end-to-end learning.The model performs action prediction over an entire video.
- TLE achieved better performance than state-of-the-art methods on HMDB51 and UCF101 while remaining computationally efficient and compact.The authors also report significantly fewer model parameters than fully connected ConvNets.
- TLE retains expressive feature interactions without undesired information loss and may generalize to other architectures and sequential data streams.The paper focuses on two-stream and C3D ConvNets but describes broader potential applications.
- Future work will investigate alternative spatial and temporal segment aggregation architectures, including hierarchical aggregation of spatio-temporal networks.