Source-linked AI summary
Learning Spatio-Temporal Representation with Local and Global Diffusion
Zhaofan Qiu, Ting Yao, Chong-Wah Ngo, Xinmei Tian, Tao Mei
TL;DR
Local convolutions limit long-range dependency modeling in complex video recognition. The paper introduces Local and Global Diffusion networks, which learn interacting local and global representations in parallel and combine them with a kernelized classifier. LGD improves performance across Kinetics datasets and generalizes across additional recognition and detection benchmarks.
Problem
Local convolutional operations inadequately capture large-range dependencies, a limitation that is especially challenging for video with complex temporal variations.
Method
LGD splits feature maps into local and global paths, models their interactions with diffusion blocks, and combines their representations using a kernelized classifier.
Results
LGD outperforms several state-of-the-art methods across six benchmarks, including Kinetics-400, Kinetics-600, four action-recognition benchmarks, and spatio-temporal action-detection benchmarks.
Takeaways & Limitations
The results validate learning local and global video representations in a unified architecture across large-scale classification and additional recognition and detection datasets.
Takeaways & Limitations
Future work includes investigating attention mechanisms, studying local-global combination more deeply, and extending LGD to inputs such as audio.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for visual recognition problems. Nevertheless, the convolutional filters in these networks are local operations while ignoring the large-range dependency. Such drawback becomes even worse particularly for video recognition, since video is an information-intensive media with complex temporal variations. In this paper, we present a novel framework to boost the spatio-temporal representation learning by Local and Global Diffusion (LGD). Specifically, we construct a novel neural network architecture that learns the local and global representations in parallel. The architecture is composed of LGD blocks, where each block updates local and global features by modeling the diffusions between these two representations. Diffusions effectively interact two aspects of information, i.e., localized and holistic, for more powerful way of representation learning. Furthermore, a kernelized classifier is introduced to combine the representations from two aspects for video recognition. Our LGD networks achieve clear improvements on the large-scale Kinetics-400 and Kinetics-600 video classification datasets against the best competitors by 3.5% and 0.7%. We further examine the generalization of both the global and local representations produced by our pre-trained LGD networks on four different benchmarks for video action recognition and spatio-temporal action detection tasks. Superior performances over several state-of-the-art techniques on these benchmarks are reported. Code is available at: https://github.com/ZhaofanQiu/local-and-global-diffusion-networks.
1. Introduction
Video representation remains challenging because local convolutions struggle to capture long-range dependencies amid complex temporal variation. LGD addresses this by learning local and global representations in parallel and modeling their diffusion.
- Video content presents larger variations and complexities that make powerful, generic spatio-temporal representation learning challenging.
- 2D and 3D convolutions process only local neighboring windows, limiting their ability to capture a holistic view of the field.
- Repeated convolution and pooling can enlarge receptive fields, but deep stacks make distant connections depend on many local operations and cause vanishing gradients.
- LGD networks split feature maps into local and global paths that describe local variation and holistic appearance at each spatio-temporal location.
- LGD blocks mutually infer between the two paths, while a kernel-based classifier combines their final representations for video recognition.
- LGD outperforms several state-of-the-art models across six benchmarks covering video action recognition and spatio-temporal action detection.
2. Related Work
Prior video representation methods include hand-crafted descriptors and deep CNN-based approaches, but long-range temporal dependency beyond local operations remains incompletely exploited. LGD simultaneously learns and combines local and global representations to address this gap.
- Existing video representation research broadly comprises hand-crafted methods and deep learning-based methods.
- Hand-crafted descriptors model spatiotemporal variation using local features, but are not optimized and generalize poorly across different video-analysis tasks.
- Deep approaches extend image CNNs to video frames, optical flow, temporal segments, or short clips using 2D and 3D convolutional architectures.
- Long-range temporal dependency beyond local operations remains not fully exploited in existing methods.
- Unlike iterative local and non-local operations, LGD learns local and global representations simultaneously and combines both paths for prediction.
3. Local and Global Diffusion
LGD models local and global video representations in parallel, allowing each path to update through diffusion with the other. A kernelized classifier then combines both representations for prediction.
- LGD blocks: LGD blocks split feature maps into local and global paths and learn their representations in parallel.The local-global pair is updated jointly through mutually interacting paths.
- LGD blocks: Each LGD block updates local and global features by modeling two diffusion directions between the paths.Global-to-local diffusion prioritizes the global vector, while local-to-global diffusion updates the global vector from local features.
- LGD blocks: Global-to-local diffusion broadcasts a projected global residual to locations before applying the local transformation F.The projection uses a learned matrix and an upsampling operation that duplicates the residual vector across locations.
- LGD blocks: Local-to-global diffusion combines the previous global feature with global average pooling of the updated local feature.Projection matrices linearly embed the local and global inputs into the updated global representation.
- Combination classifier: The classifier uses a bilinear kernel and Tensor Sketch Projection to combine final local and global representations into a high-dimensional prediction feature.The resulting representation is trained end-to-end and passed to a fully connected layer for class-label prediction.
4. Local and Global Diffusion Networks
The framework has LGD-2D and LGD-3D variants that differ in sampling and local transformation while sharing the LGD representation strategy. Training uses a two-stage procedure before tuning the combined classifier.
- LGD-2D: LGD-2D applies LGD blocks to a temporal segment network using sparsely sampled frames and 2D convolution.A shared 2D CNN processes one frame from each of T video snippets, while the global path represents all sampled frames.
- LGD-3D: LGD-3D processes T consecutive frames with pseudo-3D convolution as its local transformation.Pseudo-3D convolution decomposes 3D learning into spatial 2D convolutions and temporal 1D operations.
- LGD-3D: LGD-3D builds LGD blocks on residual units after replacing each ResNet-50 3 × 3 convolution with spatial and temporal convolutions.The architecture uses 16 consecutive frames at 112×112 resolution as its input video clip.
- Optimization: Training first optimizes the basic network by adjusting local and global representations separately, then tunes the whole network with the combination classifier.The first-stage loss includes classification errors from both the global representation and pooled local representation.
5. Experiments
Experiments evaluate LGD blocks, design choices, optimization behavior, and performance across video recognition and spatio-temporal detection benchmarks. LGD variants generally outperform baselines, while pre-training, feature combination, long clips, and deeper backbones further improve results.
- LGD block evaluation: LGD blocks outperform baseline networks for both 2D and 3D CNNs on Kinetics-600 validation.The comparison uses ResNet-50 backbones trained from scratch without the local-global combination classifier.
- LGD block evaluation: LGD networks produce lower training losses than baselines and converge faster and more stably on Kinetics-600.The local-path loss remains consistently lower than the global-path loss, which the authors speculate may reflect low-rank projection information loss.
- Ablation study: ImageNet pre-training raises top-1 accuracy from 72.5% to 74.4% for LGD-2D and from 74.2% to 75.8% for LGD-3D.These results are reported on Kinetics-600 validation with ResNet-50 backbones.
- Ablation study: The local and global combination classifier improves LGD-2D and LGD-3D performance by 0.4% and 0.5%, respectively.Training LGD-3D on 128-frame clips adds 3.1%, while ResNet-101 reaches 76.7% for LGD-2D and 81.5% for LGD-3D.
- State-of-the-art comparisons: 79.4% top-1 accuracy with RGB input gives LGD-3D relative improvements of 10.1%, 6.8%, 2.1%, and 6.2% over I3D, R(2+1)D, NL I3D, and S3D-G on Kinetics-400.Fusing RGB and Flow increases accuracy to 81.2%, reported as the best published Kinetics-400 performance at that time.
- State-of-the-art comparisons: LGD-3D reaches 81.5% on Kinetics-600 validation with RGB and 83.1% when combining RGB and Flow.The RGB result is reported as a 3.4% relative improvement over P3D with ResNet-152.
- Video representation evaluation: Two-stream LGD-3D achieves 98.2% on UCF101 and 80.5% on HMDB51, outperforming IDT, Two-stream, and TSN by 11.8%, 10.2%, and 4.0% on UCF101.It also surpasses Two-stream I3D by 0.3% on UCF101.
- Video representation evaluation: LGD-3D achieves the best reported detection performance across the evaluated J-HMDB and UCF101D cases.At standard IoU thresholds, relative improvements over the best competitor are 4.4% on J-HMDB and 5.5% on UCF101D.
6. Conclusion
The paper presents LGD networks that unify local and global video representations through diffusion blocks and a kernelized classifier, achieving state-of-the-art results across six datasets. Future work targets attention, improved representation fusion, and additional input modalities.
- LGD networks learn localized and holistic video representations jointly through diffusion operations and combine their predictions with a kernelized classifier.
- The proposed LGD-2D and LGD-3D architectures improve performance across Kinetics-400, Kinetics-600, and four additional recognition or detection datasets.
- LGD networks achieve new state-of-the-art performance on all six evaluated datasets.
- Future work includes incorporating attention mechanisms, studying local-global combination strategies, and extending LGD to inputs such as audio.