Source-linked AI summary
Grouped Spatial-Temporal Aggregation for Efficient Action Recognition
Chenxu Luo, Alan Yuille
TL;DR
Video action recognition needs temporal reasoning, but joint 3D processing is computationally expensive and prior decompositions do not clarify how spatial and temporal cues are used. GST instead splits feature channels into parallel spatial and temporal groups, achieving efficient temporal modeling and supporting layer-wise analysis of their contributions. On temporal-reasoning benchmarks, it achieves strong results with substantially lower computational cost.
Problem
Joint spatial-temporal networks are costly, while existing decompositions leave unclear how networks use spatial and temporal information across layers.
Method
GST decomposes feature channels into parallel spatial and temporal groups, applies separate operations, and concatenates the resulting features.
Results
GST achieves state-of-the-art results on several temporal-modeling datasets with much lower computational cost, including a 5% absolute advantage over two-stream TRN for its single-stream model.
Takeaways & Limitations
GST provides an efficient architecture that separately models static and dynamic information while enabling analysis of their differing roles across network layers.
Takeaways & Limitations
The approach assumes that feature channels can be effectively separated into groups specializing in static appearance and dynamic motion.
Abstract
from arXiv · showhide
Temporal reasoning is an important aspect of video analysis. 3D CNN shows good performance by exploring spatial-temporal features jointly in an unconstrained way, but it also increases the computational cost a lot. Previous works try to reduce the complexity by decoupling the spatial and temporal filters. In this paper, we propose a novel decomposition method that decomposes the feature channels into spatial and temporal groups in parallel. This decomposition can make two groups focus on static and dynamic cues separately. We call this grouped spatial-temporal aggregation (GST). This decomposition is more parameter-efficient and enables us to quantitatively analyze the contributions of spatial and temporal features in different layers. We verify our model on several action recognition tasks that require temporal reasoning and show its effectiveness.
1. Introduction
Temporal reasoning is essential for distinguishing actions with similar appearances, yet many standard datasets and models underuse temporal cues. GST addresses this gap by separating spatial and temporal channel groups, reducing computation while enabling analysis of how each contributes across network layers.
- Motivation: Many popular action-recognition datasets can be solved largely from static scenes or objects, so temporal cues may be underused.Shuffling temporal order can leave accuracy nearly unchanged on some datasets.
- Motivation: Fine-grained datasets such as Something-Something require temporal modeling because similar beginnings can lead to different action outcomes.Single frames are insufficient when classes differ by what happens later in the sequence.
- Motivation: Static appearance remains useful because it narrows interpretations and reveals state changes across sparsely sampled frames.
- Motivation: Joint spatial-temporal networks achieve strong performance but incur substantial computational cost, while prior decompositions leave spatial-temporal feature use unclear.
- Proposed approach: GST decomposes feature channels into separate spatial and temporal groups, applies different operations in parallel, and concatenates their outputs.The design is motivated by channels specializing in static appearance or motion features.
- Findings: The proposed model is more parameter-efficient, achieves strong results on temporal-modeling datasets, and reveals that lower layers emphasize static cues while higher layers emphasize dynamic cues.The network learns this soft selection without additional constraints.
2. Related Works
Related work spans static-biased 2D approaches, computationally expensive 3D networks, and decompositions that separate spatial and temporal filtering. New temporal-action datasets motivate methods that model sequence information more explicitly and efficiently.
- Action-recognition datasets: UCF101 and Kinetics have strong static bias, raising uncertainty about whether models learn actions or exploit scenes and objects.
- Action-recognition datasets: Fine-grained datasets such as Something-Something and Diving48 require extensive temporal modeling because similar actions can differ only across the full sequence.
- Temporal modeling: 2D methods aggregate frame features through sampling or late fusion, while 3D methods jointly model video volumes with higher computational demands.
- Efficient temporal modeling: The proposed GST design splits channels into spatial and spatial-temporal paths, contrasting with prior architectures that decouple filters through other structures.The two paths are later concatenated in the proposed architecture.
- Efficient temporal modeling: P3D-like approaches decouple spatial and temporal filters, and R(2+1)D reports better results with the same parameter count as 3D convolutions.
3. Approach
GST decomposes spatial-temporal processing across feature-channel groups, assigning separate spatial and temporal operations while controlling branch complexity. The module is designed as a flexible, parameter-efficient replacement for convolutional blocks that aggregates both feature types throughout the network.
- Grouped spatial-temporal decomposition: The asymmetric decomposition is motivated by separate static-appearance and dynamic-motion cues, whereas symmetric group convolutions do not improve performance.The design assigns distinct modeling roles to the two groups rather than applying the same operator across all channels.
- Grouped spatial-temporal decomposition: GST splits feature channels into spatial and temporal groups and applies different operations to each group.The spatial path uses 2D convolutions, while the temporal path uses 3D convolutions before concatenation.
- Computational costs for the spatial and temporal path: The GST complexity is controlled by α and β, which determine the temporal output-channel proportion and the input-channel allocation between branches.Larger α increases temporal-modeling capacity and computation, while β = 1/2 splits input channels evenly for more efficient architectures.
- Computational costs for the spatial and temporal path: GST uses spatial-temporal decomposition to reduce parameters while supporting multi-scale temporal modeling within a single layer.With β = 1, both paths process the full feature map; with β = 1/2, the input channels are divided between spatial and temporal modeling.
- Network architecture: The GST module can replace 3 × 3 convolutional layers in existing networks, with spatial-temporal features aggregated in intermediate blocks.The proposed architecture keeps other layers unchanged and uses average pooling for final prediction.
4. Experiments
Experiments across temporal action-recognition datasets show that GST delivers strong accuracy with reduced computational cost while exposing how spatial and temporal cues are used across network layers.
- Evaluation setup: GST is evaluated on five video datasets requiring temporal modeling, with Something-Something v1 and v2 serving as the main component-analysis benchmarks.The experiments also include Diving48, Epic-Kitchen, and EGTEA Gaze++.
- Something-Something results: GST outperforms comparable models on Something-Something v1 and v2 while using fewer or similarly many parameters and reduced temporal computation.GST-Large uses α = 1/4 and performs better than P3D, grouped 3D convolutions, larger C3D, and 2D baselines; GST uses a parameter count similar to ResNet-50.
- Something-Something results: An 8-frame GST model outperforms most current methods on Something-Something v1, while a 16-frame RGB model remains competitive with two-stream networks on v2.On v2, the single-stream model outperforms two-stream TRN by 5% absolute.
- Cross-dataset results: GST also outperforms prior methods on Diving48, Epic-Kitchen, and EGTEA Gaze++, including prior two-stream models with comparable backbones.On Diving48, even the lightweight ResNet-18 version exceeds the previous state of the art.
- Spatial-temporal analysis: Batch-normalization scaling factors provide a quantitative way to estimate spatial and temporal channel importance and reveal distinct static and dynamic cues.The analysis indicates that low-level features emphasize spatial information, whereas high-level features emphasize temporal information on datasets requiring temporal modeling.
- Spatial-temporal analysis: Temporal channel ratios affect representation capacity: too many temporal channels may encode static information, whereas α = 1/8 may provide insufficient temporal capacity.These observations help explain why reducing temporal channels can improve accuracy without substantially harming performance.
5. Conclusions
The GST module decomposes feature channels into static and dynamic parts, applying spatial and temporal convolutions separately to reduce computation and analyze their contributions.
- GST decomposes feature channels into static and dynamic parts and applies spatial and temporal convolutions separately.
- The decomposition reduces computation cost while enabling spatial and temporal features to be explored in parallel.
- Further diagnosis provides insight into how the two components contribute to the whole network.