Source-linked AI summary
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning
Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, Yu Qiao
TL;DR
Video representation learning must reconcile local redundancy with complex long-range dependency. UniFormer combines 3D convolution and spatiotemporal self-attention, learning local affinity in shallow layers and global affinity in deep layers. Across Kinetics and Something-Something benchmarks, it reports a favorable accuracy-efficiency balance and strong top-1 accuracy with ImageNet-1K pretraining.
Problem
Video understanding requires rich spatiotemporal representations despite local redundancy and complex long-range dependency between frames.
Method
UniFormer unifies 3D convolution and spatiotemporal self-attention, using local MHRA in shallow layers and global MHRA in deep layers.
Results
UniFormer achieves a favorable accuracy-efficiency balance across Kinetics-400/600 and Something-Something V1/V2 benchmarks.
Takeaways & Limitations
Joint spatiotemporal relation learning supports video representations that address redundancy and dependency within one hierarchical model.
Abstract
from arXiv · showhide
It is a challenging task to learn rich and multi-scale spatiotemporal semantics from high-dimensional videos, due to large local redundancy and complex global dependency between video frames. The recent advances in this research have been mainly driven by 3D convolutional neural networks and vision transformers. Although 3D convolution can efficiently aggregate local context to suppress local redundancy from a small 3D neighborhood, it lacks the capability to capture global dependency because of the limited receptive field. Alternatively, vision transformers can effectively capture long-range dependency by self-attention mechanism, while having the limitation on reducing local redundancy with blind similarity comparison among all the tokens in each layer. Based on these observations, we propose a novel Unified transFormer (UniFormer) which seamlessly integrates merits of 3D convolution and spatiotemporal self-attention in a concise transformer format, and achieves a preferable balance between computation and accuracy. Different from traditional transformers, our relation aggregator can tackle both spatiotemporal redundancy and dependency, by learning local and global token affinity respectively in shallow and deep layers. We conduct extensive experiments on the popular video benchmarks, e.g., Kinetics-400, Kinetics-600, and Something-Something V1&V2. With only ImageNet-1K pretraining, our UniFormer achieves 82.9%/84.8% top-1 accuracy on Kinetics-400/Kinetics-600, while requiring 10x fewer GFLOPs than other state-of-the-art methods. For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60.9% and 71.2% top-1 accuracy respectively. Code is available at https://github.com/Sense-X/UniFormer.
1 INTRODUCTION
Video understanding must address both local spatiotemporal redundancy and long-range dependency. UniFormer unifies convolution and self-attention, using different relation aggregation strategies across network depth to balance efficiency and accuracy.
- Videos contain subtle local motion redundancy and dynamic long-range dependencies across frames.
- 3D convolution captures local features and reduces adjacent-frame redundancy but struggles with long-range dependency because of its limited receptive field.
- TimeSformer compares an anchor token with all contextual tokens although only neighboring tokens contribute, wasting computation on local representations.
- UniFormer achieves the best balance between accuracy and computation on Kinetics-400 and Something-Something V2.
- UniFormer unifies 3D convolution and spatiotemporal self-attention in a concise transformer format to balance computation and accuracy.
- Its relation aggregator learns local relations in shallow layers and global relations in deep layers, while jointly encoding spatiotemporal context.
2 RELATED WORK
Video representation research has developed along two main paths: convolutional networks for spatiotemporal processing and transformers for long-term dependency modeling. Related methods modify these architectures to improve optimization, complexity, or temporal-spatial modeling.
- Convolution-based Video Networks: 3D convolutional networks have been dominant in video understanding but face difficult optimization and large computation costs.
- Convolution-based Video Networks: Prior convolutional methods inflate 2D kernels or factorize 3D kernels across dimensions to improve optimization or reduce complexity.
- Transformer-based Video Networks: Vision transformers and video variants verify strong capacity for capturing long-term dependencies in spatiotemporal learning.
3 METHOD
UniFormer uses a concise transformer architecture that combines local and global relation aggregation to address video redundancy and long-range dependency. Its blocks combine dynamic position embedding, multi-head relation aggregation, and feed-forward processing, with local operators in shallow layers and global affinity learning in deep layers.
- 3.1 OVERVIEW OF UNIFORMER BLOCK: UniFormer blocks combine Dynamic Position Embedding, Multi-Head Relation Aggregator, and Feed-Forward Network modules.The architecture is hierarchically stacked into a video network.
- 3.3 DYNAMIC POSITION EMBEDDING: DPE dynamically integrates 3D position information into every token, using local convolution to encode spatiotemporal order.Its shared parameters and locality support arbitrary input lengths, while zero padding helps border tokens encode absolute positions.
- 3.2 MULTI-HEAD RELATION AGGREGATOR: MHRA unifies 3D convolution and spatiotemporal self-attention by learning local affinity in shallow layers and global affinity in deep layers.This design targets local redundancy and long-range dependency with different token-relation mechanisms.
- 3.2 MULTI-HEAD RELATION AGGREGATOR: Local MHRA aggregates nearby tokens through a learnable relative-position matrix, reducing redundant computation in small 3D neighborhoods.It can be instantiated as channel-separated spatiotemporal convolution with pointwise-depthwise-pointwise operations.
- 3.2 MULTI-HEAD RELATION AGGREGATOR: Global MHRA compares token content across the global video view to learn long-term dependency through joint spatiotemporal relation modeling.Unlike factorized spatial and temporal attention, it performs joint spatiotemporal learning.
4 EXPERIMENTS
Experiments evaluate UniFormer on Kinetics and Something-Something benchmarks, compare design choices, and examine testing, transfer, and visualization behavior. Results support a computation-accuracy balance through local shallow-layer and global deep-layer aggregation.
- Experimental Setup: Experiments use Kinetics-400, Kinetics-600, and Something-Something V1/V2 with dataset-specific sampling strategies.Kinetics uses dense sampling, while Something-Something uses uniform sampling.
- Comparison to State-of-the-Art: 42× fewer GFLOPs and 1.0% higher performance than SlowFast are reported for UniFormer-S16f on both Kinetics datasets.UniFormer also achieves slightly better results than MoViNet with fewer input frames.
- Comparison to State-of-the-Art: 61.0% top-1 accuracy is achieved on Something-Something V1, while 71.2% is achieved on V2.These results are reported as 4.2% higher than TDNEN and 1.6% higher than Swin-B, respectively.
- Ablation Studies: Joint spatiotemporal attention outperforms divided attention and supports better transfer learning as pre-training expands from ImageNet to Kinetics-400.The joint design is more powerful than separate spatial and temporal attention in the reported ablations.
- Ablation Studies: Dynamic position embedding improves top-1 accuracy by 0.5% on ImageNet and 1.7% on Kinetics-400.The reported role is preserving spatiotemporal order.
- Ablation Studies: Local MHRA is used in the first two stages and global MHRA in the last two to balance computation and accuracy.Only local MHRA is computationally light but lacks long-term dependency modeling, while omitting local aggregation harms detailed representation learning.
- Ablation Studies: Multi-clip testing is preferred for Kinetics, whereas multi-crop testing is preferred for Something-Something.The comparison attributes this difference to scene coverage for Kinetics and temporal characteristics for Something-Something.
5 CONCLUSION
The conclusion presents UniFormer as a concise transformer that combines 3D convolution and spatiotemporal self-attention. Its local and global relation aggregation targets redundancy and dependency across network depth.
- 5 CONCLUSION: UniFormer unifies 3D convolution and spatiotemporal self-attention in a concise transformer format.The design addresses video redundancy and dependency.
- 5 CONCLUSION: Local MHRA in shallow layers reduces computation by aggregating nearby context, while global MHRA in deep layers learns global token relations.The architecture therefore assigns local and global affinity to different depths.
- 5 CONCLUSION: UniFormer achieves a preferable balance between accuracy and efficiency on Kinetics-400/600 and Something-Something V1/V2.The conclusion bases this claim on extensive experiments across these video benchmarks.
C ADDITIONAL IMPLEMENTATION DETAILS
Implementation details specify normalization, optimization, warm-up, and training schedules for the UniFormer variants across Kinetics and Something-Something.
- Architecture Details: Local MHRA uses batch normalization, global MHRA uses layer normalization, and downsampling layers add extra layer normalization.The architecture uses pre-normalization within residual functions.
- Training Details: Training uses AdamW with a cosine learning-rate schedule and 5 or 10 warm-up epochs.Warm-up is used to address early optimization difficulty.
- Training Details: UniFormer-S and UniFormer-B use distinct epoch, stochastic-depth, weight-decay, and learning-rate settings for Kinetics and Something-Something.The supplied implementation details specify separate schedules by model variant and dataset.
D VISUALIZATION
Visualization compares LLGG, LLLL, and GGGG structures using attention maps and category accuracies. The evidence links local aggregation to detail-sensitive categories and global aggregation to long-range object interactions.
- Visualization Setup: The visualization compares LLGG, LLLL, and GGGG structures with different stage-number configurations.The configurations are {3, 4, 8, 3}, {3, 5, 10, 4}, and {2, 2, 7, 3}, respectively.
- Attention Visualization: Grad-CAM visualizations use Kinetics-400 videos and display attention generated in the last layer.The comparison examines where each structure focuses.
- Attention Visualization: LLGG focuses more precisely on discriminative objects than GGGG and LLLL in the illustrated skateboard and football examples.GGGG struggles with blind all-token comparison, while LLLL lacks a global view.
- Category Analysis: LLLL performs better on detail-sensitive categories, whereas GGGG performs better when classification requires long-range relations between objects.Examples include gargling versus brushing teeth, swing dancing, and playing versus strumming guitar.
- Category Analysis: UniFormer’s LLGG structure remains competitive across the analyzed categories by combining local detail encoding with global dependency modeling.The category analysis attributes this competitiveness to using both types of context.
E.1 MORE RESULTS ON KINETICS
On Kinetics-400 and Kinetics-600, sampling choices produce different outcomes for single-clip and multi-clip testing. The authors therefore adopt frame stride 4 by default for multi-clip evaluation.
- Larger frame strides improve single-clip testing because sparser sampling covers a larger temporal range.
- Frame stride 4 consistently performs better for multi-clip testing across Kinetics-400 and Kinetics-600.
- The authors adopt frame stride 4 as the default sampling choice for multi-clip testing.
E.2 MORE RESULTS ON SOMETHING-SOMETHING
On Something-Something V1 and V2, pretraining and model configuration affect transfer performance. Larger-dataset pretraining benefits UniFormer-S more clearly, while the larger model can outperform the smaller model despite using fewer frames.
- Approximately 1.5% top-1 accuracy improvement results when UniFormer-S uses Kinetics-600 rather than Kinetics-400 pretraining.
- For UniFormer-B, Kinetics-600 pretraining produces no obvious improvement over Kinetics-400 pretraining.
- UniFormer-B with 16 frames performs better than UniFormer-S with 32 frames.
E.3 COMPARSION TO STATE-OF-THE-ART ON IMAGENET
The ImageNet experiments compare UniFormer variants with state-of-the-art models grouped by parameter count. UniFormer outperforms similarly sized or computationally comparable methods, including CNN–Transformer hybrids.
- Table 7 compares UniFormer with state-of-the-art ImageNet models and reports four designed model variants.
- UniFormer models outperform methods with similar parameters or FLOPs on ImageNet, especially when trained with Token Labeling.
- UniFormer surpasses CNN–Transformer combinations such as CvT and CoAtNet on ImageNet.