Source-linked AI summary
STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition
Dasom Ahn, Sangwon Kim, Hyunsu Hong, Byoung Chul Ko
TL;DR
Action recognition needs to combine video and skeleton features without relying on a separate cross-modal model or poorly balanced representations. STAR-transformer aggregates global-grid and joint-map tokens into multi-class tokens and processes them with full, zigzag, and binary spatiotemporal attention. Across benchmark datasets, it achieved substantial improvements over previous state-of-the-art methods, while the paper identifies high transformer computational cost as a limitation for action recognition.
Problem
Combining video and skeleton data can improve recognition, but cross-modal fusion requires a separate submodel, while transformer self-attention has high computational cost for action recognition.
Method
STAR-transformer aggregates video and skeleton features into multi-class tokens and learns their representation with full, zigzag, and binary spatiotemporal attention modules.
Results
STAR-transformer achieved substantial improvements compared with previous state-of-the-art methods on Penn-Action and NTU-RGB+D action datasets.
Takeaways & Limitations
The method provides a multi-feature representation that combines video RGB, skeleton, and joint-trajectory information for action recognition.
Takeaways & Limitations
Transformer-based action recognition remains constrained by the high computational cost of self-attention when modeling long-range frames and changing multi-feature relationships.
Abstract
from arXiv · showhide
In action recognition, although the combination of spatio-temporal videos and skeleton features can improve the recognition performance, a separate model and balancing feature representation for cross-modal data are required. To solve these problems, we propose Spatio-TemporAl cRoss (STAR)-transformer, which can effectively represent two cross-modal features as a recognizable vector. First, from the input video and skeleton sequence, video frames are output as global grid tokens and skeletons are output as joint map tokens, respectively. These tokens are then aggregated into multi-class tokens and input into STAR-transformer. The STAR-transformer encoder layer consists of a full self-attention (FAttn) module and a proposed zigzag spatio-temporal attention (ZAttn) module. Similarly, the continuous decoder consists of a FAttn module and a proposed binary spatio-temporal attention (BAttn) module. STAR-transformer learns an efficient multi-feature representation of the spatio-temporal features by properly arranging pairings of the FAttn, ZAttn, and BAttn modules. Experimental results on the Penn-Action, NTU RGB+D 60, and 120 datasets show that the proposed method achieves a promising improvement in performance in comparison to previous state-of-the-art methods.
1. Introduction
Action recognition benefits from combining video and skeleton data, but cross-modal fusion and efficient spatiotemporal modeling remain challenging. STAR-transformer addresses these issues by aggregating cross-modal tokens and using specialized attention modules, with experiments showing better performance than existing state-of-the-art models.
- Motivation: Cross-modal action recognition can improve performance, but combining video and skeleton data requires a separate cross-modal submodel and remains ambiguous.Video-only methods suffer from environmental noise, while skeleton-based methods depend on skeleton-extraction accuracy and overlap.
- Motivation: Vision Transformers model spatial relations effectively, but action recognition requires long-range temporal and changing multi-feature relationships at high computational cost.The limitation arises from applying multi-head self-attention to action-recognition features.
- Proposed approach: STAR-transformer aggregates video and skeleton features into multi-class tokens and replaces standard multi-head attention with a spatiotemporal cross-attention mechanism.Its encoder and decoder use zigzag and binary skip STAR attention to learn cross-modal features.
- Proposed approach: The model receives global grid and joint map tokens from shared-CNN processing, aggregates them, and produces separable multi-class features for downstream action recognition.The encoder–decoder output is used as input to the downstream action-recognition network.
- Results: Various benchmark experiments showed that STAR-transformer performed better than existing state-of-the-art models.The introduction reports performance evaluation across several benchmark datasets.
2. Related Work
Action-recognition research includes video-only, skeleton-based, cross-modal, and transformer-based approaches. These methods trade off robustness, pose-extraction dependence, fusion complexity, temporal modeling, and computational cost, while transformer-based cross-modal token coupling remains underdeveloped.
- Video and image-based action recognition: Video-based methods recognize actions from sequential or still images but can be slowed by long videos and degraded by environmental noise.Video frames can be processed with a single model, but background and other wild conditions affect performance.
- Skeleton-based action recognition: Skeleton-based methods model spatiotemporal joint coordinates with GCNs, 3D-CNNs, or CNNs and are less affected by background and lighting changes.Their recognition performance depends substantially on the pose extractor.
- Video and skeleton-based action recognition: Cross-modal methods fuse video and skeleton information into discriminative features and can outperform unimodal features, but their cross-modal subnetwork design remains problematic.The cited VPN approach uses cross-modal features and knowledge distillation to infuse poses into RGB streams.
- Transformer-based action recognition: Transformer-based action recognition supports long-range temporal modeling, but self-attention over many 3D video tokens incurs high computational cost.Existing transformer approaches mostly use video tokens, with relatively few using skeleton tokens.
- Transformer-based action recognition: Transformer-based coupling of cross-modal information had not yet been developed for spatiotemporal input tokens without separate submodels.The paper positions its approach as the first attempt in this setting.
3. Approach
STAR-transformer combines video and skeleton features through global grid and joint map tokens, then models them with paired full, zigzag, and binary spatio-temporal attention modules.
- Cross-Modal Learning: Video frames produce global grid tokens from last-layer features, while local features combined with joint heat maps produce joint map tokens.The two token types represent visual features and local features of individual skeleton joints, respectively.
- Cross-Modal Learning: Joint map tokens are extracted by projecting each joint onto heat maps and aggregating corresponding local feature-map values.The process is repeated across poses to obtain temporal JM-tokens.
- Cross-Modal Learning: Multi-class token aggregation concatenates GG-token and JM-token class tokens with a total class token to learn cross-modal feature representations.Joint position information is added to JM-tokens but not GG-tokens.
- STAR-transformer Encoder and Decoder: FAttn applies attention across all spatio-temporal tokens, with complexity O(T^2S^2), whereas ZAttn and BAttn divide temporal tokens into two groups.The temporal grouping reduces ZAttn and BAttn complexity to 0.25-fold relative to FAttn, or O(1/4T^2S^2).
- STAR-transformer Encoder and Decoder: ZAttn uses zigzag grouping to model action changes, while BAttn uses binary front–rear grouping to learn changes at the beginning and end of actions.The encoder and decoder pair these spatio-temporal mechanisms with FAttn; decoupling denotes zigzag or binary grouping.
4. Experimental Results
Experiments across Penn-Action and NTU-RGB+D evaluate STAR-transformer’s datasets, comparisons, token design, layer count, and spatio-temporal attention. Results support multi-class tokens and cross-modal attention, while showing overfitting beyond four layers and more distributed frame importance with ZAttn and BAttn.
- Experiment setup: Experiments used Penn-Action, NTU-RGB+D 60, and NTU-RGB+D 120, with NTU evaluations reported under XSub, XView, and XSet protocols.The datasets contain RGB, pose, and other multimodal action information at different scales.
- Comparison with state-of-the-art methods: 97.9% accuracy was achieved on Penn-Action, exceeding UNIK by 0.8% and Pr-VIPE by 1.2%.The comparison included methods using RGB and pose together as well as pose-only methods.
- Comparison with state-of-the-art methods: On NTU-RGB+D, multimodal RGB-and-pose methods generally outperformed unimodal pose methods, while STAR-transformer reached 90.3% on NTU120 XSub and 92.7% on XSet.PoseC3D exceeded STAR-transformer on NTU60 XSub by 5% and NTU120 XSet by 3.7%, using an additional pre-trained pose model.
- Ablation study: The proposed multi-class token performed 1.4% higher than a single-class token under the same cross-modal condition.The experiment attributes the gain to more effective learning between cross-modal tokens.
- Ablation study: Performance improved as transformer layers increased, but models with more than four layers overfit; three layers were therefore selected.The layer-count study was conducted on Penn-Action.
- Ablation study: FAttn concentrated importance in late frames, whereas ZAttn and BAttn assigned high importance across frames representing before, during, and after the action.These results motivated using F-Z for the encoder and F-B for the decoder.
5. Conclusion
The paper proposes STAR-transformer for jointly representing video and skeleton features in action recognition. Its multi-class tokens combine RGB, skeleton, and joint-trajectory information, while experiments on Penn-Action and NTU-RGB+D report improvements over prior state-of-the-art methods.
- STAR-transformer jointly uses video frames and skeleton-based features through a spatial-temporal cross-attention module.
- Multi-class tokens flexibly combine RGB video frames, skeletons, and joint trajectories for multi-feature representation learning.
- Experiments on Penn-Action and NTU-RGB+D confirmed substantial improvements over previous state-of-the-art methods.