Source-linked AI summary
Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition
Vittorio Mazzia, Simone Angarano, Francesco Salvetti, Federico Angelini, Marcello Chiaberge
TL;DR
The paper addresses the need for accurate, low-latency HAR from short temporal windows. It introduces AcT, a fully self-attentional Transformer using 2D pose sequences, and evaluates it with the MPOSE2021 benchmark, reporting strong efficiency and comparative performance for real-time use. The paper also identifies limited exploitation of relational information between pose points as a direction for future work.
Problem
Existing HAR commonly relies on longer temporal sequences and architectures combining convolutional, recurrent, and attention layers, whereas real-time applications require classification from short near-past windows.
Method
The paper introduces AcT, a pure Transformer encoder that processes 2D pose sequences with positional embeddings and class-token supervision, evaluated on MPOSE2021.
Results
AcT significantly outperforms commonly adopted HAR models while providing an accurate, low-latency solution for short-time real-time recognition.
Takeaways & Limitations
MPOSE2021 establishes an open benchmark for short-time HAR, while AcT’s efficiency supports deployment on computationally limited devices such as Edge AI systems.
Takeaways & Limitations
The methodology scarcely exploits relational information between estimated pose points, and future work is proposed to incorporate skeleton graphs as prior knowledge.
Abstract
from arXiv · showhide
Deep neural networks based purely on attention have been successful across several domains, relying on minimal architectural priors from the designer. In Human Action Recognition (HAR), attention mechanisms have been primarily adopted on top of standard convolutional or recurrent layers, improving the overall generalization capability. In this work, we introduce Action Transformer (AcT), a simple, fully self-attentional architecture that consistently outperforms more elaborated networks that mix convolutional, recurrent and attentive layers. In order to limit computational and energy requests, building on previous human action recognition research, the proposed approach exploits 2D pose representations over small temporal windows, providing a low latency solution for accurate and effective real-time performance. Moreover, we open-source MPOSE2021, a new large-scale dataset, as an attempt to build a formal training and evaluation benchmark for real-time, short-time HAR. The proposed methodology was extensively tested on MPOSE2021 and compared to several state-of-the-art architectures, proving the effectiveness of the AcT model and laying the foundations for future work on HAR.
I. INTRODUCTION
The paper targets short-time HAR for real-time applications and introduces AcT, a fully self-attentional pose-based model, alongside the MPOSE2021 benchmark. It evaluates the architecture, model scales, temporal reduction, introspection, and latency to assess accuracy and practical suitability.
- Short-time HAR classifies actions from up to a second of near-past information, supporting prompt responses in real-time applications such as robotics.
- The paper introduces AcT, a pure Transformer encoder architecture for pose-based action recognition.
- AcT uses 2D pose representations and is designed as an accurate, low-latency model for real-time applications.
- The study examines model scale, parameter count, attention heads, self-attention behavior, reduced temporal input, and latency on low-power devices.
- The authors claim that fully self-attentional architectures can outperform existing convolutional and recurrent models for pose-based HAR.
- The paper proposes MPOSE2021, a short-time HAR dataset with 15429 sequences from 100 actors performing 20 actions.
II. RELATED WORKS
Prior HAR work commonly used video or depth data and architectures combining convolutional, recurrent, graph, or attention components. This paper differs by pairing an entirely Transformer-based model with a short-sequence 2D pose benchmark for low-latency evaluation.
- HAR methods use video-based 2D pose data or depth-based 3D skeletal data, with 2D approaches applicable using a simple RGB camera.
- Earlier short-time pose-based work used MLSTM-FCN, combining 1D convolutions, LSTM, and squeeze-and-excitation attention.
- Related methods include hierarchical spatial-temporal networks, bone- and joint-based graph convolutional ensembles, and self-attention models for skeletal HAR.
- Unlike previous methodologies, AcT uses the Transformer encoder without convolutional or recurrent layers.
- MPOSE2021 limits samples to at most 30 frames, providing a benchmark for short-time and low-latency HAR rather than long-sequence classification.
III. THE MPOSE2021 DATASET
MPOSE2021 is an RGB-based, pose-oriented benchmark constructed from heterogeneous precursor datasets and standardized into short clips. Its actor splits, pose-detector annotations, and preprocessing resources support comparative evaluation of short-time HAR.
- MPOSE2021 combines videos from eight precursor datasets and presents them as an RGB-based dataset for short-time pose-based HAR.
- Clips are standardized to 30 frames when possible, while tail samples longer than 20 frames are retained.
- After removing or relabeling unsuitable clips, the dataset contains 15429 samples, each showing one actor performing one action, across 100 actors.
- OpenPose and PoseNet extract landmarks, and their average confidence varies across precursor datasets.
- Three actor-based splits and ten validation folds per split are used to assess accuracy and robustness across different methodologies.
- The released repository and Python package provide access to videos, RGB frames, 2D poses, visualization, and preprocessing functions.
IV. ACTION TRANSFORMER
AcT processes short sequences of 2D human poses with a fully Transformer-based architecture, producing class predictions from a learned [CLS] representation. Its design supports adaptive inference and lightweight, low-latency action recognition.
- AcT receives 2D pose sequences, projects each pose into Dmodel dimensions, and adds learnable [CLS] and positional embeddings.Each detected person is processed separately, while multiple poses can be batch-processed at inference time.
- The projected tokens pass through L Transformer encoder layers, after which only the [CLS] token enters the linear classification head.The encoder produces a representation for the full sequence, but supervision and prediction use the [CLS] token.
- AcT can accept fewer frames at inference time than the fixed training length, making its predictions more adaptive to available temporal input.The architecture permits reduced-length inputs without requiring retraining with a new fixed T.
- The resulting network is lightweight and supports accurate, low-latency, energy-efficient predictions for multiple people in video streams.The efficiency benefits come from using 2D pose representations rather than processing full RGB video frames.
B. Transformer Architecture
The Transformer encoder alternates multi-head self-attention and feed-forward blocks, correlating information across the temporal sequence. AcT variants scale heads, layers, and feed-forward dimensions while preserving a fixed per-head model dimension.
- The encoder alternates H-head self-attention and feed-forward blocks across L layers, with dropout, layer normalization, and residual connections after each block.This structure is summarized in the encoder architecture shown in Fig. 5.
- Each feed-forward block expands Dmodel to Dmlp = 4·Dmodel with GeLu activation, then projects back to Dmodel.The block is implemented as a two-layer multilayer perceptron.
- For each attention head, queries, keys, and values are computed from the input, and attention weights derive from pairwise query-key similarities.Each self-attention operation forms weighted sums over the value vectors.
- The outputs of all H heads are concatenated and linearly projected back to Dmodel, allowing attention to correlate different time windows.The mechanism operates in the time domain and produces a global class-embedding representation.
- AcT fixes Dmodel/H = 64 and varies H, Dmlp, and L to create micro, small, medium, and large model versions with increasing parameter counts.The versions differ through increasing numbers of attention heads and encoder layers.
V. EXPERIMENTS
The experiments compare four AcT variants with established baselines and state-of-the-art models on MPOSE2021, then examine attention, temporal reduction, and latency. The study also uses AcT ensembles for a fairer comparison with ensemble baselines.
- Experimental design: Four AcT variants are compared with state-of-the-art methodologies and baselines on MPOSE2021.The experiments include additional AcT-µ ensembles for comparison with ST-TR and MS-G3D.
- Experimental design: The experiments analyze attention behavior and performance under reduced temporal information.They also study model latency on two CPU types for real-time applications.
A. Experimental Settings
The experiments use OpenPose and PoseNet pose sequences with fixed temporal length and separate training, validation, and test partitions. AcT training uses AdamW with a scheduled learning-rate drop.
- Data and preprocessing: OpenPose samples use 13 keypoints and PoseNet samples use 17, each represented by positions and velocities over T = 30 frames.The resulting feature counts are P = 52 for OpenPose and P = 68 for PoseNet.
- Data and preprocessing: The dataset split contains 9421 training samples, 3141 validation samples, and 2867 test samples.Validation data are used for grid-search selection of promising hyperparameters.
- Training procedure: All models are trained with AdamW and a learning-rate drop to 1e-4 at 80% of the total epochs.The training setup uses the scheduling procedure described in the cited work, with the specified fixed drop percentage.
B. Action Recognition on MPOSE2021
AcT is benchmarked against lightweight baselines and more elaborate convolutional, recurrent, graph, and attention-based models on OpenPose and PoseNet representations. Across the reported comparisons, AcT variants perform strongly, while PoseNet is generally more difficult because of noisier detections.
- Benchmark setup: The benchmark compares MLP, Conv1D, REMNet, MLSTM-FCN, ActionXPose, MS-G3D, ST-TR, and AcT variants.The study reports mean and standard deviation across validation splits and evaluates the models on MPOSE2021 splits.
- Baseline results: The OpenPose results show that convolutional models outperform the MLP, while attention and residual blocks further improve REMNet accuracy.Explicitly modeling spatial and temporal correlations slightly improves action understanding for MLSTM-FCN and ActionXPose.
- OpenPose results: All four AcT versions outperform the other methodologies in the OpenPose experiments and show smaller standard deviations.Even AcT-µ with 227k parameters extracts general and robust temporal features.
- Ensemble comparison: AcT-µ (x2) outperforms MS-G3D (J+B) in all benchmarks except balanced accuracy on split 3, using less than one-tenth of its parameters.AcT-µ (x5) and AcT-µ (x10) achieve higher accuracy across all splits with roughly 1 to 2 million parameters.
- PoseNet results: PoseNet balanced accuracy is generally lower because noisy or missing keypoint detections produce less informative inputs.The passage reports that AcT still outperforms the other considered architectures and represents temporal correlations robustly.
C. Model Introspection
Model introspection shows that AcT concentrates attention on informative frames while distributing attention more broadly when gestures are less localized or predictions fail. Its accuracy degrades almost linearly as frames are removed, and its positional embeddings encode relative temporal structure.
- Attention visualization: AcT attends to specific frames when a gesture defines the action, but attention is more spread across frames in less coherent cases.Failed predictions are associated with attention weights that do not yield a valid global scene representation.
- Attention visualization: Bent poses receive higher attention because they are more informative for predicting the jumping-in-place action.The [CLS] token scores are computed by summing and normalizing the final-layer attention heads.
- Temporal robustness: AcT accuracy decreases with an almost linear trend when temporal frames are progressively dropped.The model permits this reduction without retraining or explicit adaptation, supporting applications with different temporal constraints.
- Positional embeddings: Nearby positional embeddings are highly similar, whereas distant positions are orthogonal or oppositely directed across all sequence frames.This pattern indicates that relative positions remain important because actions are not particularly localized in time.
D. Latency Measurements
Latency tests on Intel and ARM CPUs show that AcT variants provide strong computational efficiency, with AcT-µ combining particularly high speed with accuracy. Transformer-based models generally require less CPU effort than convolutional and recurrent alternatives.
- Latency was measured on an Intel i7-9700K and an ARM-based HiSilicon Kirin 970 using TensorFlow Lite benchmarks.The benchmark used 10 warm-up runs followed by 100 forward passes with 8 threads.
- Transformer-based architectures showed greater computational efficiency than convolutional and recurrent networks in the reported tests.The comparison covered model speed on both a high-performance Intel CPU and a mobile ARM CPU.
- On Intel, AcT-µ ran at over four times the speed of REMNet, while REMNet’s accuracy was 2% lower than AcT-S despite similar speed.
- AcT-µ outperformed the compared smaller models on both accuracy and speed, while MLSTM-FCN and ActionXPose were less accurate and slower.The reported comparison identifies AcT-µ as the strongest combined accuracy-speed option among these models.
- On ARM, AcT-µ was ten times quicker than REMNet and Conv1D, which had significantly lower accuracies than AcT-M.
VI. CONCLUSION
The paper concludes that AcT is an effective, fully self-attentional approach for accurate, low-latency action recognition using short 2D pose sequences. It also identifies Edge AI as a practical application and outlines extensions involving 3D skeletons, longer sequences, and relational pose information.
- AcT significantly outperforms commonly adopted HAR models while using a simple, fully self-attentional architecture.
- The methodology uses 2D skeletal representations of short time sequences to provide an accurate, low-latency solution for real-time applications.
- AcT’s efficiency could support Edge AI applications on computationally limited devices while maintaining good performance.
- Future work may investigate 3D skeletons, long sequence inputs, and skeleton graphs in positional embeddings.The paper notes that relational information between estimated pose points is currently scarcely exploited.