SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition

1 Introduction

• Skeletons from sensors provide a compact representation that is robust to external environmental changes and avoids additional pose- or optical-flow-estimation modules.The introduction contrasts this with RGB videos, which are sensitive to lighting, camera distance, and background variation and require substantial storage.Source: §1 · p. 2
• GCNs struggle to capture relations between physically distant joints, while transformer attention across every joint and frame is inefficient and may overlook action-critical skeletal-temporal relations.The introduction notes that separating skeletal and temporal attention can obscure these relations, while tokenization may dilute information.Source: §1 · p. 2
• SkateFormer partitions joints and frames by skeletal-temporal relation type and applies partition-specific self-attention, Skate-MSA, to capture those relations efficiently.The four types pair neighboring or distant joints with local or global motion.Source: §1 · pp. 2–3
• Skate-Embedding combines learnable skeletal features with fixed temporal index features through an outer product.The paper presents this embedding as enhancing action-recognition performance.Source: §1 · p. 3
• SkateFormer reports state-of-the-art action-recognition performance across 4-ensemble and single-modality settings, and a new state of the art in interaction recognition.The single modalities are joints, bones, joint motions, and bone motions.Source: §1 · p. 3

2.1 RNN/CNN-based approaches

• RNN-based approaches use LSTM and GRU for sequential skeleton data, while CNN-based approaches use pseudo-images or image-like skeleton representations.PoseC3D uses skeleton heatmaps generated from RGB-video pose estimates; Ske2Grid converts 2D/3D skeleton nodes into image-format grid patches.Source: §2.1 · p. 3

2.2 GCN-based approaches

• GCNs model joints and bones as graph vertices and edges, but fixed-sized kernels limit their ability to capture topology variations depending on input data.Other cited methods add compact latent spaces, contrastive learning, language supervision, or rooted-tree connections.Source: §2.2 · p. 4

2.3 Transformer-based approaches

• Transformer methods capture data-adaptive joint connections, but large attention maps are costly and separated skeletal and temporal modules can miss their joint relations.Joint-group tokenization can lose physically similar skeletal information; SkateFormer instead uses partition-specific skeletal-temporal attention at the joint-element level without tokenization.Source: §2.3 · p. 4

3.1 Overview of SkateFormer

• The model maps each skeleton sequence into higher-dimensional features, adds skeletal-temporal positional embeddings, processes them through R blocks, and pools the result for classification.Each block propagates features through skeletal-temporal relations before a feed-forward layer refines them.Source: §3.1 · p. 5

3.2 SkateFormer Block

• Each SkateFormer Block splits channels among graph convolution, temporal convolution, and Skate-MSA, then combines their outputs with a residual connection.The graph branch uses learnable joint-connectivity matrices, while the temporal branch uses grouped 1D convolutions; the block also includes a feed-forward network.Source: §3.2 · pp. 5–6

3.3 Skate-MSA

• Skate-MSA partitions neighboring or distant joints together with local or global motion, yielding four relation-specific attention branches.Local motion uses consecutive frames, while global motion samples frames at a stride; each branch is reversed to restore the feature map for combination.Source: §3.3 · pp. 6, 8
• Approximately 48× lower computational complexity than naive self-attention is reported for Skate-MSA in the experimental settings.The comparison is with naive self-attention over the full joint-frame feature map.Source: §3.3 · p. 9

3.4 Skeletal-Temporal Positional Embedding

• Temporal augmentation samples frames uniformly within a randomly selected interval, while inter-instance augmentation exchanges bone lengths across sequences to vary subject body sizes.The interval excludes sequence ends, and bone-length exchange is intended to support generalization learning.Source: §3.4 · p. 9
• Skate-Embedding forms skeletal-temporal positional features by taking the outer product of learnable joint features and fixed sinusoidal features for sampled temporal indices.The temporal indices are normalized to [−1, 1] to represent sampled frames from sequences of varying lengths.Source: §3.4 · p. 10

4.1 Datasets

• The evaluations use NTU RGB+D, NTU RGB+D 120, and NW-UCLA, including cross-subject, cross-view, and cross-setup protocols.The NTU interaction subsets comprise 11 classes from NTU RGB+D and 26 from NTU RGB+D 120.Source: §4.1 · pp. 10–11

4.2 Experiment Details

• Experiments train each model for 500 epochs with AdamW, batch size 128, and dataset-specific joint-count configurations.The listed configurations use T = 64 and L = 4 for all three datasets.Source: §4.2 · p. 11

4.3 Performance Comparison

• SkateFormer outperforms state-of-the-art methods except in E4 on X-Sub120, with relatively stronger performance in E1 and E2 than E4.The comparison covers NTU RGB+D, NTU RGB+D 120, and NW-UCLA; E1 uses joints, E2 joints plus bones, and E4 adds joint and bone motion.Source: §4.3 · pp. 11–12, Table 1 · p. 12, Table 2 · p. 12
• SkateFormer has parameter and FLOP counts comparable to GCN methods and substantially lower than transformer counterparts.The complexity comparison also reports inference time and average top-1 accuracy for joint modality.Source: §4.3 · p. 12, Table 3 · p. 13

4.4 Ablation Studies

• The full Skate-MSA configuration combining skeletal and temporal relation types achieves the highest accuracy among the tested Skate-Type variants.Skeletal-only and temporal-only relation types each improve over the baseline.Source: Table 4 · p. 13, §4.4 · p. 13
• In the Skate-Embedding ablation, among tested skeletal and temporal embedding combinations, learnable skeletal embedding paired with fixed temporal embedding achieves superior performance, suggesting a balance between embedding adaptability and stability.Source: §4.4 · p. 13
• Combining intra- and inter-instance augmentation reaches 92.6% on X-Sub60 and 97.0% on X-View60, while inter-instance augmentation alone drops X-View60 accuracy by 0.3 points.The passage attributes the X-View60 drop to limited bone-length variability across subjects in that setting.Source: §4.4 · p. 14, Table 6 · p. 14

• SkateFormer sets a new state of the art across single modalities and the four-modality ensemble, while combining partition-specific attention with skeletal-temporal positional embedding.The embedding forms an outer product of learnable skeletal features and fixed temporal index features.Source: §5 · p. 14

• The supplement includes a comparison of SkateFormer’s high-level novelty with conventional methods.The main paper states that detailed analyses are provided in the supplement because of space limits.Source: Supplementary Material · p. 20
• Supplementary analyses cover class-wise accuracy, classes where Skate-MSA improves performance, failure classes, and generalization to diverse modality inputs.Source: Supplementary Material · p. 20
• The supplement also lists Skate-MSA activation visualizations and computational-complexity analyses, including theoretical calculations and additional experiments.Source: Supplementary Material · p. 20

A Discussion on Various Partition-Based Approaches

• SkateFormer combines neighboring and distant joint partitions with local and global temporal partitions to form four skeletal-temporal relation types.Source: A Discussion on Various… · p. 20, Table 7 · p. 20
• Unlike partition-based tokenization, SkateFormer applies self-attention within each partition for fine-grained analysis.Source: A Discussion on Various… · p. 21
• SkateFormer captures skeletal-temporal relations at the joint-element level without tokenization, unlike approaches using separate skeletal and temporal attention or temporal convolution.Source: A Discussion on Various… · p. 21

B.1 Recognition Accuracy based on Action Labels

• ‘Wear a shoe’ and ‘take off a shoe’ rose from (60.81%, 78.83%) to (87.55%, 83.94%), respectively.Failure cases often involved fine finger motions, including reading, writing, and playing with a phone or tablet.Source: §B.1 · p. 21, Table 8 · p. 22
• 42 of 60 action classes improved with the full SkateFormer over baseline, eight were unchanged, and ten declined.The authors interpret this pattern as balancing gains on lower-ranked labels against slight losses on some higher-ranked labels.Source: §B.1 · p. 21, Table 8 · p. 22

B.2 Single Modality Comparision

• SkateFormer’s single-modality results significantly dominate existing state-of-the-art methods, particularly for J and B, and can match prior ensemble performance.The comparison covers NTU RGB+D and NTU RGB+D 120 evaluations.Source: Table 9 · p. 23, §B.2 · p. 23, Table 10 · p. 24

B.3 Analysis of Partition-specific Attention

• Skate-Type activations vary by action: Types 3 or 4 are stronger for sitting down and standing up, while Type 1 is prominent for reading and writing.The authors characterize this as action-adaptive partition-specific attention.Source: Figure 6 · p. 24, §B.3 · p. 24

C.1 Skeleton Tracking Indices and Labels of Joints

• NW-UCLA uses 20 Kinect v1 joints, while NTU RGB+D and NTU RGB+D 120 use 25 Kinect v2 joints, including five additional joints.The additions are the spine, both hand tips, and both thumbs.Source: §C.1 · pp. 25–26

C.2 Details of Neighboring Joint Partitions

• NW-UCLA’s 20 joints form five neighboring four-joint partitions; NTU uses K=12 on 48 considered joints per individual, forming four-joint partitions, and zero-pads 25 joints when only one individual is present.Equal-sized partitions were used because irregular input sizes increased the implementation complexity of reshaping and attention functions.Source: §C.2 · p. 26

D.1 Intra-instance Augmentation

• Skeletal augmentation applies random shear, rotation, scaling, flipping, coordinate dropout, joint dropout, and actor-order permutation.Random temporal flipping, Gaussian noise, and Gaussian blur were omitted because they degraded SkateFormer performance.Source: §D.1 · p. 27
• Trimmed-uniform sampling selects a random sequence portion and samples T frames; fixed temporal index features encode the resulting sampled indices.If the selected interval is shorter than T, linear interpolation produces T frames; inference uses p = 0.95 and a centered start index.Source: §D.1 · pp. 27–28

D.2 Inter-instance Augmentation

• Bone Length AdaIN exchanges bone lengths across different subjects’ frame sequences to diversify body sizes, with inter-instance augmentation applied at paug = 0.2.The procedure adapts input bone lengths using ratios derived from a reference sequence with the same action label.Source: §D.2 · pp. 28–29

E.1 Computational Complexity of Skate-MSA

• Skate-MSA splits features channel-wise and partitions them before attention, with each partition’s computation depending on its token dimensions rather than the full sequence dimensions.The appendix gives the naive self-attention cost and the summed Skate-MSA complexity in terms of the partition sizes.Source: §E.1 · pp. 29–30

E.2 Additional Experiments on the Computational Complexity

• 38.87× less FLOPs complexity was measured for Skate-MSA in the first two attention layers; the theoretical reduction there is 48×.The comparison concerns attention layers only, before downsampling, where the reduction effect is most pronounced.Source: Table 12 · p. 30, §E.2 · p. 30

F.1 Hyperparameters

• The model uses 32 attention heads, eight SkateFormer blocks, and temporal downsampling after every two blocks.The FFN expansion ratio is 1 for NW-UCLA and 4 for NTU RGB+D and NTU RGB+D 120.Source: §F.1 · pp. 30–31

F.2 Overall Architecture

• Table 13 presents SkateFormer’s overall architecture and the output shape of each module.The module details are identified in Table 13.Source: §F.2 · p. 31, Table 13 · p. 31

F.3 Modules

• G-Conv splits its input into eight channel-wise feature groups, applies a corresponding learnable V × V matrix to each, and concatenates the results.This process provides an inductive bias with respect to skeletal position.Source: §F.3 · pp. 31–32
• The SkateFormer Block’s FFN uses a linear-activation-linear sequence with a residual connection, expanding features by ratio e before squeezing them back to C channels.The expansion ratio is 1 for NW-UCLA and 4 for NTU RGB+D and NTU RGB+D 120.Source: §F.3 · p. 32
• Table 14 lists the symbols used in the module details.Source: Table 14 · p. 32