Spatial-temporal Transformer-guided Diffusion based Data Augmentation for Efficient Skeleton-based Action Recognition

1 Introduction

• Data scarcity from expensive motion capture and labeling makes training well-performing skeleton-based action recognition models difficult.Skeleton representations use 3D coordinate sequences and are described as robust to camera angle, illumination, and background variation.Source: §1 · p. 1
• The proposed conditional model targets natural, spatial-temporal, class-controllable action sequences without noisy training data or restriction to a specific action domain.The paper presents it as an action-synthesis algorithm intended to improve recognition in data-scarcity situations.Source: §1 · p. 2
• A pretrained DDPM samples action sequences under guidance from a pretrained visual transformer, using an action label as the condition.The introduction identifies the transformer-guided diffusion approach and the spatial-temporal transformer's role in learning joint-position relations and guiding generation toward action labels.Source: §1 · p. 2
• The guiding strategy uses visual-transformer conditional guidance to eliminate dependence on the noisy latent and is reported as more effective and practical than existing SOTA methods.The authors state that this strategy contributes significantly to diffusion performance.Source: §1 · p. 2

Denoising diffusion probabilistic models

• DDPM research includes accelerated sampling with DDIM [39], critically-damped Langevin diffusion [40], likelihood improvements [32], and conditional-generation methods [36], but skeleton-action augmentation remains rare.The section describes DDPMs as rarely used for data augmentation in skeleton-based action recognition.Source: §2 · p. 2

Skeleton-based action recognition

• Skeleton-recognition methods include RNNs [44] [45], CNNs [49] [50] [51], and GCNs [12] [53] [13] [14] [16] [54], while few existing methods address data scarcity.The cited approaches cover temporal learning, skeleton encoding, graph-based joint relations, and related recognition designs.Source: §2 · pp. 2–3

Conditioned human motion generation

• Conditioned motion generation includes text-, audio-, and action-driven methods, with Action2Motion and ACTOR [63] addressing action-conditioned generation through VAE-based approaches.The paper distinguishes its approach as using image-synthesis techniques to generate natural human motion.Source: §2 · p. 3

3 DDPMs preliminaries

• The forward process gradually adds Gaussian noise to input x0 at each timestep, while reverse denoising samples intermediate latents step by step toward a clean sample.Figure 1 distinguishes the forward and reverse processes with blue and yellow arrows, respectively.Source: §3 · p. 3, Figure 1 · p. 3
• The forward diffusion kernel enables sampling an intermediate latent xt from the input x0.The passage introduces the kernel after describing Gaussian noise with variance βt at each timestep.Source: §3 · p. 3
• Because the reverse posterior q(xt−1|xt) is intractable, a learnable VAE pθ approximates it by predicting the mean and variance of xt−1 given xt.The preliminaries also note that parameterizing σ²t through variational-bound minimization was found to lead to faster convergence and more stable training.Source: §3 · pp. 3–4

4 Spatial-temporal Transformer-guided Diffusion-based Data Augmentation for Efficient Skeleton-based Action Recognition

• Given a class label, DDPM sampling starts from Gaussian noise and uses transformer gradients at each step to produce a target-class skeleton image convertible to 3D joints.Source: §4 · p. 4

4.1 Transformer-guided Diffusion

• The guidance estimates a clean image from each noisy latent, then applies cross-entropy between the estimate and target label to steer sampling.The loss is used to evaluate similarity between synthetic and target action-sequence distributions.Source: §4.1 · p. 4

4.2 Spatial-temporal transformer

• ST-Trans uses MobileViT, adapting MobileNetv2 and MobileViT blocks for tiny skeleton images to train faster and improve guidance performance.The visual-transformer choice leverages attention to learn relations among image patches.Source: §4.2 · p. 5
• The spatial-temporal transformer outperforms other state-of-the-art CNN-based classifiers when guiding diffusion across different metrics, and its compact design performs better for tiny skeleton images.Source: §4.2 · p. 5

5.1.1 Dataset

• NTU RGB+D evaluation uses subject- or setup-based splits, with 20% of training data reserved to pretrain the diffusion and transformer models; HumanAct12 contains 1,191 clips across 12 action categories.NTU RGB+D has 120 categories and 114,480 clips; HumanAct12 is an adjusted PH-SPD dataset with SMPL parameters.Source: §5.1.1 · p. 5

5.1.2 Evaluation metrics

• The evaluation framework covers both naturality/diversity and data-augmentation performance for action recognition.Source: §5.1.2 · p. 5

5.1.3 Evaluation metrics for naturality and diversity

• Naturality and diversity are measured with FID, pretrained-RNN action-recognition accuracy, and synthetic–real feature distances overall and per action.Lower FID indicates greater similarity to real data; higher recognition accuracy indicates a more similar distribution, and Table 1 marks 95% confidence intervals.Source: §5.1.3 · p. 5, Table 1 · p. 6

5.1.4 Evaluation metrics of data augmentation task for action recognition

• Action-recognition augmentation is evaluated on NTU RGB+D using cross-subject and cross-setup accuracy for comparability with other methods.Source: §5.1.4 · p. 5

5.2.1 Skeleton image representation

• Each action is encoded as a J × T × 3 matrix of joint coordinates, with J = T and zero-padded interpolation to a 32 × 32 input image.The square representation avoids resizing operations that could alter pixel values and supports lossless conversion back to joint coordinates.Source: §5.2.1 · p. 6

5.2.2 Experimental details

• The diffusion model and transformer are pretrained on HumanAct12 and NTU RGB+D for 500K AdamW iterations, using a 1e −4 learning rate, batch size 1024, and 1,000 diffusion steps.The study averages results over 20 random synthetic-dataset generations with 95% confidence intervals; action-recognition models follow their authors’ settings. Experiments use PyTorch on Ubuntu 18.04 with an Intel 9700K CPU and two Nvidia RTX Titans.Source: §5.2.2 · p. 6
• Because the study uses a different action representation and retrains competitors, its SOTA results may differ from the competitors’ original publications.Source: §5.2.2 · p. 6

5.3.1 Naturality and diversity evaluation

• The proposed method outperforms GAN-based methods and two VAE-based approaches on different naturality and diversity metrics.The authors attribute the gains to diffusion and spatial-temporal-transformer guidance.Source: §5.3.1 · p. 7
• The method is not best on some metrics, but shows higher robustness and lower fluctuation than other state-of-the-art approaches.Source: §5.3.1 · p. 7

5.3.2 Data augmentation evaluation

• Accuracy increases as more real training data is replaced, peaks at 20%-30% replacement, and decreases at 50%.The authors report that the synthetic data improves recent action-recognition models; at 50% replacement, their method compares favorably with two motion-generation competitors.Source: §5.3.2 · p. 7, Table 2 · p. 7
• Recognition performance rises as extra synthetic sequences are added, peaking at 40% added data; the proposed samples provide larger, more stable gains than competitors.The experiment mixes real and synthetic data across three state-of-the-art recognition methods.Source: §5.3.2 · pp. 7–8, Table 3 · p. 7

• Figure 3 presents eight samples selected from a single action sequence.Source: §5.4 · p. 8, Figure 3 · p. 8
• The “point to something” examples are described as natural and smooth, with almost no noise.Source: §5.4 · p. 8, Figure 3 · p. 8
• The “kick something” examples are described as realistic, with diverse and normal pose representations.Source: §5.4 · p. 8, Figure 3 · p. 8

• The spatial-temporal transformer outperforms the compared guiding classifiers, including EfficientNet, a U-Net classifier, and fine-tuned CLIP.The authors attribute its advantage to handling action sequences with rich spatial-temporal information.Source: §5.5 · p. 8, Table 4 · p. 8
• Fewer than 1,000 diffusion steps are insufficient for high-quality sequences, while more than 1,000 yield no obvious performance increase; the cosine schedule is more stable across metrics than the linear schedule.Source: §5.5 · p. 8, Table 4 · p. 8
• The guiding-strategy ablation reports a smaller model and faster convergence, while looser dependence on clean training data eases data preparation and model design.Source: §5.5 · p. 8, Table 5 · p. 8

6 Conclusions and Future Studies

• Across two data augmentation tasks, the synthetic sequences helped different SOTA action recognition approaches boost performance.The authors also report superior capability in synthesizing high-quality sequences compared with existing SOTA methods.Source: §6 · p. 9
• The model takes an action label and uses a spatial-temporal transformer to guide diffusion sampling of high-quality sequences with corresponding labels.The transformer classifies clean intermediate latent samples generated step by step from a Gaussian distribution.Source: §6 · p. 9
• The method is limited in generating long-period, consistent action sequences, which the authors identify as a future research direction.They propose investigating long-term action synthesis and further improving synthetic sequence quality.Source: §6 · p. 9