Source-linked AI summary
STEP: Spatial Temporal Graph Convolutional Networks for Emotion Perception from Gaits
Uttaran Bhattacharya, Trisha Mittal, Rohan Chandra, Tanmay Randhavane, Aniket Bera, Dinesh Manocha
TL;DR
Emotion perception from gait is relatively understudied, and annotated gait videos are difficult to collect at scale. STEP classifies perceived emotion from walking-video gaits and uses STEP-Gen to generate synthetic annotated gaits. It reports higher accuracy than prior learning-based and gait-based emotion-recognition methods.
Problem
Automatic emotion perception from gaits has been limited, primarily because emotion-annotated gait data are scarce and difficult to collect at scale.
Source: §1 · pp. 1–2Method
STEP classifies perceived emotion from walking-video gaits, while STEP-Gen uses a CVAE with push-pull regularization to generate annotated synthetic gaits.
Source: §1 · p. 2, §3.2 · p. 3Results
Takeaways & Limitations
E-Gait includes 2,177 real-world and 4,000 synthetic gait videos annotated with emotion labels.
Source: §1 · p. 2The generative model is limited to single-person gait sequences, and classification accuracy is governed by video quality and pose-extraction quality.
Source: §6 · p. 9
Abstract
from arXiv · showhide
We present a novel classifier network called STEP, to classify perceived human emotion from gaits, based on a Spatial Temporal Graph Convolutional Network (ST-GCN) architecture. Given an RGB video of an individual walking, our formulation implicitly exploits the gait features to classify the emotional state of the human into one of four emotions: happy, sad, angry, or neutral. We use hundreds of annotated real-world gait videos and augment them with thousands of annotated synthetic gaits generated using a novel generative network called STEP-Gen, built on an ST-GCN based Conditional Variational Autoencoder (CVAE). We incorporate a novel push-pull regularization loss in the CVAE formulation of STEP-Gen to generate realistic gaits and improve the classification accuracy of STEP. We also release a novel dataset (E-Gait), which consists of $2,177$ human gaits annotated with perceived emotions along with thousands of synthetic gaits. In practice, STEP can learn the affective features and exhibits classification accuracy of 89% on E-Gait, which is 14 - 30% more accurate over prior methods.
1. Introduction
Emotion perception from walking gaits is understudied partly because emotion-annotated gait videos are scarce, although gait movement offers a non-facial cue. STEP classifies four perceived emotions, while STEP-Gen supplies synthetic gaits; on E-Gait, STEP improves accuracy over prior methods.
1. Introduction
- Emotion perception has largely relied on facial, speech, and physiological cues, while facial cues can be unreliable in some contexts.The introduction motivates other non-verbal cues when self-reported emotions are unreliable or misleading.Source: §1 · p. 1
- 14% accuracy increase over prior learning-based method [38] and 21–30% absolute improvement over prior gait-based emotion-recognition methods.The reported comparison is on E-Gait.Source: Figure 1 · p. 1, §1 · p. 2
- Gait movement includes affective features such as arm swinging, stride length, posture, and collapsed upper body.Prior psychology work reported that observers identified sadness, anger, happiness, and pride from such features.Source: §1 · pp. 1–2
- Automatic emotion perception from gaits has been limited, primarily because emotion-annotated gait data are scarce and difficult to collect at scale.The introduction notes that compiling thousands of annotated real-world gait videos is challenging.Source: §1 · p. 2
- STEP classifies perceived emotion from walking-video gaits, while STEP-Gen generates annotated synthetic gaits to improve STEP accuracy.The paper reports that the synthetic gaits improve STEP accuracy by 6% in its benchmarks.Source: §1 · p. 2
- E-Gait contains 2,177 real-world and 4,000 synthetic gait videos; STEP achieves 87% classification accuracy on the dataset’s four emotion labels.The labels are angry, sad, happy, and neutral.Source: §1 · p. 2
2. Related Work
Prior emotion-perception work draws mainly on face, speech, and physiological signals, with a smaller body of gait-based classifiers. Gait generation research offers video and pose-generation approaches; this work models gaits as skeletal graphs and uses spatial-temporal graph convolutions inside a VAE.
2. Related Work
- Emotion-perception methods have used facial features, speech features, and physiological signals, with some work combining face and speech.The paper presents gait-based emotion perception as complementary to these approaches.Source: §2 · p. 2
- Gait-emotion classifiers have used PCA, SVMs, autocorrelation matrices, and symmetric positive definite representations of joint movements.The cited examples include Karg et al., Crenn et al., Venture et al., and Daoudi et al..Source: §2 · p. 2
- Gait generation includes GAN- and VAE-based methods; this work represents gaits as skeletal graphs and uses spatial-temporal graph convolutions inside a VAE.The related work also discusses action recognition from gaits and prior video- and pose-generation approaches.Source: §2 · pp. 2–3
3. Background
The background introduces graph and temporal graph convolutions alongside conditional variational autoencoders. These provide the representation and generation foundations used by the paper’s methods.
3.1. GCN and ST-GCN
- A graph convolutional network propagates feature representations across graph nodes using an adjacency matrix, learned weights, and a nonlinear activation.Its input is a feature matrix, and its output is another feature matrix; multiple kernels can produce a feature tensor.Source: §3.1 · p. 3
- ST-GCN represents a video’s skeletal poses as one graph, with spatial edges connecting body joints and temporal edges connecting each joint across time.Each vertex represents a joint, with its 3D position as the feature; the representation has a fixed adjacency matrix.Source: §3.1 · p. 3
3.2. Conditional Variational Autoencoder
- A variational autoencoder uses an encoder to map training data to a lower-dimensional latent distribution and a decoder to generate similar synthetic data.The account frames this generation approach in terms of Bayesian inference.Source: §3.2 · p. 3
- A conditional VAE learns separate latent distributions for training classes, allowing its decoder to generate samples conditioned on a given class.Under the stated Gaussian-decoder assumption, the class negative log likelihood is represented by an MSE loss.Source: §3.2 · p. 3
- The paper adds a push-pull regularization loss to the standard CVAE loss used for class-conditioned generation.The passage identifies this as a novel addition and refers to its later description in Section 5.3.Source: §3.2 · p. 3
4. STEP and STEP-Gen
STEP represents walking as a sequence of view-normalized skeletal graphs and uses spatial-temporal graph networks for gait generation and emotion classification. Its classifier combines learned gait representations with posture and movement features.
4.1. Extracting Gaits from Videos
- Walking videos are converted to 3D skeletal sequences and view-normalized; a shared edge set gives gaits with different joint positions a common adjacency matrix.The graph connects skeletal joints spatially and tracks each joint across time.Source: §4.1 · pp. 3–4, Figure 2 · p. 4
4.2. STEP-Gen: The Generation Network
- STEP-Gen is a CVAE whose encoder maps labeled gaits to a latent distribution and whose decoder samples that space to produce synthetic gaits with matching dimensions.The encoder uses ST-GCN layers, pooling, and convolutions; the decoder uses deconvolution, repetition, and ST-GDCN layers.Source: §4.2 · p. 4
- Push-pull regularization encourages synthetic gaits to retain frame-to-frame movement while limiting drift from real gaits, using the first, middle, and last frames as anchors.The modified loss is claimed to satisfy the ELBO bound under a classwise mixture-of-Gaussians decoder assumption.Source: §4.2 · p. 5
4.3. STEP: The Classification Network
- STEP appends 29-dimensional posture and movement features to Baseline-STEP’s learned representation, then uses fully connected layers and Softmax to predict four emotion labels.The affective features include joint angles and distances, body areas, joint velocities, and accelerations.Source: §4.3 · pp. 5–6
5. Experiments and Results
Experiments evaluate STEP and STEP-Gen on E-Gait, comparing classifier and generator variants and examining augmentation and robustness. STEP+Aug achieves the strongest reported classification accuracy, while additional experiments assess generated-gait quality and learned features.
5.1. Training Parameters
- STEP trains for 500 epochs and STEP-Gen for 150 epochs, using batch size 8 and Adam with initial learning rate 0.1.Both use momentum 0.9 and weight decay 5 × 10−4.Source: §5.1 · p. 6
5.2. Dataset: Emotion-Gait
- E-Gait contains 2,177 real gaits and 1,000 synthetic gaits for each of the four emotion classes.The real gaits include 342 collected by the authors and 1,835 from the Edinburgh Locomotion MOCAP Database, labeled by domain experts.Source: §5.2 · p. 6
5.3. Evaluation Metrics
- Generation is evaluated with FID, which measures generated-sample closeness to real inputs while retaining diversity; classifiers are evaluated by accuracy.The accuracy measure is reported as (TP + TN)/TD.Source: §5.3 · p. 6
5.4. Evaluation Methods
- Comparisons include vid2vid, baseline CVAE, prior gait-emotion and action-recognition classifiers, and Baseline-STEP, hybrid STEP, and STEP+Aug variants.Baseline-STEP uses network-learned gait features; STEP combines these with affective features, and STEP+Aug also trains on synthetic gaits.Source: §5.4 · pp. 6–7
5.5. Results on E-Gait
- STEP-Gen has an FID of 11.11 versus 11.33 for Baseline-CVAE; vid2vid's FID of 0 reflects memorization, while STEP-Gen converges in about 28 epochs and generates diverse, realistic gaits.Lower FID indicates higher fidelity; the baseline CVAE fails to converge after 150 epochs.Source: §5.5 · p. 7
- 89% accuracy for STEP+Aug, compared with 83% for STEP and 78% for Baseline-STEP; most prior gait-emotion methods score below 60%.The two compared action-recognition methods score 65% and 75%, respectively.Source: Table 1 · p. 7
- STEP's 95% confidence interval is [81.19, 85.33]; STEP+Aug exceeds 75% accuracy for each emotion class across 3,177 gaits.The interval is reported with a standard deviation of 1.96.Source: §5.5 · p. 8, Figure 6 · p. 8
5.6. Overfitting Analysis
- Synthetic augmentation raises accuracy but requires exponentially more data for each percentage-point gain; saliency maps show coordinated alternating limb movement in positive happy examples.The displayed saliency map shows only gradient magnitude along the walking direction; the paper notes that spine movements along other axes are not captured there.Source: §5.6 · p. 8
6. Limitations and Future Work
The generative model is limited to single-person gait sequences, and classification accuracy depends on video quality and pose extraction. Future work would extend the approach to crowds, improve generators, and combine gait with face or speech cues.
6. Limitations and Future Work
- The generative model is limited to generating gait sequences for a single person.The authors propose extending it to multi-person or crowd videos.Source: §6 · p. 9
- Classification accuracy is governed by video quality and the pose extraction algorithm.Source: §6 · p. 9
- The authors identify better generators as a future need for improving classification accuracy.They cite the complexity of generating annotated real-world videos.Source: §6 · p. 9
- Combining gait-based emotion classification with facial-expression or speech modalities is proposed as future work.Source: §6 · p. 9