Source-linked AI summary
Towards a Deeper Understanding of Skeleton-based Gait Recognition
Torben Teepe, Johannes Gilg, Fabian Herzog, Stefan Hörmann, Gerhard Rigoll
TL;DR
Silhouette-based gait recognition can lose spatial detail, be difficult to obtain in real-world settings, and include appearance clues beyond gait. This paper uses a multi-branch GCN approach for skeleton-based gait recognition and reports a 3× improvement over the baseline on OUMVLP-Pose, alongside evidence that its model captures gait features.
Problem
Silhouette representations can lose fine-grained spatial information, be difficult to obtain in real-world settings, and contain appearance clues that are not actual gait features.
Source: §1 · pp. 1–2Method
Results
3× improvement over the baseline on OUMVLP-Pose, with training reported as faster and more stable than previous graph-based approaches.
Source: §5 · p. 8Takeaways & Limitations
Visualizations and ablations indicate that the model focuses on moving body parts and outer joints and captures real gait features rather than visual re-identification cues.
Source: §5 · p. 8Skeleton-based recognition does not yet match appearance-based methods in handling CASIA-B walking conditions; pose estimation may be less accurate for people wearing coats and bags.
Source: §4.4 · p. 6
Abstract
from arXiv · showhide
Gait recognition is a promising biometric with unique properties for identifying individuals from a long distance by their walking patterns. In recent years, most gait recognition methods used the person's silhouette to extract the gait features. However, silhouette images can lose fine-grained spatial information, suffer from (self) occlusion, and be challenging to obtain in real-world scenarios. Furthermore, these silhouettes also contain other visual clues that are not actual gait features and can be used for identification, but also to fool the system. Model-based methods do not suffer from these problems and are able to represent the temporal motion of body joints, which are actual gait features. The advances in human pose estimation started a new era for model-based gait recognition with skeleton-based gait recognition. In this work, we propose an approach based on Graph Convolutional Networks (GCNs) that combines higher-order inputs, and residual networks to an efficient architecture for gait recognition. Extensive experiments on the two popular gait datasets, CASIA-B and OUMVLP-Pose, show a massive improvement (3x) of the state-of-the-art (SotA) on the largest gait dataset OUMVLP-Pose and strong temporal modeling capabilities. Finally, we visualize our method to understand skeleton-based gait recognition better and to show that we model real gait features.
1. Introduction
Gait offers a way to identify people at a distance, but changing walking conditions and silhouette-based representations complicate recognition. The paper turns to skeleton-based GCNs to focus on motion features and reports strong performance on OUMVLP-Pose.
1. Introduction
- Gait describes limb movement during walking and can be observed at a distance without a person’s compliance.The introduction also notes that gait is hard to disguise and has potential for surveillance and forensic identification.Source: §1 · p. 1
- Changing carried items, clothing, or walking surfaces can alter gait, making features invariant to these influences a central recognition challenge.Footwear such as sneakers, boots, or heels may considerably change a person’s gait.Source: §1 · p. 1
- Silhouettes can be difficult to obtain in changing real-world backgrounds and can expose clothing, carried items, and other appearance clues.Background-subtraction artifacts may affect contours, and silhouette-based recognition can resemble visual re-identification.Source: §1 · pp. 1–2
- Skeleton-based methods focus on posture and temporal movement, while current model-based approaches still trail appearance-based methods.The paper introduces GCNs to help bridge that performance gap.Source: §1 · p. 2
- The paper proposes an efficient multi-branch graph-based GCN and reports state-of-the-art results by a large margin on OUMVLP-Pose.It also presents ablations and visualizations to examine learned features.Source: §1 · p. 2
2. Related Work
The related work reviews silhouette- and skeleton-based gait recognition alongside graph-based skeleton action recognition. It traces the field from silhouette methods to newer pose-based and GCN approaches.
2. Related Work
- The section surveys silhouette-based gait recognition, skeleton-based gait recognition, and related skeleton-based human-understanding work.Source: §2 · p. 2
2.1. Gait Recognition
- Silhouette methods still set the state of the art, while skeleton-based approaches have begun to challenge their lead.Figure 2 compares RGB frames with silhouette and skeleton gait representations from CASIA-B.Source: §2.1 · p. 2, Figure 2 · p. 2
2.1.1 Silhouette-based
- Silhouette methods range from single-image gait summaries to sequence-based temporal models and shuffled set-based approaches.Gait Energy Images summarize a cycle but lose most temporal information; sequence methods use 3D-CNNs or LSTMs, while set-based methods [4] model no temporal information.Source: §2.1.1 · p. 2
2.1.2 Skeleton-based
- Pose-based methods use keypoints for gait modeling, progressing from spatial-temporal networks and handcrafted 3D features to GCNs.PTSN combines a CNN and LSTM; PoseGait uses joint angles, bone lengths, and joint motion; GaitGraph [25] introduced GCNs for skeleton-based gait recognition, while a later hybrid [26] combines GCN and CNN branches.Source: §2.1.2 · pp. 2–3
2.2. Skeleton-based Action Recognition
- Skeleton-based action recognition supplied graph architectures used in this paper, including ST-GCN and ResGCN.ST-GCN [29] interleaves spatial graph and temporal convolutions; 2s-AGCN [17] adds adaptive adjacency and bone input, while ResGCN [22] uses residual connections and higher-order multi-branch inputs.Source: §2.2 · p. 3
3. Skeleton-based Gait Recognition
The method represents gait as a temporal graph sequence of keypoints and processes it with graph and temporal convolutions in a residual network. It uses joint positions, motion velocities, and bone geometry as inputs.
3.1. Notation
- Gait is represented as a sequence of graphs whose nodes are joints, edges are bones, and time-varying features form a tensor.The graph structure is encoded by an adjacency matrix, while the feature tensor stores each joint’s x/y coordinates and confidence.Source: §3.1 · p. 3
3.2. Graph Convolutions
- Graph convolutions update skeleton features by aggregating information over the graph, with self-loops preserving identity features.The update uses a normalized adjacency matrix and a learned transformation; aggregation is interpreted as messages from direct neighbors.Source: §3.2 · p. 3
3.3. Pose Estimation
- The authors obtain skeletons from RGB images using keypoint estimation, with HRNet-derived 2D keypoints for CASIA-B and provided keypoints for OUMVLP.The CASIA-B HRNet estimator is offline but yields higher keypoint accuracy than real-time approaches.Source: §3.3 · p. 3
3.4. Network Architecture
- The adapted ResGCN processes each block with spatial graph convolutions followed by temporal 2D convolutions, bottlenecks, and residual connections.Bottleneck layers reduce feature channels, and residual connections link features before and after each spatial and temporal block.Source: §3.4 · pp. 3–4
3.5. Multi-Branch Input
- The multi-branch inputs include joint positions relative to the pose center and motion velocities computed from the same joint across nearby frames.The velocity input uses differences between frames separated by one or two steps.Source: §3.5 · p. 4
- A further input branch represents bone geometry with bone lengths and per-bone angles.Bone lengths are computed from the coordinate differences between connected joints.Source: §3.5 · p. 4
4. Experiments
Experiments on CASIA-B and OUMVLP-Pose compare skeleton methods with model-based and appearance-based approaches, test temporal modeling, and inspect learned joint activations.
4.1. Datasets
- CASIA-B has 124 subjects across 11 views and three walking conditions; OUMVLP has 10,307 subjects across 14 views.CASIA-B uses a 74-subject training and 50-subject test split; OUMVLP uses 5,153 training and 5,154 test subjects.Source: §4.1 · pp. 4–5
4.2. Implementation Details
- Training uses supervised contrastive loss, sequence and left-right-flip augmentation, keypoint noise, and three-embedding test-time augmentation.Model sizes, sequence lengths, and training durations differ between CASIA-B and OUMVLP-Pose.Source: §4.2 · p. 5
4.3. Comparison with State-of-the-Art
- On CASIA-B, our approach closely matches GaitGraph, while GCN approaches handle walking-condition changes better than PoseGait.The comparison uses averaged Rank-1 accuracy by probe angle, excluding identical-view cases.Source: §4.3 · pp. 5–6, Table 2 · p. 6
- Skeleton GCN methods considerably outperform the CNN-Pose baseline on OUMVLP-Pose.The comparison is reported in averaged Rank-1 accuracy.Source: Table 3 · p. 6, §4.3 · p. 6
4.4. Comparison with Appearance-based Methods
- Appearance-based methods retain the best results and better condition robustness, while skeleton methods narrow but do not close the gap; a hybrid improves all conditions, especially coats.The comparison uses cross-view CASIA-B results and AlphaPose keypoints for model-based OU-MVLP results.Source: §4.4 · p. 6, Table 4 · p. 7
4.5. Ablation Studies
- With multi-input precomputation and test-time augmentation absent, shuffled testing after sorted-sequence training drops substantially; spatial modeling still learns appearance-invariant bodily features.Multi-branch inputs are excluded from this ablation because their precomputed temporal information would harm its validity.Source: §4.5 · p. 7, Table 6 · p. 7
- Across AlphaPose and OpenPose keypoints, the approach gains almost the same performance over CNN-Pose, indicating that skeleton methods can scale with pose-estimator performance.The two estimators differ in approach and reported COCO test-dev mAP.Source: §4.5 · p. 7
4.6. Gait Recognition Analysis
- Figure 5 shows strongest activations at hands and feet; arm- and leg-swing dominate, with hip and head movement also contributing.Moving limbs activate more strongly than fixed parts in the temporal context; ear keypoints are especially relevant for the head.Source: §4.6 · pp. 7–8, Figure 5 · p. 8
5. Conclusion
The multi-branch skeleton approach reports a 3× improvement over the baseline on OUMVLP-Pose and faster, more stable training than previous graph-based approaches. Ablations and joint-activation visualizations indicate strong temporal modeling and a focus on moving, outermost joints, which the authors interpret as evidence that the model captures real gait features.
5. Conclusion
- 3× over the baseline on OUMVLP-Pose, with faster and more stable training than previous graph-based approaches.The architecture divides pre-computed temporal information into joints, motion, and bones branches.Source: §5 · p. 8
- Ablations show strong temporal modeling, while joint-activation visualizations focus on moving body parts and the outermost joints.The authors conclude these observations indicate that the model captures real gait features rather than visual re-identification.Source: §5 · p. 8
- The authors report that more accurate pose estimators could improve gait recognition and help move it from laboratory settings toward real-world use.Source: §5 · p. 8