Source-linked AI summary
GaitGraph: Graph Convolutional Network for Skeleton-Based Gait Recognition
Torben Teepe, Ali Khan, Johannes Gilg, Fabian Herzog, Stefan Hörmann, Gerhard Rigoll
TL;DR
Silhouette-based gait recognition faces challenges from complex backgrounds and may include non-gait visual clues. GaitGraph uses pose-estimated skeleton graphs with GCNs and achieves state-of-the-art model-based performance on CASIA-B.
Problem
Silhouette-based methods struggle with cluttered, changing backgrounds and may represent visual clues beyond gait.
Method
GaitGraph extracts 2D skeleton poses from RGB images and models their spatial-temporal graph structure with GCNs.
Results
GaitGraph achieves state-of-the-art results in model-based gait recognition and competitive results against appearance-based methods on CASIA-B.
Takeaways & Limitations
The approach provides a modern model-based gait recognizer using skeleton poses and GCN-based spatial and temporal modeling.
Takeaways & Limitations
The method cannot be evaluated on OU-MVLP because most available gait datasets lack the RGB images it requires.
Abstract
from arXiv · showhide
Gait recognition is a promising video-based biometric for identifying individual walking patterns from a long distance. At present, most gait recognition methods use silhouette images to represent a person in each frame. However, silhouette images can lose fine-grained spatial information, and most papers do not regard how to obtain these silhouettes in complex scenes. Furthermore, silhouette images contain not only gait features but also other visual clues that can be recognized. Hence these approaches can not be considered as strict gait recognition. We leverage recent advances in human pose estimation to estimate robust skeleton poses directly from RGB images to bring back model-based gait recognition with a cleaner representation of gait. Thus, we propose GaitGraph that combines skeleton poses with Graph Convolutional Network (GCN) to obtain a modern model-based approach for gait recognition. The main advantages are a cleaner, more elegant extraction of the gait features and the ability to incorporate powerful spatio-temporal modeling using GCN. Experiments on the popular CASIA-B gait dataset show that our method archives state-of-the-art performance in model-based gait recognition. The code and models are publicly available.
1. INTRODUCTION
GaitGraph addresses limitations of silhouette-based gait recognition by using pose-estimated skeletons and graph convolutional modeling. On CASIA-B, it reports state-of-the-art model-based performance and competitive results against appearance-based methods.
- Gait recognition supports identification at a great distance without subject cooperation, with applications in security, access control, and forensics.
- Surface type, clothing, carried items, clutter, and occlusions challenge learning invariant gait features.
- Silhouette pipelines rely mainly on background subtraction, which is cumbersome in cluttered, rapidly changing real-world scenes.
- Skeleton poses provide a cleaner gait representation because silhouettes also capture appearance cues such as physique and hairstyle.
- GaitGraph applies a GCN to human skeleton pose graphs, replacing silhouette extraction and using less sensitive personal data.
- GaitGraph reports state-of-the-art results among model-based methods and competitive results against appearance-based methods.
2. RELATED WORK
Prior gait-recognition methods differ in spatial representation and temporal modeling. Appearance-based approaches use silhouettes, whereas model-based approaches use body structure, and recent pose-based methods combine skeleton features with CNNs, LSTMs, or handcrafted temporal features.
- Gait-recognition methods can be grouped by spatial feature extraction and temporal modeling.
- Spatial feature extraction: Appearance-based methods use binary silhouettes, whose extraction becomes more complicated in dynamic and changing settings.
- Spatial feature extraction: Model-based approaches represent the underlying physical structure of the body.
- Temporal modeling: Temporal modeling includes single-image, sequence-based, and set-based approaches, trading temporal information and computational cost differently.
- Pose-based methods: Recent pose-based gait methods combine skeleton features with CNNs and LSTMs or use handcrafted temporal features.
3. SKELETON-BASED GAIT RECOGNITION
GaitGraph converts image sequences into skeleton-pose graphs and processes them with a spatio-temporal ResGCN. The representation combines graph structure, joint features, pose estimation, residual graph blocks, temporal convolutions, and supervised contrastive learning.
- Preliminaries: A skeleton graph G=(V,E) represents joints as nodes and bones as edges through an adjacency matrix A.
- Preliminaries: The gait input combines graph structure A with a feature tensor X containing joint features across T frames.
- Preliminaries: Each joint feature contains 2D coordinates and confidence, while the node dimension indexes the skeleton joints.
- Graph convolutions: Graph convolution aggregates skeleton features using graph structure and layer-wise learned transformations.
- Human pose extraction: Human pose extraction detects keypoint locations from RGB images using heatmaps, with HRNet pretrained on COCO providing 2D poses.
- The pipeline estimates a pose for each image and feeds the pose sequence through ResGCN to produce a feature embedding.
- Network and implementation: The ResGCN architecture combines graph convolution, temporal 2D convolution, residual connections, average pooling, and a fully connected layer, trained with supervised contrastive loss.
4. EXPERIMENTS
Experiments on CASIA-B evaluate GaitGraph across views and walking conditions against model-based and appearance-based methods, with ablations testing its spatial and temporal modeling. The results show strong model-based performance, competitive appearance-based performance, and reliance on temporal ordering.
- Dataset and evaluation: Experiments use CASIA-B to compare GaitGraph across multiple views and walking conditions with model-based and appearance-based methods.CASIA-B contains 124 subjects, 11 views, three walking conditions, and 110 sequences per subject.
- Limitations: The study cannot evaluate GaitGraph on OU-MVLP because most available gait datasets lack RGB images required by the pose-based pipeline.CASIA-B is therefore used for comparison with other methods.
- Dataset and evaluation: The evaluation follows a popular protocol with 74 training subjects and 50 test subjects, using NM, BG, and CL probe subsets.The gallery contains NM #1–4, while probes use NM #5–6, BG #1–2, and CL #1–2.
- Comparison with state-of-the-art methods: GaitGraph significantly improves over PoseGait across cross-views and walking conditions among model-based methods.Both methods use similarly performing pose extractors, isolating the comparison primarily to feature extraction architecture.
- Comparison with state-of-the-art methods: GaitGraph achieves competitive results against appearance-based methods despite using a lower-dimensional feature representation.The compared appearance-based methods use silhouette images explicitly or implicitly.
- Spatio-temporal study: Shuffling sequences at test time after training on sorted sequences causes a profound performance drop, supporting strong temporal modeling by GaitGraph.A corresponding GaitPart ablation shows only a slight drop from the sorted to shuffled condition.
5. CONCLUSION
GaitGraph interprets gait as a sequence of skeleton graphs extracted from RGB images and applies graph-based modeling to gait information. On CASIA-B, it reports state-of-the-art model-based and competitive appearance-based results, with ablations supporting temporal gait modeling.
- Conclusion: GaitGraph represents gait as a sequence of skeleton graphs derived from 2D human pose estimates and models their inherent graph structure.The approach uses a human pose estimator instead of silhouette extraction.
- Conclusion: Experiments on CASIA-B show state-of-the-art results for model-based gait recognition and competitive results against appearance-based methods.The reported ablations support the recovery of temporal gait features rather than primarily relying on appearance.