Source-linked AI summary
GaitSet: Cross-view Gait Recognition through Utilizing Gait as a Deep Set
Hanqing Chao, Kun Wang, Yiwei He, Junping Zhang, Jianfeng Feng
TL;DR
Gait recognition needs representations that retain useful information without the temporal constraints of ordered sequences or the information loss of gait templates. GaitSet models gait as an unordered deep set using frame-level extraction, set pooling, and horizontal pyramid mapping, achieving strong accuracy and robustness across datasets and conditions. Its main scope boundary is that reported comparisons can involve different training and testing subsets.
Problem
Existing gait representations either lose temporal information through template compression or impose unnecessary sequential constraints, limiting flexibility for practical cross-view recognition.
Method
GaitSet independently extracts frame features, aggregates them with permutation-invariant set pooling, and maps the resulting set-level feature into a discriminative representation.
Results
GaitSet achieves the highest recognition accuracy among compared state-of-the-art algorithms and shows flexibility and robustness across datasets, views, and walking conditions.
Takeaways & Limitations
Gait can be represented as a deep set that aggregates spatiotemporal information from different sequences for cross-view gait recognition.
Takeaways & Limitations
In one CASIA-B comparison, other models used only NM for training and testing, whereas GaitSet used NM, BG, and CL, so matched-subset evaluation can yield better performance.
Abstract
from arXiv · showhide
Gait is a unique biometric feature that can be recognized at a distance; thus, it has broad applications in crime prevention, forensic identification, and social security. To portray a gait, existing gait recognition methods utilize either a gait template which makes it difficult to preserve temporal information, or a gait sequence that maintains unnecessary sequential constraints and thus loses the flexibility of gait recognition. In this paper, we present a novel perspective that utilizes gait as a deep set, which means that a set of gait frames are integrated by a global-local fused deep network inspired by the way our left- and right-hemisphere processes information to learn information that can be used in identification. Based on this deep set perspective, our method is immune to frame permutations, and can naturally integrate frames from different videos that have been acquired under different scenarios, such as diverse viewing angles, different clothes, or different item-carrying conditions. Experiments show that under normal walking conditions, our single-model method achieves an average rank-1 accuracy of 96.1% on the CASIA-B gait dataset and an accuracy of 87.9% on the OU-MVLP gait dataset. Under various complex scenarios, our model also exhibits a high level of robustness. It achieves accuracies of 90.8% and 70.3% on CASIA-B under bag-carrying and coat-wearing walking conditions respectively, significantly outperforming the best existing methods. Moreover, the proposed method maintains a satisfactory accuracy even when only small numbers of frames are available in the test samples; for example, it achieves 85.0% on CASIA-B even when using only 7 frames. The source code has been released at https://github.com/AbnerHqC/GaitSet.
1 INTRODUCTION
Gait recognition is valuable because it enables nonintrusive identification at a distance, but clothing, carried items, viewpoint, and other factors make cross-view recognition challenging. GaitSet treats gait silhouettes as an unordered set, using a global-local network to provide flexible, efficient, and effective recognition.
- Gait recognition supports crime prevention, forensic identification, and social security because it can identify subjects remotely without intrusive interaction.
- Existing methods either compress silhouettes into gait templates that omit temporal and fine-grained spatial information or process sequences with unnecessary ordering constraints.
- GaitSet regards gait as a set of silhouettes, assuming pose order is not key information because poses in a gait period are individually recognizable and their ordering pattern is universal.
- GaitSet independently extracts frame-level features, applies Set Pooling for a set-level feature, and uses Horizontal pyramid mapping to obtain a discriminative deep-set representation.
- GaitSet accepts any number of nonconsecutive silhouettes from different viewpoints and walking conditions, without constraints beyond silhouette size.
- GaitSet computes each sample representation once and recognizes identities by comparing Euclidean distances between sample representations.
- GaitSet substantially improves state-of-the-art performance on CASIA-B and OU-MVLP while showing robustness to view and walking-condition variations.
2 RELATED WORKS
Gait recognition research has mainly used compressed templates or ordered video sequences, while unordered-set learning has been developed largely in other vision domains. GaitSet applies deep set learning to gait recognition as an alternative representation strategy.
- Gait recognition methods generally use either template-based representations, such as GEI and CGI, or video-based silhouette sequences.
- Template-based methods compress gait information into one image before representation extraction and matching.
- Video-based methods extract spatiotemporal gait features using approaches including 3D convolutions, skeletons, LSTM attention, and spatial-temporal graph attention.
- Deep learning on unordered sets was initially developed for point-cloud tasks and later applied across domains including image captioning.
- GaitSet’s framework includes set pooling, convolution and pooling blocks, and horizontal pyramid pooling for processing gait silhouettes as a set.
- Set-based deep learning formalizes permutation-invariant functions, but the topic had not been studied in depth for gait recognition beyond the authors’ earlier conference work.
3 GAITSET
GaitSet represents gait as an unordered set of silhouettes, extracting frame-level features, pooling them into set-level features, and mapping them into discriminative representations. Its design combines permutation-invariant pooling, multiscale spatial mapping, and efficient nearest-neighbor identification.
- 3.1 Problem Formulation: GaitSet treats gait silhouettes as a set and applies convolutional feature extraction, permutation-invariant Set Pooling, and Horizontal Pyramid Mapping.F extracts frame-level features, G maps them to set-level features, and H learns a discriminative representation.
- 3.2 Set Pooling: Set Pooling condenses frame-level feature maps into a set-level feature while supporting arbitrary frame order and arbitrary set cardinality.The paper analyzes statistical, joint, pixel-wise attention, and frame-wise attention instantiations.
- 3.3 Horizontal Pyramid Mapping: Horizontal Pyramid Mapping splits set-level feature maps into strips across multiple scales, pools each strip, and uses independent fully connected layers for discriminative mapping.Combining global max and average pooling outperforms either operation alone, while different strips represent different spatial positions and receptive fields.
- 3.4 Multilayer Global Pipeline: The multilayer global pipeline combines set-level features from different network layers to retain both fine-grained and coarse-grained walking information.The main pipeline emphasizes a person’s profile, whereas the multilayer branch preserves more walking-movement details.
- 3.6 Training and Test: During testing, GaitSet concatenates multiscale features and identifies the nearest gallery representation using Euclidean distance.Recognition complexity is O(df|G|), motivating a post-trained linear projection that reduces feature dimensionality while preserving competitive accuracy.
4 EXPERIMENTS
The experiments evaluate GaitSet against state-of-the-art methods, conduct ablations, study feature-dimension reduction, and assess practicality under limited silhouettes, multiple views, and multiple walking conditions.
- Experiments compare GaitSet with state-of-the-art methods on the CASIA-B and OU-MVLP gait datasets.
- Ablation studies are conducted on CASIA-B to evaluate components of GaitSet.
- The study analyzes feature-dimension reduction and practicality under limited silhouettes, multiple views, and multiple walking conditions.
4.1 Datasets
CASIA-B contains 124 subjects walking under normal, bag-carrying, and coat-wearing conditions, recorded from 11 views. The evaluation uses separate training and testing subjects under short-, medium-, and long-training settings.
- CASIA-B contains 124 subjects, three walking conditions, 11 views, and 113,640 videos.The conditions are normal, walking with a bag, and wearing a coat or jacket.
- Training settings: The short-training setting uses subjects 001–024 for training and 025–124 for testing without subject overlap.
- Training settings: The medium-training setting uses subjects 001–062 for training and 063–124 for testing.
- Training settings: The long-training setting uses subjects 001–074 for training and 075–124 for testing.
4.2 Parameter Setting
GaitSet is trained on aligned 64 × 44 silhouettes using Adam, PyTorch 0.4.0, and four NVIDIA 1080TI GPUs; unless otherwise stated, training sets contain 30 silhouettes.
- The input consists of aligned silhouettes sized 64 × 44, directly provided by the datasets.
- Training uses the Adam optimizer and Python with PyTorch 0.4.0 on a computer equipped with four NVIDIA 1080TI GPUs.
- Unless otherwise stated, the training set cardinality is 30.
4.3 Brief Introduction of Compared Methods
The compared methods use varied representations and architectures, including template projection, motion co-clustering, three-dimensional convolutions, ensembles, autoencoding, GAN-based view transformation, angle-centered loss, and GEI classification.
- ViDP projects gait templates into a latent space with a unitary linear projection to learn view-invariant representations.
- CMCC uses motion co-clustering, while CNN-3D applies a three-layer 3D-CNN to adjacent frames and averages predictions from 16 samples.
- CNN-Ensemble aggregates outputs from eight different networks.
- AE extracts view-invariant features, MGAN projects gait features between angles, and ACL learns discriminative features robust to local parts and temporal-window sizes.
- GEINet classifies gait energy images using a two-layer CNN followed by two fully connected layers.
4.4 Main Results
GaitSet is evaluated against state-of-the-art methods on CASIA-B and OU-MVLP under multiple training settings and walking conditions. Results indicate strong performance across views, improved accuracy with more training data, and reduced but competitive accuracy for coat-wearing subjects.
- Benchmark comparisons: GaitSet is compared with state-of-the-art methods on CASIA-B and OU-MVLP using averaged rank-1 accuracy while excluding identical-view cases.The CASIA-B comparison covers three experimental settings, and the OU-MVLP results also exclude identical-view cases.
- CASIA-B results: 90%+: On CASIA-B LT, GaitSet exceeds 90% accuracy for all NM views except 180°.The reported pattern is especially strong on the normal-walking subset under the large-sample training setting.
- Training-scale effects: More training data improves GaitSet accuracy on the CASIA-B NM subset.The paper reports this improvement when comparing medium- and large-sample training settings.
- Walking conditions: 15%+: On CASIA-B CL, GaitSet exceeds the best previously reported performance by over 15%, although recognition remains less satisfactory than on other conditions.The paper attributes the reduced coat-wearing performance to altered appearance, occluded motion, and fewer CL training samples.
- Walking conditions: GaitSet achieves satisfactory performance on the CASIA-B BG subset.This result is reported alongside the stronger NM performance and the reduced CL performance.
4.5 Ablation Experiments and Model Studies
Ablation studies examine set-based gait representation, set pooling, HPM design, and training strategies on CASIA-B LT. The results support set processing, independent HPM weights, and combined loss training as effective design choices.
- Set representation: 6%+: Using gait as a set exceeds an otherwise identical GEI-based network by more than 6% on NM and more than 19% on CL.The paper links the gains to temporal information preservation in high-level feature maps and a larger effective training-data volume.
- Set pooling: Pixel-wise attention achieves the highest NM and BG accuracy among set-pooling strategies, while max pooling achieves the highest CL accuracy.Max pooling is selected for the final version because it is concise and remains highly competitive on NM and BG.
- HPM design: 7%+: Independent HPM fully connected-layer weights increase accuracy by more than 7% on each subset.The experiments also observed faster network convergence with independent weights.
- Training strategies: 96.1%: Combining cross-entropy pretraining with triplet-loss training reaches the highest CASIA-B NM rank-1 accuracy.All three strategies exceed 95% with batch normalization and dropout, while dropout is essential for robust cross-entropy training and batch normalization improves every strategy.
4.6 Feature Dimension Reduction
The study reduces GaitSet’s high-dimensional final representation to improve testing efficiency while preserving recognition performance. A 1024-dimensional representation retains 95% NM accuracy, but overly small dimensions degrade performance.
- Motivation: 15,872 dimensions: Concatenated HPM outputs create a large final representation that impairs testing efficiency.Because identification complexity is O(d_f|G|), reducing the feature dimension is important when the gallery is large.
- Trade-offs: Overly large output dimensions can overfit, whereas overly small dimensions constrain the fully connected layers’ learning capacity and reduce performance.The paper also reports that high-dimensional HPM outputs are less robust on CL for cross-entropy-trained models.
- Dimensional reduction: 1024 dimensions retain 95% NM recognition accuracy, equal to 6.5% of the original 15,872-dimensional representation.The reduction uses postprocessing that compresses the learned feature representation for practical applications.
4.7 Practicality
GaitSet’s set-based representation is evaluated under few-frame, multiview, and mixed-walking-condition scenarios. The experiments show that combining suitable silhouettes can preserve or improve recognition performance across practical variations.
- Parameter and Evaluation Settings: The HPM output dimension has different accuracy relationships across CASIA-B’s NM, BG, and CL subsets and training strategies.Figure 6 compares the three subsets from left to right using different plotted lines for training strategies.
- Parameter and Evaluation Settings: CASIA-B LT rank-1 accuracies are averaged across 11 non-identical views and then across 10 experimental repetitions.This averaging protocol defines the reported evaluation values for the silhouette-volume experiment.
- Few Silhouettes: 82% accuracy is achieved using only 7 silhouettes, with performance rising monotonically as the input set grows and approaching the best result beyond 25 silhouettes.The 25-silhouette threshold is consistent with the number of frames in one gait period.
- Multiple Views: Multiview inputs improve recognition because GaitSet aggregates information from different views, combining parallel and vertical information.Multiview experiments exclude cases where probe views are already present in the gallery.
- Multiple Walking Conditions: Mixed walking conditions remain beneficial when silhouettes provide complementary information, with bag-carrying and coat-wearing combinations improving accuracy over less informative substitutions.The BG and CL combination helps, whereas replacing some normal-condition silhouettes with BG or CL can add noise and degrade performance.
5 CONCLUSION
The paper frames gait as a deep set and uses GaitSet to extract spatial and temporal information while aggregating data across sequences. Experiments on two benchmark datasets report leading accuracy, flexibility, and robustness in complex environments.
- GaitSet regards gait as a deep set and extracts spatial and temporal information more effectively and efficiently than template- or sequence-based methods.
- The approach aggregates spatiotemporal information from different sequences to enhance cross-view gait recognition accuracy.
- Experiments on two benchmark gait datasets report the highest recognition accuracy compared with other state-of-the-art algorithms.
- GaitSet demonstrates flexibility and robustness across various complex environments, supporting its potential for practical applications.
- The deep-set structure could be adapted to person re-identification and video-based face recognition with few minor changes.