PaperlayerView exact revision on arXiv

Source-linked AI summary

Gait Recognition in the Wild with Dense 3D Representations and A Benchmark

Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, Tao Mei

arXiv:2204.02569v1cs.CV

TL;DR

Gait recognition in the wild is underserved by 2D representations that discard 3D viewpoint and shape information and face diverse viewpoints and occlusions. The paper introduces SMPLGait, combining silhouette appearance features with 3D SMPL-based viewpoint and shape learning, and builds Gait3D with 4,000 subjects and over 25,000 sequences. Comprehensive experiments demonstrate SMPLGait’s effectiveness and the potential of 3D representations, while irregular walking speed and routes remain an open challenge.

  • Problem

    Existing gait recognition methods rely mainly on 2D representations that discard useful 3D viewpoint and shape information and perform poorly in wild scenarios with diverse viewpoints and occlusions.

  • Method

    SMPLGait uses a silhouette branch for appearance features and a 3D-STN branch to learn viewpoint and shape knowledge from 3D SMPL models.

  • Results

    SMPLGait demonstrates effectiveness against existing gait recognition approaches, and combining 3D with 2D representations further improves performance.

  • Takeaways & Limitations

    Gait3D and the experiments indicate the potential of dense 3D representations for gait recognition in the wild.

  • Takeaways & Limitations

    Irregular walking speed and routes in Gait3D make temporal gait modeling significantly different from laboratory datasets, leaving temporal representation learning as future work.

Abstract

from arXiv · show

Existing studies for gait recognition are dominated by 2D representations like the silhouette or skeleton of the human body in constrained scenes. However, humans live and walk in the unconstrained 3D space, so projecting the 3D human body onto the 2D plane will discard a lot of crucial information like the viewpoint, shape, and dynamics for gait recognition. Therefore, this paper aims to explore dense 3D representations for gait recognition in the wild, which is a practical yet neglected problem. In particular, we propose a novel framework to explore the 3D Skinned Multi-Person Linear (SMPL) model of the human body for gait recognition, named SMPLGait. Our framework has two elaborately-designed branches of which one extracts appearance features from silhouettes, the other learns knowledge of 3D viewpoints and shapes from the 3D SMPL model. In addition, due to the lack of suitable datasets, we build the first large-scale 3D representation-based gait recognition dataset, named Gait3D. It contains 4,000 subjects and over 25,000 sequences extracted from 39 cameras in an unconstrained indoor scene. More importantly, it provides 3D SMPL models recovered from video frames which can provide dense 3D information of body shape, viewpoint, and dynamics. Based on Gait3D, we comprehensively compare our method with existing gait recognition approaches, which reflects the superior performance of our framework and the potential of 3D representations for gait recognition in the wild. The code and dataset are available at https://gait3d.github.io.

1. Introduction

The paper addresses the limitations of 2D gait representations in unconstrained settings by introducing a 3D SMPL-based framework and the Gait3D benchmark. Experiments show SMPLGait is effective, while combining 2D and 3D representations further improves performance.

  • Motivation: 2D silhouettes and skeletons discard human-body viewpoint, shape, and dynamics information when projecting 3D bodies into 2D.Wild scenarios also introduce diverse camera viewpoints and occlusions that challenge methods developed on laboratory datasets.
  • Motivation: 3D meshes retain body pose and shape while providing viewpoint information that can normalize orientations during cross-view matching.These properties motivate dense 3D representations for gait recognition in the wild.
  • Proposed Framework: SMPLGait combines a silhouette branch for appearance features with a 3D-STN branch that learns viewpoint and shape knowledge from 3D human meshes.The 3D branch addresses appearance ambiguity caused by extreme viewpoint changes in the wild.
  • Dataset: Gait3D contains 4,000 subjects and over 25,000 sequences captured by 39 cameras in an unconstrained indoor scene.It provides recovered 3D meshes, pose, shape, viewpoint parameters, silhouettes, and keypoints for multimodal gait recognition.
  • Experimental Findings: SMPLGait demonstrates effectiveness against existing gait recognition approaches, and combining 3D with 2D representations further improves performance.The experiments indicate the potential and complementarity of 3D and 2D representations for gait recognition in the wild.

2. Related Work

Prior gait recognition research includes both 2D and 3D representations, but existing 3D approaches often discard viewpoint and shape information or depend on specialized devices. The paper positions 3D human meshes as a richer representation for unconstrained gait recognition.

  • 2D Gait Recognition: 2D gait recognition includes model-based methods using structural body parameters and model-free methods using silhouettes extracted from video frames.Deep learning methods have achieved strong results on established 2D benchmarks such as CASIA-B and OU-MVLP.
  • 3D Gait Recognition: Earlier 3D approaches used temporal motion models, tracked 3D motion, estimated 3D pose, or represented bodies with articulated cylinders and voxels.These methods explored 3D gait information through several body-modeling strategies.
  • 3D Gait Recognition: Existing wild-scenario methods struggle with diverse viewpoints and occlusions, while sparse 3D models lose helpful body-shape and viewpoint information.The paper therefore explores 3D meshes as a richer representation of human viewpoint and shape.
  • Gait Recognition Datasets: Gait recognition datasets have mainly belonged to the CASIA and OU-ISIR series, which supported early work on RGB images and silhouettes.The paper introduces Gait3D to provide large-scale 3D mesh data collected in unconstrained scenarios.
  • 3D Human Mesh Recovery: The SMPL model represents varied human body shapes in natural poses using shape, pose, scale, and viewpoint parameters.SMPL-based mesh recovery methods estimate 3D human shape, pose, and viewpoint from natural images.

3. The 3D Gait Recognition Method

SMPLGait combines silhouette-based appearance learning with SMPL-based 3D viewpoint, shape, and pose modeling in an end-to-end gait-recognition framework.

  • Network Structure: SMPLGait uses two branches: a CNN processes silhouette sequences, while a 3D-STN processes SMPL representations to model 3D information.The silhouette branch captures appearance, and the 3D branch learns transformations for viewpoint-related alignment.
  • Network Structure: The silhouette branch extracts frame-level 2D spatial features from binary silhouette images using a CNN-based backbone.Set Pooling and Horizontal Pyramid Pooling aggregate the frame features for sequence-to-sequence matching.
  • Network Structure: The 3D-STN takes SMPL vectors containing pose, shape, camera scale, and translation parameters and outputs frame-level transformation vectors.Its fully connected layers map the SMPL vector to the spatial dimensions of the silhouette feature map.
  • Network Structure: The 3D Spatial Transformation Module reshapes each transformation vector into a matrix and applies it to the silhouette feature map after zero-padding.This aligns 2D appearance features in feature space using the 3D-derived transformation.
  • Training and Inference: The framework is trained end-to-end with a weighted combination of triplet loss and cross-entropy loss, then uses cosine similarity for query-gallery matching.The two inference inputs are silhouette and SMPL sequences, respectively.

4. The Gait3D Benchmark

Gait3D is a large-scale in-the-wild benchmark with multi-camera video and recovered 3D human representations, supported by a defined matching protocol and dataset statistics.

  • Gait Representations: The dataset provides 3D human meshes estimated from video frames, including body poses, shapes, and viewpoints in 3D space.It also includes SMPL parameters, 3D poses, 2D silhouettes, and 2D poses for each frame.
  • Data Collection: The raw videos comprise about 1,090 hours recorded at 1,920 × 1,080 resolution and 25 FPS from 39 supermarket cameras over seven days.Camera scenes include entrances, shelf areas, freezers, dining areas, and checkout counters.
  • Sequence Matching: Cross-camera matching uses IoU and person-ReID features for tracking, followed by DBSCAN clustering and human filtering to produce 4,000 subjects and 25,309 sequences.The initial unsupervised clustering yields 5,336 sequence clusters before singleton groups and outliers are removed.
  • Dataset Statistics: Most sequences exceed 50 frames, most subjects have 2–25 sequences, and the longest sequence contains 500 frames.Frame sizes are mostly between 100–400 × 200–800, larger than person bounding boxes in existing datasets.
  • Evaluation Protocol: The benchmark splits subjects into 3,000 training and 1,000 test IDs, with 1,000 query sequences and 5,369 gallery sequences for open-set instance retrieval.One sequence per test identity forms the query set, while the remaining sequences form the gallery.

5. Experiments

Experiments compare 2D model-free and model-based methods with SMPLGait on Gait3D, then analyze input size, frame count, and training-ID scale. SMPLGait performs best, while combining 2D and 3D representations improves performance.

  • Experimental results: SOTA model-free methods perform worse on Gait3D than on in-the-lab datasets, revealing the greater difficulty of in-the-wild recognition.Performance varies substantially across methods in the unconstrained setting.
  • Experimental results: Model-based methods underperform model-free methods because sparse joints omit body shape and appearance, while irregular walking harms temporal modeling.The cited explanation also identifies uncertain walking speed and route as real-scenario challenges.
  • Experimental results: SMPLGait outperforms the other methods by a large margin, indicating the potential of 3D representations for in-the-wild gait recognition.The ablation further shows that integrating 2D and 3D representations better addresses the in-the-wild challenge.
  • More analysis of Gait3D: Larger input size improves almost all methods, except GaitGL, whose 3D CNN becomes harder to optimize because of increased physical-space misalignment.The compared input sizes are 88 × 128 and 44 × 64.
  • More analysis of Gait3D: Rank-1 accuracy peaks at around 30 training frames, showing that additional frames can introduce redundant or noisy information.Performance first increases and then decreases as sampled frames rise from 10 to 50.
  • More analysis of Gait3D: Performance grows stably as training IDs increase from 0.5K to 3K, reflecting Gait3D’s scalability.The experiment increments the training population by 0.5K IDs while fixing other settings.

6. Discussion

The discussion addresses privacy, data bias, and future research directions for 3D gait recognition. It describes restricted data release, demographic balancing, and unresolved challenges in temporal and multimodal learning.

  • Ethical issues: The dataset will not release human-cognizable videos, RGB frames, or person bounding boxes, and access is restricted to research use through case-by-case applications.These measures are presented as privacy protections.
  • Ethical issues: The authors report relatively balanced genders and ages to reduce data bias.This is stated as the dataset’s approach to mitigating demographic imbalance.
  • Future work: Future work includes learning more discriminative features directly from 3D meshes and modeling temporal information under irregular walking speed and routes.These conditions distinguish Gait3D from laboratory datasets.
  • Future work: The paper identifies multimodal fusion of silhouettes, 2D/3D skeletons, and 3D meshes as another direction for in-the-wild gait recognition.The proposed modalities are listed as complementary information sources for future study.
  • Potential negative impact: Large-scale deployment of gait recognition in urban monitoring networks could create privacy and security risks, motivating restricted research-only dataset distribution.The authors frame this as a potential negative impact rather than a demonstrated outcome.

7. Conclusion

The paper introduces SMPLGait, a 3D SMPL-based framework, and Gait3D, a large-scale dataset for gait recognition in the wild. Gait3D supplies diverse 3D and 2D gait representations for over 25,000 sequences from 4,000 subjects.

  • SMPLGait is a 3D SMPL model-based framework for gait recognition in the wild.
  • Gait3D contains over 25,000 gait sequences from 4,000 subjects captured by cameras in a large supermarket.
  • Gait3D provides 3D meshes, 3D SMPLs, 3D poses, 2D silhouettes, and 2D poses as diverse gait representations.

B. Appendix: Additional Experimental Results

Additional experiments examine how test-sequence length affects accuracy and describe cross-domain and qualitative evaluations of SMPLGait.

  • The experiments also evaluate domain gaps between datasets and provide exemplar SMPLGait results for qualitative assessment.
  • Testing uses sequence subsets from 10% to 100% of frames in 10% increments to evaluate sequence-length effects.
  • Accuracy improves as the number of tested frames increases, requiring a trade-off between accuracy and efficiency in practice.

B.2. Cross-domain Experiments

The cross-domain experiments measure how well a gait model trained on one dataset transfers directly to another. They compare CASIA-B, OU-LP, Gait3D, and GREW using 2D silhouettes with GaitSet.

  • Cross-domain evaluation trains GaitSet on one source dataset and tests it directly on another target dataset.
  • Because existing datasets lack 3D representations, the experiments use 2D silhouettes for gait representation.
  • The compared datasets are CASIA-B, OU-LP, Gait3D, and GREW, with GaitSet selected as the evaluated model-free method.

B.2.2 Main Results

Cross-domain results show substantial gaps between in-the-lab and in-the-wild gait recognition, while Gait3D-trained models achieve stronger transfer across domains and competitive accuracy on laboratory datasets.

  • 6.90% and 6.10% Rank-1 are obtained by GaitSet models trained on CASIA-B and OU-LP, respectively, when evaluated cross-domain.These results reflect a large domain gap between in-the-lab research and in-the-wild application.
  • 16.50% Rank-1 is achieved when training on GREW and testing on Gait3D, compared with 43.86% Rank-1 for the reverse transfer.The authors report that Gait3D contains 3,000 training IDs whereas GREW contains 20,000, despite both datasets being collected in the wild.
  • Table 5 reports cross-domain experiments in which models are trained on each source dataset and directly tested on target datasets.
  • Gait3D-to-GREW transfer achieves 43.86% Rank-1 and is reported to generalize more powerfully than GREW-to-Gait3D transfer.The comparison is presented as evidence of a significant domain gap between Gait3D and GREW.
  • 66.71% Rank-1 on CASIA-B and 97.84% on OU-LP are achieved by GaitSet trained on Gait3D, with the latter close to 99.89% from OU-LP-trained models.These results are reported as further evidence of the gap between laboratory datasets and in-the-wild application, and of Gait3D's ability to support generalized gait representations.

B.3. Exemplar Results of SMPLGait

SMPLGait’s exemplar retrievals show successful matching across multi-viewpoint, occluded, and multi-person cases, while similar clothing and body shapes remain challenging. The framework also leaves temporal dynamics insufficiently explored for practical gait recognition.

  • Figure 8–11 show query silhouette and 3D mesh sequences followed by the top-5 gallery sequences, with green boxes marking correct matches and red boxes marking errors.
  • 3D representations work well in multi-viewpoint, occluded, and multi-person cases by providing shape, pose, and viewpoint information.
  • Similar clothing and body shapes seriously interfere with matching, making them a main challenge for gait recognition.
  • The framework uses only a few gait-sequence frames, such as 30 frames, so temporal dynamics are not fully explored.
  • Large-scale deployment of gait recognition could create privacy and security issues, so Gait3D is distributed for research under case-by-case licensing.
Loading 2204.02569v1…