PaperlayerView exact revision on arXiv

Source-linked AI summary

LidarGait: Benchmarking 3D Gait Recognition with Point Clouds

Chuanfu Shen, Chao Fan, Wei Wu, Rui Wang, George Q. Huang, Shiqi Yu

arXiv:2211.10598v2cs.CV

TL;DR

Camera-based gait recognition has limited access to 3D structure, and visual ambiguity is a concern in outdoor settings. LidarGait learns gait features from point-cloud depth projections, and experiments report advantages over existing methods, with an umbrella-condition exception.

  • Problem

    Camera-based gait recognition does not directly use 3D geometry, and poor illumination or complex outdoor backgrounds can impair visual processing.

    Source: §1 · pp. 1–2

  • Method

    LidarGait projects sparse 3D point clouds into depth images and uses convolutional networks to learn gait features containing structural information.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    LidarGait outperforms existing point-based and camera-based methods across evaluated conditions except the umbrella subset.

    Source: §5.2 · p. 7, §7 · p. 8

  • Takeaways & Limitations

    The experiments indicate that 3D structural information contributes to gait-recognition performance, and LiDAR performs well across varied challenges.

    Source: §1 · p. 2

    LidarGait performs poorly when umbrellas are included in point clouds, and it currently uses only one modality despite SUSTech1K’s synchronized RGB and point-cloud data.

    Source: §7 · p. 8

Abstract

from arXiv · show

Video-based gait recognition has achieved impressive results in constrained scenarios. However, visual cameras neglect human 3D structure information, which limits the feasibility of gait recognition in the 3D wild world. Instead of extracting gait features from images, this work explores precise 3D gait features from point clouds and proposes a simple yet efficient 3D gait recognition framework, termed LidarGait. Our proposed approach projects sparse point clouds into depth maps to learn the representations with 3D geometry information, which outperforms existing point-wise and camera-based methods by a significant margin. Due to the lack of point cloud datasets, we built the first large-scale LiDAR-based gait recognition dataset, SUSTech1K, collected by a LiDAR sensor and an RGB camera. The dataset contains 25,239 sequences from 1,050 subjects and covers many variations, including visibility, views, occlusions, clothing, carrying, and scenes. Extensive experiments show that (1) 3D structure information serves as a significant feature for gait recognition. (2) LidarGait outperforms existing point-based and silhouette-based methods by a significant margin, while it also offers stable cross-view results. (3) The LiDAR sensor is superior to the RGB camera for gait recognition in the outdoor environment. The source code and dataset have been made available at https://lidargait.github.io.

1. Introduction

Gait recognition needs to account for 3D human geometry, while camera-based approaches face limitations in single-view representation and visually challenging outdoor conditions. The paper introduces LidarGait and SUSTech1K to study point-cloud gait recognition.

1. Introduction

  • Camera-based gait methods use single-view 2D features or estimated 3D representations, which can be imprecise in difficult conditions; outdoor visual ambiguity can also impair detection and segmentation.The introduction identifies poor illumination and complex backgrounds as sources of visual ambiguity.Source: §1 · pp. 1–2
  • SUSTech1K is introduced as a large-scale LiDAR gait dataset with 25,239 sequences from 1,050 subjects and annotations for varied conditions.The listed challenges include illumination, occlusion, clothing, and carrying.Source: §1 · p. 2
  • Existing point-based methods underperformed silhouette methods in the authors’ experiments, which they attribute to coarse global-context modeling rather than gait’s need for fine-grained local information.The methods examined were drawn from point-based object classification.Source: §1 · p. 2
  • LidarGait projects 3D point clouds into depth images and uses convolutional networks to learn fine-grained gait features with 3D structural information.The approach is contrasted with point-wise methods that model global context from sparse point clouds.Source: §1 · p. 2

2. Related Work

Related work spans image-based, estimated-structure, sensor-based, and point-cloud approaches to gait recognition. The authors identify both practical constraints in existing 3D approaches and a shortage of large LiDAR gait datasets.

2. Related Work

  • Image-based gait methods commonly use silhouettes and other templates, while structure- and dynamics-based approaches rely heavily on estimation models.The passage groups these methods as 2D representation-based approaches.Source: §2 · p. 2
  • Common 3D gait sources include sensors such as Kinect, limited to indoor, close-distance use, and geometry estimated from cameras, whose performance is reported as insufficient for real-world applications.The passage also mentions multi-camera reconstruction as a source of 3D geometry.Source: §2 · p. 3
  • Existing gait benchmarks include in-the-lab, synthetic, and in-the-wild datasets, but prior LiDAR datasets had at most 30 subjects, limiting statistically reliable evaluation.The passage notes that synthetic data can introduce a cross-domain issue and that common 3D alternatives have practical constraints.Source: §2 · p. 3
  • Point-cloud object classification uses projection-, point-, and graph-wise models, while sparse point distributions remain challenging for modeling geometry and correlations.The paper draws representative object-classification models into its study of point-based gait recognition.Source: §2 · p. 3

3. The SUSTech1K Benchmark

SUSTech1K is a synchronized LiDAR and RGB gait dataset collected by a mobile robot across varied walking conditions. Its protocol, representations, and evaluation setup support analysis across attributes and views.

3. The SUSTech1K Benchmark

  • SUSTech1K contains 1,050 identities, 25,239 sequences, 763,416 point-cloud frames, and 3,075,575 RGB images with corresponding silhouettes.A mobile robot carried a 128-beam LiDAR scanner and a monocular camera; the modalities are synchronized and timestamped.Source: §3 · p. 3
  • Participants walked four round-trip and four one-way paths normally, then repeated the paths with random attributes such as carrying objects; the dataset includes outdoor challenges including occlusion, illumination, uniforms, and umbrellas.The collection covers three campus scenes, and each subject can contribute 48 gait sequences across paths, rounds, and modalities.Source: §3 · p. 4
  • Camera representations use detected, tracked, and segmented RGB imagery, while LiDAR sequences are cleaned with noise and ground removal; camera segmentation degrades in low light.Tracking errors were manually corrected where bounding boxes were inaccurate.Source: §3 · p. 4
  • Evaluation uses an open-set split without sample overlap and cross-view retrieval grouped by attributes, reporting Rank-1 and Rank-5 accuracy.The protocol compares probe sets against galleries from each view.Source: §3 · p. 4

4. Gait Recognition with Point Clouds

The section formalizes point-cloud gait recognition as learning identity embeddings from sequences and presents LidarGait’s depth-projection, spatial-encoding, and temporal-aggregation pipeline. It also describes projection variants, training losses, and inference distance.

4.1. Problem Setting

  • Each sequence contains T frames and N points per frame with C feature channels; the network learns an embedding representing its associated identity.Source: §4.1 · pp. 4–5

4.2. LidarGait

  • Range-view projection maps LiDAR points to depth-image pixels using beam-angle resolutions, retains the nearest point at collisions, and fills unoccupied pixels with zero.Source: §4.2 · p. 5
  • LidarGait encodes projected depths with a convolutional structural encoder (GaitBase [10]) and aggregates variable-length sequences with Set Pooling [5].Source: §4.2 · p. 5, §4.1 · p. 5, Figure 5 · p. 6
  • MV-LidarGait adds two orthogonal projections to the LiDAR range view, extracts each view independently, and fuses features frame by frame.Source: §4.2 · p. 6

4.3. Traning and Inference

  • Training combines BA+ triplet and cross-entropy losses, while inference compares probe-gallery pairs using Euclidean distance.Source: §4.1 · p. 5, §4.3 · p. 6

5. Experiments

Experiments compare LiDAR and camera gait recognition across SUSTech1K conditions and cross-view settings. They report LidarGait’s broad advantage and test depth and projected-view contributions.

5.1. Experimental Setup

  • The evaluation uses SUSTech1K’s training, validation, and test splits and probes conditions including clothing, illumination, carrying, occlusion, and uniforms.Normal-condition sequences form the gallery, while variant-condition sequences form the probes.Source: §5.1 · p. 6

5.1.1 Comparative Methods

  • Comparisons include four camera-based gait methods and four point-cloud methods; LidarGait uses GaitBase with set pooling on range-view depths.Methods share the training strategy, and inputs are aligned and resized to 64 × 64.Source: §5.1.1 · pp. 6–7

5.2. Comparative Results

  • LidarGait outperforms the evaluated point-based and camera-based methods, leading in all conditions except the umbrella subset.The table reports evaluations across attributes on the SUSTech1K valid and test sets.Source: §5.2 · p. 7, Table 2 · p. 7
  • LiDAR performs stably across views, while camera performance is poor for queries at 0°, 90°, and 180°; greater sensor distance affects both modalities.The cross-view comparison uses the same GaitBase encoder for both modalities.Source: §5.2 · p. 7, Figure 7 · p. 7

5.3. Ablation Study

  • Adding depth to LiDAR silhouettes raises rank-1 accuracy from 64.70% to 86.77%, while camera silhouettes retain more detail because of their higher resolution.Range view is the strongest single projected view; combining viewpoints improves LidarGait by 0.73%, mainly on umbrellas, while adding BEV does not help MV-LidarGait.Source: §5.3 · pp. 7–8, Figure 8 · p. 8

6. Discussion

The discussion describes dataset ethics and access restrictions.

6. Discussion

  • SUSTech1K access is limited to non-commercial research through case-by-case applications, and RGB faces are blurred to protect privacy.The dataset received SUSTech IRB review, and participants gave written consent for research use and sharing.Source: §6 · p. 8
  • Recorded data may be used for 20 years after publication; afterward it must be deleted and cannot be used.Source: §6 · p. 8

7. Conclusions and Future work

The conclusion presents LidarGait and SUSTech1K as contributions to point-cloud gait recognition. It also identifies umbrella performance and multi-view fusion choices as open considerations.

7. Conclusions and Future work

  • LidarGait uses multi-view representations to exploit human 3D geometry, and SUSTech1K contains 25,239 sequences from 1,050 subjects across varied conditions.The conclusion reports the method’s results on SUSTech1K as demonstrating LiDAR’s superiority and LidarGait’s effectiveness.Source: §7 · p. 8

A. The Suboptimal Performance on Umbrella Subset

  • Umbrellas remain a weak case because they are included in point clouds and projection images, causing misalignment; umbrella removal or random erasing are suggested.The table reports LidarGait outperforming other methods across subsets except umbrella-carrying cases.Source: §7 · p. 8, A. The Suboptimal Performance on… · p. 9

B. Multi-view Fusion

  • Among tested fusion variants, only frame-level concatenation improved performance; BEV did not, so fusion focused on front-range and right-side views.The study also compared summation and sequence-level fusion.Source: B. Multi-view Fusion · p. 9

C. Effect of Sequence Length

The sequence-length experiment examines randomly selected frame counts at inference. Both modalities improve with more frames, while LiDAR also reports results from a one-frame setting.

C. Effect of Sequence Length

D. Qualitative results

The qualitative feature comparison finds more discriminative, compact LidarGait features than PointNet features. The section also presents a LiDAR–camera comparison across inference frame counts.

D. Qualitative results

  • The listed comparison metrics are LiDAR Rank1 and Rank5 and camera Rank1 and Rank5.Source: D. Qualitative results · p. 9
  • Figure 9 compares LiDAR and camera performance across the number of frames used at inference.Source: Figure 9 · p. 9
  • LidarGait features show clearer discrimination and more compact intra-class distributions than PointNet features.PointNet features are described as less discriminative and more sparsely distributed within classes.Source: D. Qualitative results · p. 9

E. Evaluation if Variants as Gallery

This evaluation uses variation-condition sequences as gallery sets and normal-condition sequences as probe sets. The authors report performance degradation when normal cases serve as probes.

E. Evaluation if Variants as Gallery

F. Exemplar Sequences of SUSTech1K

The exemplars illustrate LiDAR gait representations across challenging conditions and contrast them with camera-derived silhouettes. The authors highlight LiDAR’s geometry cues and viewpoint, as well as an umbrella-related alignment issue.

F. Exemplar Sequences of SUSTech1K

  • The exemplars include normal, occlusion, and poor-illumination conditions.Source: F. Exemplar Sequences of SUSTech1K · p. 9
  • LiDAR provides informative geometry cues for gait recognition and an additional viewpoint perspective.Source: F. Exemplar Sequences of SUSTech1K · p. 9
  • LiDAR depth and camera silhouettes are shown together, with a noted LiDAR misalignment issue when the probe carries an umbrella.Source: Figure 10 · p. 10
  • Under occlusion, camera-derived silhouettes can be low quality, while LiDAR is described as obtaining high-quality gait representations.The authors attribute the difficulty of 2D segmentation to separating the pedestrian of interest in 2D space despite people living in 3D space.Source: F. Exemplar Sequences of SUSTech1K · p. 10
  • The authors report that failed gait representations commonly appear in GREW and Gait3D in real-world conditions, motivating investigation of more robust representations.Source: F. Exemplar Sequences of SUSTech1K · p. 10

Related papers

Loading 2211.10598v2…