Source-linked AI summary
On Learning Disentangled Representations for Gait Recognition
Ziyuan Zhang, Luan Tran, Feng Liu, Xiaoming Liu
TL;DR
Gait recognition must remain discriminative despite clothing, carrying, viewing-angle, and other appearance changes that challenge silhouette- and model-based features. The paper introduces GaitNet to disentangle appearance, canonical, and pose information from RGB videos, adds the frontal-view FVG dataset, and reports superior performance across CASIA-B, USF, and FVG, with gait advantages over face recognition in selected conditions.
Problem
Existing gait features based on silhouettes or articulated body models degrade under clothing, carrying, and viewing-angle variations.
Method
GaitNet disentangles appearance, canonical, and pose features from RGB frames, then uses LSTM aggregation for dynamic gait and temporal averaging for static canonical features.
Results
GaitNet outperforms state-of-the-art methods on CASIA-B, USF, and FVG, and demonstrates gait-recognition advantages over face recognition in selected surveillance scenarios.
Takeaways & Limitations
The results support disentangled RGB-video representations as a viable approach for gait recognition across confounding conditions and challenging frontal views.
Takeaways & Limitations
GaitNet failures are attributed mainly to low-resolution and illumination conditions, while alignment and super-resolution improvements remain outside the study’s scope.
Abstract
from arXiv · showhide
Gait, the walking pattern of individuals, is one of the important biometrics modalities. Most of the existing gait recognition methods take silhouettes or articulated body models as gait features. These methods suffer from degraded recognition performance when handling confounding variables, such as clothing, carrying and viewing angle. To remedy this issue, we propose a novel AutoEncoder framework, GaitNet, to explicitly disentangle appearance, canonical and pose features from RGB imagery. The LSTM integrates pose features over time as a dynamic gait feature while canonical features are averaged as a static gait feature. Both of them are utilized as classification features. In addition, we collect a Frontal-View Gait (FVG) dataset to focus on gait recognition from frontal-view walking, which is a challenging problem since it contains minimal gait cues compared to other views. FVG also includes other important variations, e.g., walking speed, carrying, and clothing. With extensive experiments on CASIA-B, USF, and FVG datasets, our method demonstrates superior performance to the SOTA quantitatively, the ability of feature disentanglement qualitatively, and promising computational efficiency. We further compare our GaitNet with state-of-the-art face recognition to demonstrate the advantages of gait biometrics identification under certain scenarios, e.g., long distance/lower resolutions, cross viewing angles.
1 INTRODUCTION
The paper addresses gait recognition’s sensitivity to clothing, carrying, viewing angle, and speed by learning disentangled representations directly from RGB videos. It introduces GaitNet and the FVG frontal-view dataset, reporting superior performance across three benchmarks and advantages over face recognition in selected scenarios.
- 1 INTRODUCTION: Existing appearance- and model-based gait features are vulnerable to clothing, carrying, viewing angle, and walking-speed changes, motivating learned disentanglement.Handcrafted representations can lose gait information or retain redundant appearance information under changing walking conditions.
- 1 INTRODUCTION: GaitNet learns pose, canonical, and appearance features directly from RGB walking videos instead of relying on handcrafted silhouettes or skeletons.Pose and canonical features support recognition, while appearance captures clothing-related variation.
- 1 INTRODUCTION: The model combines LSTM-aggregated dynamic pose features with averaged static canonical features for video-to-video gait similarity.Similarity is computed by summing cosine distances between corresponding dynamic and static features.
- 1 INTRODUCTION: FVG targets challenging frontal-view RGB gait recognition and includes variations in viewing angle, walking speed, carrying, clothing, background, and time gaps.The database uses three frontal-view angles and is described as the first HD gait database with nearly twice as many subjects as existing RGB gait databases.
- 1 INTRODUCTION: GaitNet outperforms state-of-the-art methods on CASIA-B, USF, and FVG datasets.The paper also evaluates gait against face recognition and demonstrates gait’s strength for person recognition from surveillance-quality videos.
2 RELATED WORK
Prior gait-recognition work mainly uses silhouette-derived appearance representations or articulated body models, while newer RGB methods seek more discriminative learned features. GaitNet is positioned as a complementary approach that preserves dynamic and static information across multiple covariates, and the paper compares gait and face recognition across varied conditions.
- Gait Representation: Appearance-based methods such as GEI are simple and effective but suffer from intra-subject changes caused by clothing, carrying, view, and walking speed.GEI averages silhouette images, so frame-order and dynamic information are not retained.
- Gait Representation: GaitNet learns directly from raw RGB frames and combines complementary dynamic and static gait features to exploit information lost by GEI-based methods.The method is described as learning more discriminative representations from data with multiple covariates.
- Gait Databases: The paper compares FVG with established gait databases, including CASIA-B and USF, across dataset characteristics and captured variations.CASIA-B includes view, clothing, and carrying variations, while FVG is introduced for frontal-view evaluation.
- Disentanglement Learning: The work relates to disentanglement methods that separate semantic factors such as content and pose, but applies the idea specifically to gait representation.Its comparison emphasizes learning gait features from RGB data rather than relying solely on handcrafted silhouettes or pose-derived representations.
- Gait vs. Face recognition: Gait and face recognition are evaluated side by side across varied CASIA-B and FVG conditions to examine their respective strengths for covert identification at a distance.The study contrasts standalone state-of-the-art gait and face algorithms rather than focusing only on multimodal fusion.
3 PROPOSED APPROACH
GaitNet learns disentangled pose, canonical, and appearance representations from RGB video to separate gait identity from changing conditions. It aggregates pose over time and canonical features across frames into dynamic and static gait features for matching.
- 3.1 Overview: GaitNet addresses clothing, viewing-angle, and other confounding variations by learning gait features that remain discriminative across subjects while invariant to changing conditions.The design targets the problem that appearance can dominate gait representations, especially when training data contain limited clothing variation.
- 3.3 Temporal Aggregation: A multi-layer LSTM aggregates pose trajectories into a dynamic gait feature, while averaging canonical features produces a static gait feature for final matching.The dynamic feature captures temporal changes in body-part trajectories; the static feature summarizes subject-specific canonical characteristics.
- 3.2 Feature Disentanglement: The encoder splits each frame into appearance, canonical, and pose features, with canonical features representing subject-specific body characteristics and pose features representing frame-level gait information.Canonical features are intended to remain constant across frames and conditions, while pose features vary across frames but are assumed constant across conditions.
- 3.2 Feature Disentanglement: Cross reconstruction combines appearance and canonical features from one frame with pose from another, encouraging pose-irrelevant features to remain stable within a video.This objective also ensures that the three features remain sufficiently representative to reconstruct video frames.
- 3.2 Feature Disentanglement: Pose similarity and canonical consistency losses constrain pose and canonical representations across videos and conditions while preserving canonical discriminativeness across subjects.Canonical consistency explicitly targets constancy across frames and same-subject conditions, whereas pose similarity discourages clothing and body information from leaking into pose features.
4 FRONT-VIEW GAIT (FVG) DATABASE
FVG is a newly collected frontal-view gait database designed to evaluate recognition under reduced motion cues and challenging walking variations.
- Collection: FVG contains 226 subjects, each performing 12 walks toward a camera from approximately 16 meters away.Videos were recorded at 1,080 × 1,920 resolution and 15 FPS, with body heights ranging from 101 to 909 pixels.
- Collection: The database combines three camera angles with variations in walking speed, carrying, clothing, and multi-person walking.The 12 walks cover angles of −45°, 0°, and 45° from the camera’s optical axis.
- Protocols: FVG evaluation uses 136 subjects for training and 90 for testing, with normal frontal walking as the gallery condition.Five protocols probe walking speed, carrying a bag while wearing a hat, changing clothes, multiple persons, and all variations.
- Protocols: The protocols target frontal-view recognition, where consecutive frames show smaller visual changes than in side-view walking.Performance is measured using True Accept Rate at 1% and 5% false-accept rates.
5 EXPERIMENTAL RESULTS
The experiments evaluate GaitNet on established gait databases while visualizing whether its learned features preserve and separate appearance, pose, and body information.
- Evaluation Setup: GaitNet is evaluated on CASIA-B, USF, and FVG, enabling comparisons with prior methods on widely used databases.The comparisons follow the respective experimental protocols of the baselines.
- Feature Visualization: Feature visualizations reconstruct input frames while showing that canonical features preserve subject-specific body shape and appearance features recover clothing.The figure compares frontal and side views under normal and clothing conditions.
- Feature Visualization: Pairwise decoding shows that pose-irrelevant features preserve appearance and body information, whereas pose features preserve pose information.Synthetic images combine pose-irrelevant features from one frame with pose features from another.
5.1 Ablation Study
Ablations show that GaitNet’s disentanglement and incremental identity objectives improve feature separability and recognition, while canonical and dynamic gait features are more discriminative than appearance features.
- Feature Visualization: Canonical and dynamic gait features show stronger subject discrimination than appearance and frame-level pose features in t-SNE visualizations.Canonical features exhibit compact within-subject variation and separable between-subject differences.
- Loss Function’s Impact on Recognition Performance: 85.6% accuracy with cross-reconstruction and pose-similarity losses substantially exceeds 60.2% with cross-reconstruction alone and 56.0% without disentanglement losses.The ablation identifies pose-similarity loss as the more critical of the two disentanglement losses for identification.
- Loss Function’s Impact on Recognition Performance: 92.1% rank-1 accuracy is achieved by the incremental identity loss, exceeding 82.6% for averaged LSTM output and 72.5% for single-step output.The corresponding t-SNE visualization shows more discriminative dynamic features throughout the sequence.
- Classification Features: 85.9% rank-1 accuracy from dynamic gait features and 76.3% from static gait features exceed 33.4% from appearance features.The results support using learned canonical and pose information for classification while appearance has limited discriminative power.
- Feature Fusion: A score weight of about 0.2 performs best on CASIA-B, while about 0.75 performs best on FVG; the authors nevertheless use α=0.5 thereafter.CASIA-B favors the static feature relatively more, whereas FVG favors the dynamic feature.
5.2 Evaluation on Benchmark Datasets
Across CASIA-B, USF, and FVG, GaitNet achieves strong recognition under viewpoint and appearance changes, with canonical features improving robustness over the preliminary model.
- CASIA-B: GaitNet achieves the best CASIA-B performance among the compared methods across multiple viewing angles and normal, carrying, and clothing conditions.The comparison follows Protocol 1 and includes GaitNet-pre and other state-of-the-art methods.
- CASIA-B: 87.3% average Rank-1 accuracy is achieved across 10 CASIA-B viewing angles, with 14% improvements over GaitNet-pre at both 0° and 180°.The authors attribute the gains to more discriminative canonical body-shape information under different views.
- CASIA-B: GaitNet outperforms the compared methods on both CASIA-B appearance-condition subsets, while other methods show a larger performance gap between carrying-bag and clothing conditions.The clothing protocol is described as particularly challenging.
- USF: GaitNet achieves 99.7% identification accuracy on the USF Probe A evaluation.The experiment uses a non-overlapping training and test partition following the comparison setting.
- FVG: On FVG, GaitNet reaches state-of-the-art performance across five protocols and surpasses classic, CNN-based, and pose-estimation LSTM baselines.Frontal-view recognition is especially difficult for GEI-based methods because of limited walking information, while GaitNet improves over GaitNet-pre.
5.3 Comparison to Face Recognition
GaitNet is compared with ArcFace across CASIA-B and FVG protocols and viewing conditions. Gait is stronger early and under difficult views or hat variation, while face can recover late when resolution and frontal views improve.
- CASIA-B: GaitNet is significantly superior to face recognition at the beginning of CASIA-B sequences across frontal-frontal, side-side, and side-frontal scenarios.Its static gait feature captures body shape in low-resolution images, while faces are less discriminative early in the sequence.
- CASIA-B: Face recognition continues to struggle through the end of CASIA-B sequences for side-side and side-frontal viewing.
- FVG: Gait outperforms face recognition throughout the FVG NM-BGHT protocol, where videos include carrying bags and hats.The authors attribute this advantage to hats strongly affecting face recognition but not gait recognition.
- FVG: Face recognition outperforms gait during the last 20% of the FVG ALL* protocol.At that point, low resolution is less problematic and FVG provides frontal-view faces.
5.4 Runtime Speed
The runtime experiment evaluates five gait recognition methods on FVG using a common GPU and PyTorch implementation. Runtime is reported in milliseconds per frame, but the supplied passages do not state the comparative values.
- Runtime evaluation: Runtime is measured for five gait recognition methods processing one FVG video on the same GeForce GTX 1080 Ti GPU.The implementations use PyTorch and enable GPU batch processing, with batch size equal to the number of probe samples.
- Runtime results: The supplied runtime passages specify the evaluation protocol but do not provide comparative runtime values or identify the fastest method.
- Runtime results: Table 11 reports runtime comparisons on FVG in milliseconds per frame.
6 CONCLUSION
The paper concludes that GaitNet disentangles appearance and gait representations from RGB frames and uses temporal modeling to represent video sequences. Experiments on three datasets report superior results and support the approach’s generalization and promise.
- Conclusion: GaitNet disentangles appearance and gait representations from raw RGB frames and uses a multi-layer LSTM to exploit temporal information.
- Conclusion: Experiments on CASIA-B, USF, and FVG compare the proposed method extensively with state-of-the-art methods.
- Conclusion: The reported superior results indicate generalization and promise for the proposed feature-disentanglement approach.