Source-linked AI summary
Gait Recognition via Disentangled Representation Learning
Ziyuan Zhang, Luan Tran, Xi Yin, Yousef Atoum, Xiaoming Liu, Jian Wan, Nanxin Wang
TL;DR
Gait recognition needs identity features that remain useful despite clothing, carrying, and viewpoint changes, especially in frontal-view RGB video. This paper introduces GaitNet, which disentangles pose and appearance with an autoencoder-based CNN and aggregates pose temporally with an LSTM, alongside the FVG dataset. Across CASIA-B, USF, and FVG, the method reports superior results to state-of-the-art approaches, while its disentanglement relies on the absence of discrete gait or appearance labels.
Problem
Gait recognition requires features that are discriminative across subjects yet invariant to appearance variations such as clothing, carrying, and viewing angle.
Method
GaitNet uses an autoencoder-based CNN to disentangle pose and appearance from RGB frames, then a multi-layer LSTM aggregates pose features into a gait representation.
Results
GaitNet outperforms state-of-the-art methods on CASIA-B, USF, and FVG, including 81.2% versus 65.4% TDR at 1% FAR against the PE-LSTM baseline on FVG.
Takeaways & Limitations
The results support feature disentanglement from raw RGB video as a promising approach for gait recognition under varied covariates and frontal-view conditions.
Takeaways & Limitations
Gait and appearance labels are unavailable because walking patterns and clothing cannot be defined as discrete classes.
Abstract
from arXiv · showhide
Gait, the walking pattern of individuals, is one of the most important biometrics modalities. Most of the existing gait recognition methods take silhouettes or articulated body models as the gait features. These methods suffer from degraded recognition performance when handling confounding variables, such as clothing, carrying and view angle. To remedy this issue, we propose a novel AutoEncoder framework to explicitly disentangle pose and appearance features from RGB imagery and the LSTM-based integration of pose features over time produces the gait feature. In addition, we collect a Frontal-View Gait (FVG) dataset to focus on gait recognition from frontal-view walking, which is a challenging problem since it contains minimal gait cues compared to other views. FVG also includes other important variations, e.g., walking speed, carrying, and clothing. With extensive experiments on CASIA-B, USF and FVG datasets, our method demonstrates superior performance to the state of the arts quantitatively, the ability of feature disentanglement qualitatively, and promising computational efficiency.
1. Introduction
Gait recognition seeks features that remain discriminative across appearance and viewpoint changes, but existing silhouette- and skeleton-based representations are limited. GaitNet addresses this by disentangling pose from appearance in RGB video, aggregating pose over time, and introducing the frontal-view FVG database.
- Motivation: Gait recognition must remain invariant to clothing, carrying, viewing angle, and walking-speed variations while preserving identity information.Silhouette-based GEI and skeleton-based features each have documented sensitivity, computational, or pose-estimation limitations.
- Method: GaitNet automatically disentangles pose and appearance features from RGB frames using an autoencoder-based CNN with novel loss functions.The pose representation is intended for gait recognition rather than relying on handcrafted GEI or skeleton features.
- Method: A multi-layer LSTM aggregates pose features over time to produce a sequence-level gait representation.The contribution specifically identifies multi-layer LSTM aggregation as part of GaitNet.
- Dataset: FVG is a high-definition frontal-view gait database covering viewing angles, walking speeds, carrying, clothing, background, and time-gap variations.The dataset nearly doubles the number of subjects compared with prior RGB gait databases.
- Results: GaitNet outperforms state-of-the-art methods on the CASIA-B, USF, and FVG benchmarks.The paper reports this as a cross-dataset result rather than as a single-protocol comparison.
2. Related Work
Prior gait-recognition work mainly uses silhouette-derived appearance representations or pose-based models, both of which face limitations under covariates. GaitNet instead learns disentangled gait and appearance information directly from RGB frames without adversarial training or gait and appearance labels.
- Gait Representation: Appearance-based methods such as GEI and GEnI use silhouette masks but can suffer from clothing, carrying, viewpoint, and walking-speed changes.GEI represents a video using an averaged silhouette image, favoring simplicity but losing some dynamic information.
- Gait Representation: Model-based methods use articulated body skeletons and can be more robust to covariates, but depend on pose-estimation accuracy and require higher computational cost.These methods represent gait through estimated body structure rather than raw RGB appearance.
- RGB-Based Learning: GaitNet learns gait information from raw RGB frames, which retain richer information and support learning representations across multiple covariates.The paper contrasts this CNN-based approach with a Conditional Random Field method for RGB gait recognition.
- Gait Databases: FVG differs from prior databases by focusing on near-frontal views while including walking speed, carrying, clothing, cluttered background, and time variations.The database uses three different near-frontal-view angles toward the camera.
- Disentanglement Learning: Unlike several disentanglement approaches, GaitNet uses one encoder and novel loss functions without adversarial training.The method also does not use discrete gait or appearance labels because those properties cannot be defined as discrete classes.
3. Proposed Approach
The approach disentangles frame-level pose and appearance from RGB video, then aggregates pose dynamics over time for gait identification. It combines cross-reconstruction, gait-similarity, and temporal identity objectives in a jointly trained encoder-decoder and LSTM system.
- Gait recognition requires features that distinguish subjects while remaining invariant to viewing angle, walking speed, and appearance.
- Appearance and Pose Feature Disentanglement: GaitNet uses an encoder-decoder to split each frame into appearance feature fa and pose feature fg, which together reconstruct the input.
- Appearance and Pose Feature Disentanglement: Cross reconstruction combines one frame’s appearance feature with another frame’s pose feature to reconstruct the latter frame and encourage appearance consistency across video frames.
- Appearance and Pose Feature Disentanglement: Gait-similarity training uses multiple videos of the same subject to reduce appearance leakage into pose features when appearance changes between conditions.
- Gait Feature Learning via Aggregation: A multi-layer LSTM processes pose features sequentially to model temporal walking dynamics and produce a sequence-based gait representation for identity classification.
- Gait Feature Learning via Aggregation: The method supervises intermediate LSTM outputs through incremental identity loss and uses averaged LSTM outputs as the gait feature to improve robustness to stopping instances.
- Implementation Details: At test time, the LSTM gait feature is matched using cosine similarity; the system uses segmented frames from Mask R-CNN soft foreground masks.
4. Front-View Gait Database
The FVG database targets frontal-view RGB gait recognition under limited frame-to-frame gait cues. It includes subject, session, viewpoint, speed, carrying, clothing, background, and time-gap variations, with five evaluation protocols.
- Collection: FVG contains 226 subjects walking toward a camera in 12 videos each, captured at 1,080 × 1,920 resolution.
- Collection: The database spans three sessions with variations in walking speed, carrying status, clothing or shoes, background, and time gaps.
- Protocols: Frontal-view walking is challenging because consecutive frames can differ less than in side-view walking, reducing exploitable gait information.
- Protocols: The five protocols evaluate Walking Speed, Carrying Bag, Changing Clothes, Cluttered Background, and all variations.
- Protocols: Each protocol trains on the first 136 subjects, tests on the remaining 90, and uses normal frontal-view walking video 2 as the gallery.
5. Experiments
Experiments evaluate GaitNet across ablations, cross-view and cross-condition CASIA-B protocols, FVG, and efficiency comparisons. Results show improved disentanglement, strong recognition across variations, and faster processing than pose-estimation methods.
- Ablation Study: 81.2% TDR at 1% FAR exceeds the pose-estimation PE-LSTM baseline’s 65.4% on FVG’s ALL protocol.The comparison indicates that GaitNet’s learned pose feature captures more discriminative information than joint locations alone.
- Ablation Study: GaitNet’s full disentanglement model improves CASIA-B cross-condition accuracy from 56.0% to 85.6%.Adding Lxrecon raises accuracy to 60.2%, while combining Lxrecon and Lgait-sim reaches 85.6%.
- CASIA-B: GaitNet achieves the best performance across CASIA-B comparisons spanning multiple view angles and NM, BG, and CL conditions.Protocol 1 evaluates crossing view angles and three appearance variations using separate comparison models.
- CASIA-B: 81.8% average rank-1 accuracy is achieved across ten CASIA-B view angles under Protocol 2.At 0° and 180° views, the improvement margins are 30% and 26%, respectively.
- FVG: On FVG, GaitNet outperforms classic and CNN-based methods across five protocols, while clothing remains the most challenging variation.GEI-based methods particularly suffer under frontal view because of limited walking information.
- Efficiency: GaitNet is significantly faster than the pose-estimation method when processing USF videos on the same GPU.The efficiency comparison attributes the difference partly to Mask R-CNN efficiency and the slower AlphaPose version required for gait recognition.
6. Conclusions
The paper presents GaitNet, an autoencoder-based method that disentangles appearance and gait features from RGB frames, then uses multi-layer LSTMs to model temporal information for video-level gait representation.
- GaitNet disentangles appearance and gait feature representations from raw RGB frames using an autoencoder-based method.
- A multi-layer LSTM structure explores temporal information to generate a gait representation for each video sequence.
- The method is extensively compared with state-of-the-art methods on CASIA-B, USF, and the collected FVG datasets.