PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Gait Recognition: A Survey

Alireza Sepas-Moghaddam, Ali Etemad

arXiv:2102.09546v2cs.CV

TL;DR

Deep gait recognition is affected by appearance, viewpoint, occlusion, and environmental variation, motivating systematic organization of the field. This survey reviews datasets, protocols, methods, and challenges through a four-dimensional taxonomy, reporting dataset-specific state-of-the-art results and identifying cross-dataset evaluation as an important gap.

  • Problem

    Vision-based gait recognition is challenged by appearance, viewpoint, occlusion, and environmental variations that make segmentation and recognition more difficult.

  • Method

    The paper systematically surveys deep gait recognition methods, datasets, and evaluation protocols using a taxonomy spanning body, temporal, feature, and neural-architecture representations.

  • Results

    The survey identifies dataset-specific leading methods, with reaching 90.4% average performance on CASIA-B and 99.9% on OU-ISIR, while reaches 89.18% on OU-MVLP.

  • Takeaways & Limitations

    The taxonomy organizes the deep gait recognition landscape, while cross-dataset evaluation remains needed to assess generalizability to varied real-world conditions.

Abstract

from arXiv · show

Gait recognition is an appealing biometric modality which aims to identify individuals based on the way they walk. Deep learning has reshaped the research landscape in this area since 2015 through the ability to automatically learn discriminative representations. Gait recognition methods based on deep learning now dominate the state-of-the-art in the field and have fostered real-world applications. In this paper, we present a comprehensive overview of breakthroughs and recent developments in gait recognition with deep learning, and cover broad topics including datasets, test protocols, state-of-the-art solutions, challenges, and future research directions. We first review the commonly used gait datasets along with the principles designed for evaluating them. We then propose a novel taxonomy made up of four separate dimensions namely body representation, temporal representation, feature representation, and neural architecture, to help characterize and organize the research landscape and literature in this area. Following our proposed taxonomy, a comprehensive survey of gait recognition methods using deep learning is presented with discussions on their performances, characteristics, advantages, and limitations. We conclude this survey with a discussion on current challenges and mention a number of promising directions for future research in gait recognition.

1 INTRODUCTION

Gait recognition exploits walking patterns as biometric cues but must handle appearance, viewpoint, occlusion, environment, and low-resolution challenges. This survey addresses recent deep-learning advances through a systematic review, taxonomy, dataset and protocol coverage, comparisons, and future directions.

  • 1 INTRODUCTION: Vision-based gait recognition captures subjects from a distance without cooperation, but often suffers from low spatial resolution.The paper focuses on vision-based systems relying mainly on deep learning.
  • 1 INTRODUCTION: Appearance, viewpoint, occlusion, and environmental variation make gait segmentation and recognition more difficult.Examples include carried objects, clothing, camera viewpoint, self-occlusion, complex backgrounds, and lighting changes.
  • 1 INTRODUCTION: Deep learning methods increasingly replaced non-deep approaches after 2015 and dominated gait recognition publications by 2019 and 2020.The paper illustrates this shift through Figures 1 and 2 and traces milestones including GaitNet in 2016.
  • 1 INTRODUCTION: The survey reviews deep gait recognition through a four-dimensional taxonomy covering body, temporal, feature, and neural-architecture representations.It also covers datasets, test protocols, state-of-the-art comparisons, challenges, and future research directions.
  • 1 INTRODUCTION: The paper reviews 15 public vision-based datasets, compares reported state-of-the-art results, and identifies open challenges for future research.Its taxonomy-guided review is intended to support topic exploration and algorithm design.

2 REVIEW METHODOLOGY

The survey uses a structured search and screening process to collect deep gait-recognition literature, supplemented by forward and backward citation searches. It distinguishes subject-dependent from subject-independent evaluation protocols.

  • 2 REVIEW METHODOLOGY: Candidate papers were found through Google Scholar and major digital libraries using combinations of gait-recognition and deep-learning search terms.The sources included IEEE Xplore, ACM Digital Library, ScienceDirect, and CVF Open Access.
  • 2 REVIEW METHODOLOGY: The review excluded papers that did not use deep learning and restricted publication searches to results after 2014.Forward and backward searches were repeated with newly identified resources.
  • 2 REVIEW METHODOLOGY: Subject-dependent protocols include samples from all subjects in training and testing, whereas subject-independent protocols use disjoint test subjects.Subject-independent evaluation further divides test data into gallery and probe sets.

3 TEST PROTOCOLS AND DATASETS

Gait datasets vary in modality, scale, viewpoints, environments, appearance conditions, and walking factors, while evaluation protocols define how training, gallery, probe, and test data are arranged. The survey emphasizes standardized protocols and broad dataset coverage for meaningful comparison.

  • 3 TEST PROTOCOLS AND DATASETS: Subject-dependent evaluation shares subjects between training and testing, while subject-independent evaluation tests on subjects excluded from training.Subject-independent protocols commonly use separate gallery and probe sets.
  • 3 TEST PROTOCOLS AND DATASETS: Large and diverse datasets are preferred because they support effective training of deep neural networks.The survey summarizes dataset modalities, subject and sequence counts, viewpoints, and environmental conditions.
  • 3.2 Datasets: Some datasets provide specialized protocols, whereas SOTON has no predefined test protocol and is divided differently across studies.CASIA-A commonly uses cross-view testing, and CASIA-C and OU-ISIR Speed use cross-speed evaluation.
  • 3.2 Datasets: CASIA-B provides multi-view RGB and silhouette data for 124 persons under normal, coat, and bag walking conditions.It includes 11 viewing angles from 0° to 180° and is the most widely used gait dataset.
  • 3.2 Datasets: The reviewed datasets target variations including speed, clothing, carried objects, viewpoint, background, acquisition time, and modality.Examples include OU-ISIR Speed, OU-ISIR Clothing, OU-ISIR LP Bag, CASIA-E, TUM GAID, and USF HumanID.
  • 3.2 Datasets: OU-MVLP contains 259,013 silhouette sequences acquired from 14 views across two sessions per subject.Its subjects span ages 2 to 87 years with an approximately balanced gender distribution.

4 PROPOSED TAXONOMY

Existing taxonomies organize gait-recognition methods by sensors, classifiers, covariates, or feature types, but do not specifically focus on successful deep-learning methods. The survey therefore proposes a four-dimensional taxonomy for the deep gait-recognition landscape.

  • 4 PROPOSED TAXONOMY: Earlier taxonomies classify gait methods using sensor and classifier types, covariate factors, or feature types.These perspectives organize the literature but do not specifically center on deep gait recognition.
  • 4 PROPOSED TAXONOMY: The proposed taxonomy characterizes deep gait-recognition methods through body representation, temporal representation, feature representation, and neural architecture.The paper describes each dimension in the following sections.

4.1 Body Representation

Body representation organizes gait recognition inputs around silhouettes or skeletons, each offering different descriptive and robustness properties.

  • Silhouettes are the most frequently used representation and efficiently describe body state in individual frames.They are computed by background subtraction and binarization, with low computational cost.
  • Silhouette sequences encode gait characteristics including speed, cadence, leg angles, cycle time, step length, and stride length.
  • Silhouettes focus recognition on gait rather than clothing or other nongait factors, but remain sensitive to appearance changes.
  • Skeletons provide static and dynamic joint-based features and are generally more robust to viewpoint changes than silhouettes.They can be captured with depth cameras or estimated using pose-estimation methods.

4.2 Temporal Representation

Temporal representation distinguishes compact templates from sequence volumes, balancing frame-count independence against preservation of temporal order and relationships.

  • Temporal representations commonly use templates or volumes to encode information from gait sequences.
  • Templates aggregate walking information from at least one gait cycle into a single map, making recognition independent of sequence frame count after creation.
  • Templates may be generated at a network’s initial layer or after convolution and pooling layers as convolutional templates.
  • Examples include GEI, CGI, FDEI, GEnI, and PEI temporal gait templates.
  • Sequence volumes preserve frame order and relationships, learned either with recurrent strategies or with 3D tensors.

4.3 Feature Representation

Feature representation separates holistic learning from partial-region learning, with partial representations addressing uneven regional contributions and robustness to occlusion and viewpoint changes.

  • Feature representation defines the support region for learning as either global or partial.Partial regions include patches, body components, and vertical or horizontal bins.
  • Global representations learn silhouettes or skeletons holistically, whereas partial representations divide gait data into local regions for further processing.
  • Global methods tend to be more sensitive to occlusions, appearance changes, and missing key body parts.
  • Learning the differing importance of partial regions can improve overall gait recognition performance.
  • Relationships among partial features preserve scale, rotation, and location attributes, improving robustness to orientation and view changes.

4.4 Neural Architectures

Deep gait recognition uses multiple neural architectures, with CNNs most prevalent and other models contributing temporal modeling, latent representations, or appearance and viewpoint transformation.

  • CNNs are the most widely used architecture for gait recognition and learn embeddings from silhouette- or skeleton-based body shapes.
  • State-of-the-art gait recognition models do not need very deep CNNs because silhouettes and skeletons lack considerable input complexity.
  • DBNs extract hierarchical representations by stacking restricted Boltzmann machines and have been applied to silhouette-derived gait features.
  • RNNs learn temporal relationships using memory states and gating, with LSTM and GRU being the most widely used architectures.
  • RNN-based systems either model joint-position movement, combine RNNs with CNNs, or learn relationships among partial gait-template representations.
  • DAEs learn bottleneck or latent representations through encoder-decoder transformations between input data and reconstructed data.
  • GANs synthesize realistic samples and can transform viewpoint or clothing while attempting to preserve identity features.Such transformations often improve gait recognition performance, but identity preservation remains a central challenge.

4.4.6 Capsule Networks

Capsule Networks address CNN limitations through capsule activations and routing, while hybrid architectures combine complementary networks for gait representation and recognition.

  • Capsule Networks: CapsNets address scalar activations and pooling-based information-routing limits using capsule activations and routing-by-agreement.Capsules explicitly encode viewpoint-invariant relationships among object parts.
  • Neural Architectures: 3D CNNs learn spatio-temporal dynamics from stacked gait frames but lack flexibility for variable-length sequences.Multiple 3D CNNs, local temporal pooling, and multiscale integration have been used to mitigate this limitation.
  • Neural Architectures: GCNs jointly model human-body structure and temporal relationships in gait sequence volumes to learn viewpoint- and appearance-robust features.Their graph structure matches the hierarchical and graph-like nature of the human body.
  • Hybrid Architectures: Hybrid deep networks commonly combine CNNs with RNNs, DAEs with GANs or RNNs, and RNNs with CapsNets.CNN-RNN architectures learn temporal relationships after spatial encoding, while other hybrids target generation, disentanglement, or feature routing.

5 STATE-OF-THE-ART

The survey organizes deep gait recognition methods by body, temporal, feature, and neural-architecture dimensions, revealing dominant design patterns and benchmark performance.

  • Body Representation: Silhouettes represent over 81% of deep gait methods, compared with 13% using skeletons and approximately 5% using both.The survey notes that high-performing methods cited in its analysis all adopt silhouette representations.
  • Temporal Representation: Gait templates account for 70% of methods, while around 30% use sequence volumes to preserve frame order and relationships.GEI and set-pooling are the most common gait templates.
  • Feature Representation: Over 87% of methods use global feature representations, while recent high-performing approaches use partial representations from local regions.The survey identifies partial representation learning as a promising direction for discriminating key gait features.
  • Neural Architectures: 2D CNNs comprise 48% of published solutions, while hybrid methods comprise 26%, including 16% CNN-RNN combinations.3D CNNs and GANs each account for 8%, with smaller shares for DAEs, RNNs, CapsNets, DBNs, and GCNs.
  • Loss Functions: Cross-entropy is used by 20% of solutions and triplet loss by 17%, with triplet loss appearing in several recent state-of-the-art methods.Triplet loss compares anchor, positive, and negative samples to separate identities in feature space.
  • Datasets: CASIA-B appears in 80% of published literature, followed by OU-ISIR at 40% and OU-MVLP at 18% within two years of introduction.OU-ISIR LP Bag and TUM GAID each account for 5% of methods in the supplied passage.
  • Performance Comparison: On standard protocols, method reports 90.4% average performance on CASIA-B and 99.9% on OU-ISIR, while method reports 89.18% on OU-MVLP.Several near-state-of-the-art methods use multiple neural architectures or multiple loss functions.

6 VULNERABILITY TO ADVERSARIAL ATTACKS

Deep gait recognition systems are vulnerable to adversarial attacks that synthesize realistic, similar gait videos or images to manipulate recognition outcomes.

  • Attack Models: Traditional spoofing relies on trained impostors imitating target walking styles, whereas adversarial attacks synthesize gait inputs with quantitative similarity and visual realism.GANs are identified as powerful tools for image and video manipulation.
  • Attack Models: The first reported gait-specific adversarial attack used a GAN with separate foreground and background encoder-decoder processing.Source-video foreground was segmented and combined with a target background before generation.
  • Implications: Results from and [187] suggest adversarial attacks can surprisingly degrade deep gait recognition performance, motivating efficient countermeasures.The survey presents this vulnerability as a real threat to gait recognition systems.

7 CHALLENGES AND FUTURE RESEARCH DIREC-

The survey identifies unresolved challenges involving disentanglement, labels, data scale, generalization, viewpoints, and multimodal information, and outlines directions for addressing them.

  • Disentanglement: Complex interactions among occlusion, viewpoints, appearance, sequence order, motion, and lighting complicate gait data, while most methods lack explicit disentanglement.Few methods separate identity from non-identity components into meaningful disjoint variables.
  • Disentanglement: Novel generative models and loss functions could improve representations by explicitly disentangling identity and non-identity components.The survey presents this as a direction for further research rather than an established result.
  • Learning Without Labels: Most deep gait methods require labeled data, although labels may be expensive and time-consuming to obtain in real-world applications.Self-supervised learning is proposed for using unlabeled gait data to learn more efficient and generalizable representations.
  • Data Synthesis: Deep gait methods require large datasets for effective training and reliable evaluation, and commonly rely on CASIA-B, OU-ISIR, and OU-MVLP.The survey discusses GAN-based synthesis for dataset creation, augmentation, privacy, and increased variation.
  • Data Synthesis: OU-ISIR and OU-MVLP mainly provide normal walking without occlusion, carrying, or clothing variation, limiting generalization to such test conditions.The survey states that solutions trained on these datasets usually fail to generalize well under appearance and environmental variations.
  • Generalization: Cross-dataset gait recognition remains unperformed on well-known datasets because notable methods use the same dataset for training and testing.This boundary matters because deployed systems encounter test or runtime data beyond their training dataset.
  • Multi-view Recognition: Most current methods perform single-view recognition and ignore inter-view relationships, while many multi-view descriptors assume fixed camera topologies.Real-world data collection can involve unpredictable viewing angles.
  • Multimodal Recognition: Fusing gait with face or ear biometrics is explored because these modalities may be less sensitive to some appearance, clothing, viewpoint, and occlusion variations.The supplied passage describes multimodal fusion as existing related work.

8 SUMMARY

The survey organizes deep gait recognition research through a four-dimensional taxonomy and reviews representative methods, datasets, evaluation protocols, challenges, and future directions.

  • The survey introduces four taxonomy dimensions: body representation, temporal representation, feature representation, and neural architectures.
  • It reviews representative deep gait recognition methods, discussing their characteristics, advantages, and limitations.
  • It examines commonly used datasets, evaluation protocols, and corresponding performance results reported in the literature.
  • The survey concludes by identifying current challenges and promising future research directions.
Loading 2102.09546v2…