PaperlayerView exact revision on arXiv

Source-linked AI summary

Gait Recognition Based on Deep Learning: A Survey

Claudio Filipi Gonçalves dos Santos, Diego de Souza Oliveira, Leandro A. Passos, Rafael Gonçalves Pires, Daniel Felipe Silva Santos, Lucas Pascotti Valem, Thierry P. Moreira, Marcos Cleison S. Santana, Mateus Roder, João Paulo Papa, Danilo Colombo

arXiv:2201.03323v1cs.CVcs.LG

TL;DR

Gait recognition offers identification without individual cooperation or specialized equipment, but conventional methods face feature-extraction and recognition challenges. This survey reviews deep-learning approaches and datasets; it finds 2D-CNNs and silhouette representations predominate in the surveyed work.

  • Problem

    Gait recognition can identify people without cooperation or specialized equipment, but methods face challenges including handcrafted feature extraction and variations such as viewing angle and occlusion.

    Source: §1 · p. 2

  • Method

    The survey reviews recent deep-learning gait-recognition approaches and architectures and provides an illustrated, categorized catalog of public datasets.

    Source: §1 · p. 3

  • Results

    Approximately 50% of surveyed solutions use 2D-CNNs, while silhouettes appear in more than 85% of the available solutions.

    Source: §3.7 · p. 17

  • Takeaways & Limitations

    The survey describes nine commonly used gait datasets and their extraction processes and limitations, including efforts to address factors such as clothing, shoes, bags, and viewing angle.

    Source: §5 · p. 28

    The surveyed datasets use controlled environments and one person per video, so models may lack or misbehave in real-world operation.

    Source: §5 · p. 29

Abstract

from arXiv · show

In general, biometry-based control systems may not rely on individual expected behavior or cooperation to operate appropriately. Instead, such systems should be aware of malicious procedures for unauthorized access attempts. Some works available in the literature suggest addressing the problem through gait recognition approaches. Such methods aim at identifying human beings through intrinsic perceptible features, despite dressed clothes or accessories. Although the issue denotes a relatively long-time challenge, most of the techniques developed to handle the problem present several drawbacks related to feature extraction and low classification rates, among other issues. However, deep learning-based approaches recently emerged as a robust set of tools to deal with virtually any image and computer-vision related problem, providing paramount results for gait recognition as well. Therefore, this work provides a surveyed compilation of recent works regarding biometric detection through gait recognition with a focus on deep learning approaches, emphasizing their benefits, and exposing their weaknesses. Besides, it also presents categorized and characterized descriptions of the datasets, approaches, and architectures employed to tackle associated constraints.

1 INTRODUCTION

Gait recognition can identify people from movement without cooperation or specialized equipment, but remains difficult amid viewing-angle changes, occlusions, and other variation. This survey reviews recent deep-learning approaches, architectures, and datasets, addressing a gap in earlier gait surveys.

1 INTRODUCTION

  • Gait recognition uses body measurements and movement patterns, and its silhouette-based identification is difficult to reproduce.It can also operate without individual cooperation or high-resolution images and specialized equipment.Source: §1 · p. 2
  • Gait information can be collected from a distance and without sophisticated equipment, making it useful when identification is unassisted and commonly cheaper than other approaches.The introduction links lower equipment requirements to surveillance systems and cellphone accelerometers.Source: §1 · p. 2
  • Handcrafted gait methods face viewing-angle changes, intra-class variation, occlusions, shadows, and body-segment localization challenges, while deep learning reduces reliance on manually extracted features.The introduction describes deep learning as producing strong gait-recognition results and addressing these challenges.Source: §1 · p. 2
  • Earlier gait surveys covered general, vision-based, surveillance, re-identification, or wearable-sensor topics, but did not focus on deep learning or several architectures beyond CNNs and LSTMs,,,,,.The survey identifies Autoencoders, Capsule Networks, Deep Belief Networks, and GANs [23] among architectures that earlier reviews often left out.Source: §1 · pp. 2–3
  • The survey aims to review recent deep-learning gait-recognition work, explain biometric and gait concepts and tools, and catalog public datasets.Its paper selection used architecture-related search terms, discarded works with similar techniques and architectures, and limited studies to those published within five years.Source: §1 · p. 3

2 THEORETICAL BACKGROUND

The theoretical background surveys biometric identification, gait recognition, and deep-learning architectures, framing gait as a passive alternative and deep learning as a response to handcrafted-feature constraints. It outlines biometric criteria and trade-offs, representative approaches across biometric modalities, and architectures used for gait recognition.

2.1 Biometric Identification

  • Biometric traits are assessed for universality, distinctiveness, permanence, and collectability, alongside practical performance, acceptability, and circumvention concerns [39].Image-based systems can be fooled with simple images, whereas vein recognition is described as more fraud-resistant but requires specialized equipment.Source: §2.1 · pp. 3–4, §2.1.5 · p. 6

2.1.1 Fingerprint Recognition. / 2.1.2 Iris Recognition.

  • Fingerprints rely on distinctive, stable ridge and furrow patterns, while iris recognition is described as accurate, low-cost, convenient, and contact-free,.Fingerprint systems face non-ideal-condition accuracy and spoofing challenges; one iris method reported hit rates above 83%,.Source: §2.1.1 · p. 4, §2.1.2 · p. 4

2.1.3 Face Recognition.

  • Face recognition moved from holistic and local-descriptor methods toward deep learning, with DeepFace reaching 97.35% accuracy against one human competitor’s 97.53%.Face recognition still faces expression, pose, illumination, aging, and obstruction challenges.Source: §2.1.3 · p. 5

2.1.4 Multimodal Biometric Recognition. / 2.1.5 Other Identifiers.

  • Multimodal biometrics combines distinct biometric methods and auxiliary information to improve performance and reliability over a single technique.Other surveyed identifiers include ear, heartbeat, gesture, eye movement, and nose features; nose-based identification may be altered by plastic surgery,,,,.Source: §2.1.4 · p. 5, §2.1.5 · p. 6

2.2.1 Convolutional Neural Networks. / 2.2.2 Capsule Networks. / 2.2.3 Recurrent Neural Networks.

  • CNNs stack convolutional kernels for image features and classification, CapsNets use routing by agreement to preserve spatial relationships, and gated RNNs suit long gait sequences [32], [10].GRUs use fewer gates than LSTMs and can reach similar results with reduced computational burden and faster training and inference [10].Source: §2.2.1 · p. 7, §2.2.2 · p. 7, §2.2.3 · p. 8

2.2.4 Autoencoders. / 2.2.5 Deep Belief Networks. / 2.2.6 Generative Adversarial Networks. / 2.2.7 Deep learning techniques’ summary.

  • Autoencoders encode inputs into smaller feature spaces and reconstruct them, DBNs learn representations through stacked RBMs, and GANs pit sample-generating networks against discriminators [108],, [23].Table 1 summarizes deep-learning techniques used for gait recognition; the same work may appear under multiple architectures.Source: §2.2.4 · p. 8, §2.2.5 · p. 9, §2.2.6 · p. 9

2.3 Gait Recognition

  • Video-based gait recognition can passively identify people without requiring their cooperation, while footprint methods require costly acquisition equipment.Gait approaches are presented as alternatives to biometric systems requiring active information provision or specialized equipment.Source: §2.3 · p. 10
  • Template-based methods capture movement dynamics, while nontemplate methods emphasize body shape and its attributes.Examples of template features include Walking Path Image, Gait Information Image, and Gait Energy Image,, and.Source: §2.3 · p. 11

3 GAIT RECOGNITION THROUGH DEEP LEARNING-BASED APPROACHES

This section surveys deep-learning approaches to gait recognition, their results, and the datasets and representations they use. It also compares architecture and representation trends while noting challenges in separating gait data’s underlying factors.

3.1 Convolutional Neural Networks

  • CNN-based gait studies report strong results across varied setups, including 91.5% accuracy on OU-ISIR Large Population and 91.9% accuracy on OU-MVLP.DeepGait combines deep convolutional features with Joint Bayesian and reports state-of-the-art results over OU-ISIR Large Population; other work reports context-sensitive performance, including 97.5% accuracy on TUM-GAID and 58.20% on CASIA B.Source: §3.1 · pp. 12–13

3.2 Capsule Networks

  • Capsule-based approaches combine gait images at different network stages; one study reports 74.4% identification accuracy on OU-ISIR.Other surveyed capsule models use multi-scale partial representations or multi-sensor data, with reported benefits under challenging conditions or across datasets.Source: §3.2 · pp. 13–14

3.3 Recurrent Neural Networks

  • RNN studies use temporal information for cross-view recognition and gait diagnosis, with reported robustness across CASIA B and OU-ISIR baselines.Wearable-sensor work studies simulated abnormalities in ten healthy individuals; an IMU-based approach reports state-of-the-art verification and identification on whuGAIT and OU-ISIR.Source: §3.3 · p. 14

3.4 Autoencoders

  • Autoencoders address incomplete gait cycles by reconstructing missing images, while other work separates pose and appearance to handle clothing occlusions.ITCNet reports 86% Rank-1 and 95% Rank-5 on OU-ISIR Large Population using 20 images per instance; another approach introduces the FVG dataset and is reported robust for walking toward the camera.Source: §3.4 · pp. 14–15

3.5 Deep Belief Networks

  • Deep Belief Networks are surveyed for wearable-sensor gait assessment in Parkinson’s and movement-disorder diagnosis, and for encoding sEMG patterns to support gait tracking.The surveyed studies report DBN-based approaches as suitable for gait assessment and as having high potential for tracking.Source: §3.5 · p. 15

3.6 Generative Adversarial Networks

  • GAN approaches include view-invariant gait features, view-specific features using Period Energy Images, and synthetic walking videos for evaluating attacks.The surveyed studies report competitive or state-of-the-art comparisons on several datasets; the synthetic-video study evaluates attack detection and visual fidelity.Source: §3.6 · p. 15

3.7 Discussion

  • About 50% of reviewed solutions use 2D-CNNs, while silhouettes appear in more than 85% of representation-based approaches.Hybrid methods make up 26% of solutions; CNN-RNN combinations account for about 15%.Source: §3.7 · p. 17
  • Gait data’s interacting factors can hinder recognition, and most deep-learning methods have not explicitly separated its underlying structure into disjoint variables.Factors include occlusion, viewpoint, appearance, sequence order, body-part movement, and lighting.Source: §3.7 · p. 17

4 DATASETS

Datasets support model training and evaluation, but gait-specific datasets are scarce, hampering development. Creating them requires many recordings per subject, substantial storage, and participant consent.

4 DATASETS

  • Gait-specific datasets are scarce, and this shortage hampers development of models that recognize people by how they walk or move.Training and evaluation depend on datasets containing subjects relevant to the task.Source: §4 · pp. 18–19
  • Creating gait datasets requires many movement recordings per subject, high storage capacity for the resulting videos, and each participant’s permission for extraction and public distribution.Creating datasets without formal consent may bring lawsuits.Source: §4 · p. 19

4.1 CMU MoBo Dataset

  • CMU MoBo contains gait videos from 20 people and provides silhouettes and bounding boxes to alleviate segmentation.It is available without reservations or agreement forms through a university FTP server.Source: §4.1 · p. 19

4.2 TUM GAID Dataset

TUM GAID combines RGB, audio, and depth recordings and includes clothing and time variation. HID-UMD aggregates walking sequences from two datasets, with multiple views and foreground masks.

4.2 TUM GAID Dataset

  • TUM GAID combines RGB images, audio, and depth for 305 people, with 3,370 records overall and 32 people re-recorded for variation.The authors describe it as the only dataset allowing recognition by combining video, depth, and audio.Source: §4.2 · p. 19
  • TUM GAID includes clothing, carrying-condition, and time variations, with a well-defined evaluation protocol.The first recording was in winter and the second in warmer weather; 32 people were recorded in both periods.Source: §4.2 · p. 19, Figure 7 · p. 20

4.3 HID-UMD Dataset

  • HID-UMD provides walking videos from four angles and corresponding binary foreground masks, combining two datasets for gait and facial biometrics research.One dataset has 25 individuals in four poses; the other has 55 individuals walking along a T-shaped path recorded by two orthogonal cameras.Source: §4.3 · pp. 19–20

4.4 CASIA

CASIA comprises four gait datasets spanning multiple viewpoints, clothing and walking conditions, nighttime infrared imagery, and foot-pressure data.

4.4 CASIA

  • CASIA A contains 20 individuals recorded in 12 videos each across parallel, 45-degree, and 90-degree views.The sequences vary in duration according to walking velocity.Source: §4.4 · p. 20, Figure 8 · p. 21
  • CASIA B records 124 individuals from 11 angles, with three repetitions and variations in clothing and walking speed.Silhouettes are provided for foreground segmentation.Source: §4.4 · p. 20, Figure 9 · p. 21, Figure 10 · p. 21
  • CASIA C contains nighttime infrared recordings of 153 subjects walking normally, slowly, quickly, or normally with a backpack.The images were captured at night.Source: §4.4 · p. 20, Figure 11 · p. 22
  • CASIA D pairs 3,496 gait-pose images with 2,658 cumulative foot-pressure images from 88 people in an indoor environment.Its acquisition process synchronizes image capture with footsteps data.Source: §4.4 · pp. 20–21, Figure 12 · p. 22

4.5 OU-ISIR Biometric Database

OU-ISIR aggregates eight dataset groups covering large populations, age, carried objects, inertial sensing, similar actions, and treadmill-based gait variation.

4.5 OU-ISIR Biometric Database

  • The OU-ISIR project, begun in 2007, aggregates eight dataset groups.The survey describes it as the world’s largest gait-recognition dataset project.Source: §4.5 · p. 22
  • The treadmill collection has four subsets covering speed, clothing, view, and gait-fluctuation variation.The subsets include 34 subjects for speed variation, 68 for clothing variation, 168 for view variation, and 185 for gait fluctuation.Source: §4.5 · p. 22
  • The Large Population Dataset has 4,016 subjects filmed twice from four angles; Speed Transition Dataset B records 25 people at speeds from 1 to 5 km/h.Speed Transition Dataset A contains 179 scenes at a constant 4 km/h.Source: §4.5 · p. 23
  • The Multi-view Large Population Dataset contains 10,307 samples across 14 camera angles, with participants aged 2–87 years.It was developed for motion recognition methods with cross-vision.Source: §4.5 · p. 23
  • The Large Population Dataset with Bag targets object-carrying gait and marks four possible carried-object regions, while providing binary masks.The dataset comprises 62,528 people aged 2–95 years.Source: §4.5 · p. 23
  • The Large Population Dataset with Age was created to investigate gait recognition by age and gender, with 62,846 individuals aged 2–90 years.Its videos include binary masks after background removal.Source: §4.5 · p. 23
  • The Inertial Sensor Dataset supports identification from motion sensors and accelerometers, with data from 744 subjects aged 2–78 years.The survey characterizes it as the largest inertial sensor-based gait database.Source: §4.5 · p. 23
  • The Similar Actions Inertial Dataset includes 460 participants and six floor characteristics, including flat, stairs, and ramps.Its walking properties were obtained together with data presented in the Inertial Sensor Dataset.Source: §4.5 · p. 23

4.6 University of South Florida Dataset

The USF dataset contains gait sequences from 122 subjects, with variation in footwear, carried items, surface conditions, camera view, and recording time.

4.6 University of South Florida Dataset

  • The USF dataset comprises 1,870 sequences from 122 subjects wearing two shoe types.It also includes people carrying or not carrying a briefcase.Source: §4.6 · p. 23
  • The dataset varies whether subjects carry a briefcase, alongside its shoe-type, surface, view, and recording-time conditions.These conditions provide multiple forms of variation within the dataset.Source: §4.6 · pp. 23–24
  • Recordings vary across grass and concrete surfaces and left or right camera views.Videos were captured outdoors at two different time instants.Source: §4.6 · p. 24

4.7 Southampton Dataset

The Southampton database combines three gait datasets spanning indoor and outdoor walking, multiple views, varied environments, and barefoot temporal recordings.

4.7 Southampton Dataset

  • SOTON Small contains 12 subjects walking an indoor track at varying speeds, in different shoes and clothes, and with or without bags.Source: §4.7 · p. 24
  • SOTON Large includes 114 subjects recorded outdoors, on an indoor laboratory track, and on a treadmill, across six camera angles and more than 5,000 sequences.Source: §4.7 · p. 24
  • SOTON Temporal uses 12 synchronized cameras in dynamic environments and includes 25 subjects aged 20–55, all filmed barefoot.The environments vary in background, lighting, walking surface, and camera position.Source: §4.7 · p. 24

4.8 AVA Multi-View Dataset for Gait Recognition (AVAMVG)

AVAMVG provides calibrated multiview gait videos and silhouettes for 3D-based recognition; KY4D provides sequential 3D models and image sequences across straight and curved paths.

4.8 AVA Multi-View Dataset for Gait Recognition (AVAMVG)

  • AVAMVG contains 200 six-channel multiview videos from 20 actors, also usable as 1,200 single-view videos.Sequences depict different trajectories, were captured with task-calibrated cameras, reconstructed in 3D, and include corresponding binary silhouettes.Source: §4.8 · p. 24

4.9 Kyushu University 4D Gait Database

  • KY4D contains sequential 3D models and image sequences of 42 subjects walking four straight and two curved trajectories, recorded by 16 cameras at 1032 × 776 pixels.Source: §4.9 · p. 24
  • KY4D divides its data into straight-path Dataset A, curved-path Database B with 1.5 m or 3.0 m radii, and an infrared shadow set of 54 subjects walking straight.The infrared set contains time-series shadow images captured with oblique infrared lights and a ceiling-mounted camera.Source: §4.9 · p. 24

4.10 WhuGAIT Datasets

whuGAIT provides wearable inertial gait data from 118 people in six task-specific subsets, including segmentation, windowed samples, and authentication arrangements.

4.10 WhuGAIT Datasets

  • whuGAIT [140], released in 2018 with source code and pretrained models, contains 3D accelerometer and 3-axis gyroscope data from 118 people.Twenty participants were recorded over three days, while 98 were recorded on one day.Source: §4.10 · pp. 25–26
  • The six whuGAIT subsets include two-step segmentation datasets and 2.56-second windowed samples, with training and testing counts specified for each.Datasets 1–4 distinguish the full cohort from the 20 people recorded over three days.Source: §4.10 · p. 26
  • whuGAIT Dataset 5 supports authentication using paired samples from the same or different subjects, and Dataset 6 uses the same structure with vertical rather than horizontal alignment.Dataset 5 uses data from 98 people for training and 20 for validation.Source: §4.10 · pp. 26–27

4.11 Datasets in Usage Contexts

The survey organizes gait-recognition datasets by usage context, acquisition environment, and spectrum. It identifies twelve covariates, four environment scenarios, and two spectrum classes.

4.11 Datasets in Usage Contexts

  • Table 3 organizes datasets by twelve covariates, including viewpoint, pace, clothing, surface, and foot pressure.The listed covariates also include objects, shoes, time, silhouette, gait fluctuation, treadmill walking, and overground walking.Source: §4.11 · p. 27, Table 3 · p. 27
  • Figure 18 groups dataset construction environments into static indoor, static outdoor, active indoor, and active outdoor scenarios.The figure represents the datasets’ development environments.Source: §4.11 · p. 28, Figure 18 · p. 28
  • Figure 19 categorizes dataset spectrum representations as color-based or thermal information.The figure provides an illustrated schema of these two classes.Source: §4.11 · p. 28, Figure 19 · p. 28

5 CONCLUSIONS AND FUTURE DIRECTIONS

The survey reviews deep-learning gait recognition, its biometric context, and nine commonly used datasets, including their extraction processes and limitations. It highlights controlled video datasets as a boundary for real-world deployment and identifies several future research directions.

Survey contribution

  • The paper provides historical background on biometric identification and gait recognition, describing the field’s main concerns and challenges.Its review focuses on significant recent gait-recognition work using deep-learning techniques.Source: §5 · p. 28

Dataset review

  • The survey describes nine commonly employed datasets and their extraction processes and limitations, noting efforts toward recognition independent of gender, angle, clothing, shoes, and carried bags.The datasets include CMU MoBo, TUM GAID, HID-UMD, CASIA, OU-ISIR, USF, SOTON, AVAMVG, KY4D, and WhuGait [140].Source: §5 · p. 28

Survey contribution

  • The review covers gait identification for monitored-resource access and highlights that gait systems do not require individuals’ willingness to be identified.It also compares gait biometrics with other biometric techniques and surveys recent methodologies and state-of-the-art architectures.Source: §5 · pp. 28–29

Limitations

  • The reviewed video datasets contain only one person per video and were recorded in fully controlled environments, limiting confidence in real-world operation such as public-street identification.The survey notes limited background variation and treadmill recordings, and says the reviewed models are probably prone to lack and misbehavior in real-world operation.Source: §5 · p. 29

Future directions

  • Future directions include Transformer networks [107], gender and age recognition, hazardous-environment monitoring, and scenes containing multiple people.The survey notes that gait features may suit hazardous environments with limited illumination, where extracting more descriptive biometric characteristics is difficult.Source: §5 · p. 29
  • The survey identifies growing interest in clothing and carried-object variation and hybrid gait systems combining gait with biometrics such as face and ear.It notes that real-life problems often require robustness to uncontrolled environments with multiple people, unlike controlled single-person scenes.Source: §5 · p. 29
Loading 2201.03323v1…