PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning-Based Gait Recognition Using Smartphones in the Wild

Qin Zou, Yanling Wang, Qian Wang, Yi Zhao, Qingquan Li

arXiv:1811.00338v3cs.LGeess.SPstat.ML

TL;DR

The paper addresses gait recognition under unconstrained conditions, where existing studies largely use limited roads and specified walking speeds. It combines convolutional and recurrent networks to represent inertial gait features and reports improved performance over standalone models.

  • Problem

    Gait recognition in unconstrained or free environments remains challenging, while CNN–LSTM combinations for inertial feature extraction require further investigation.

    Source: B. Deep Learning for Gait Recognition · p. 3

  • Method

    A hybrid architecture combines a DCNN and an LSTM to extract spatial and temporal representations from smartphone inertial gait time series.

    Source: §I · p. 2, §VI · p. 13

  • Results

    The proposed hybrid network obtained significantly improved performance over standalone CNN and LSTM models.

    Source: §VII · p. 14

  • Takeaways & Limitations

    Smartphone inertial data can support gait recognition in the wild using walking-session segmentation and hybrid deep feature extraction.

    Source: §VII · p. 14

    The sequential CNN+LSTM authentication network did not perform well for identification, where training samples were much fewer than for authentication.

    Source: §VI · p. 13

Abstract

from arXiv · show

Compared to other biometrics, gait is difficult to conceal and has the advantage of being unobtrusive. Inertial sensors, such as accelerometers and gyroscopes, are often used to capture gait dynamics. These inertial sensors are commonly integrated into smartphones and are widely used by the average person, which makes gait data convenient and inexpensive to collect. In this paper, we study gait recognition using smartphones in the wild. In contrast to traditional methods, which often require a person to walk along a specified road and/or at a normal walking speed, the proposed method collects inertial gait data under unconstrained conditions without knowing when, where, and how the user walks. To obtain good person identification and authentication performance, deep-learning techniques are presented to learn and model the gait biometrics based on walking data. Specifically, a hybrid deep neural network is proposed for robust gait feature representation, where features in the space and time domains are successively abstracted by a convolutional neural network and a recurrent neural network. In the experiments, two datasets collected by smartphones for a total of 118 subjects are used for evaluations. The experiments show that the proposed method achieves higher than 93.5\% and 93.7\% accuracies in person identification and authentication, respectively.

I. INTRODUCTION

Gait is an unobtrusive biometric that is difficult to conceal, and smartphones offer convenient, inexpensive inertial-data collection. This paper addresses the limited robustness of existing smartphone gait-recognition methods under unconstrained walking conditions by combining deep spatial and temporal feature learning.

  • Gait is unobtrusive and more difficult to conceal than many other biometric characteristics.Source: §I · p. 1
  • Accelerometers and gyroscopes are commonly integrated into smartphones, making inertial gait data convenient and inexpensive to collect.Source: §I · p. 1
  • Most existing smartphone gait-recognition methods require walking along a specified road and/or at a normal speed, limiting wider application.Source: §I · p. 2
  • The proposed architecture seamlessly integrates a DCNN and an LSTM to represent gait features across spatial and temporal domains.One-dimensional convolution kernels preserve time-series structure before LSTM processing.Source: §I · p. 2
  • Two datasets are collected for evaluating smartphone-based gait recognition in unconstrained conditions, with one dataset containing 20 subjects and the other 118 subjects.Six sub-datasets are constructed for quantitative evaluation and performance comparison.Source: §I · p. 2

B. Deep Learning for Gait Recognition

Prior deep-learning gait studies established the value of CNNs and recurrent models but largely used constrained data and left inertial-signal integration underexplored. The paper therefore targets unconstrained gait recognition with a hybrid CNN–LSTM approach and a segmentation network for extracting walking sessions.

C. Gait Cycle Segmentation

The method extracts walking data from continuous smartphone recordings and segments it into individual steps. It uses triaxial acceleration to reduce orientation sensitivity and identifies step boundaries from local maxima under empirical timing and amplitude rules.

  • Cycle-partitioned gait data can improve performance over non-cycle-partitioned data, motivating step-based segmentation.Source: C. Gait Cycle Segmentation · p. 4
  • The method divides extracted continuous gait data into separate steps after removing nonwalking sessions.Source: C. Gait Cycle Segmentation · p. 4
  • ACCo combines triaxial acceleration values to reduce the influence of smartphone orientation during gait-cycle segmentation.The three acceleration components are used to form the orientation-robust curve.Source: C. Gait Cycle Segmentation · p. 4
  • Step-separation points are selected at local maxima of ACCo, followed by automatic extraction using subject-specific thresholds and manual checking.Source: C. Gait Cycle Segmentation · p. 4
  • The step-separation rules require ACCo above 10m/s2 and consecutive maxima separated by 0.8s to 1.6s.Source: C. Gait Cycle Segmentation · p. 4

1) Problem formulation:

The formulation maps an inertial gait sequence to subject-membership probabilities, while the recognition network combines convolutional and recurrent feature extraction before classification.

1) Problem formulation:

2) Network structure:

  • CNN and LSTM feature extractors produce featcnn and featlstm, whose concatenation is passed to a fully connected classifier.Source: 2) Network structure · pp. 5–6
  • An RNN generates hidden states across time and layers from the input sequence using recurrent, inter-layer, and bias parameters.Source: 2) Network structure · p. 5
  • The LSTM extends the recurrent unit with a cell, input gate, output gate, and forget gate for temporal information interaction.Source: 2) Network structure · p. 6
  • The CNN uses one-dimensional kernels and four convolutional layers with two max-pooling layers to abstract and downsample gait time-series features.Source: 2) Network structure · p. 6, Table II · p. 6

3) Loss Function:

The network produces subject probabilities, and cross-entropy training penalizes disagreement between predicted and target outputs to support convergence.

  • For each input sample, the network outputs o=(o1, o2, ..., on), where each oi=P(si|x) lies between 0 and 1.Source: 3) Loss Function · p. 6
  • The cross-entropy loss increases as the predicted probability differs more from the target output.Source: 3) Loss Function · p. 6
  • This loss property helps the network converge during training.Source: 3) Loss Function · p. 6

B. Neural Network for Authentication

Authentication is formulated as binary classification over pairs of gait sequences, using a CNN feature extractor with an LSTM-based authentication network under non-overlapping training and test subjects.

1) Dataset #1:

The study constructs smartphone gait datasets from 20- and 118-subject collections using two-step or fixed-time segmentation, interpolation, and authentication pair formation.

1) Dataset #1:

  • The 118-subject collection is segmented into continuous two-step samples and interpolated to a fixed length of 128.Source: 1) Dataset #1 · p. 7
  • The 20-subject collection contains 49,275 two-step samples, with 44,339 for training and 4,936 for testing.Source: 2) Dataset #2 · p. 7
  • A fixed-time 118-subject dataset uses 2.56-second windows sampled at 50Hz, with 1.28-second overlap and 29,274 total samples.Source: 3) Dataset #3 · p. 7
  • The corresponding 20-subject fixed-time dataset has 39,314 samples without overlap, including 35,373 training samples and 3,941 test samples.Source: 4) Dataset #4 · p. 7
  • The authentication dataset contains 74,142 paired samples from 118 subjects, split across 98 training subjects and 20 test subjects with balanced positive and negative classes.Source: 5) Dataset #5 · p. 7
  • The app records accelerometer and gyroscope data at 50Hz while allowing the smartphone to be placed in a pocket, hand, or on a desk.Source: 1) Data collection · p. 8

2) Identification experiment:

The identification experiments compare LSTM-, CNN-, and hybrid network structures across smartphone gait datasets and data settings. They evaluate classification performance using accuracy and include both same-subject and cross-subject dataset configurations.

  • The experiments compare LSTM-based, CNN-based, and CNN+LSTM-based networks for gait classification.Source: 2) Identification experiment · p. 8
  • The evaluation uses multiple smartphone gait datasets, including datasets with 10 and 118 subjects and manually annotated walking/non-walking labels.Dataset #7 contains 577 samples from 10 subjects, while Dataset #8 contains 1,354 samples from 118 subjects.Source: 1) Datasets · p. 8
  • Figure 7 compares single-layer, bidirectional, and double-layer LSTM networks across the evaluated configurations.Source: Figure 7 · p. 9
  • 90.22% accuracy is achieved on Dataset #7, whose training and test samples come from the same 10 subjects without sample overlap.The result measures separation of walking from non-walking data.Source: 2) Experimental results · p. 9
  • 85.57% accuracy is achieved on Dataset #8, where training and test samples come from different subjects.The reported result indicates generalization to subjects not represented in training.Source: 2) Experimental results · p. 9

1) Different LSTM networks:

The study evaluates three LSTM architectures across eight combinations of accelerometer and gyroscope inputs, using interpolated and time-fixed data. The best configuration uses double-layer LSTM with six-axis interpolated input.

E. User-Identification Performance

User-identification experiments compare traditional signal-processing methods with several deep-learning architectures. Accuracy is the evaluation metric, with Fourier serving as the strongest traditional baseline described here.

2) Performance of traditional methods:

Traditional methods are evaluated against deep-learning methods for smartphone gait identification, while the experiments also examine feature separability and training strategies. Deep-learning methods generally achieve higher reported accuracy, and freezing one network improves performance over single-network methods.

2) Performance of traditional methods:

  • 81.55% and 93.64% accuracy are achieved by Fourier on Dataset #1 and Dataset #2, respectively, making it the best traditional method on both datasets.The best results use interpolated six-axis inertial inputs.Source: 2) Performance of traditional methods · p. 10

3) Performance of deep learning methods:

  • On the IdNet dataset, all seven deep-learning methods exceed 99.2% accuracy, with little difference because the walking style is relatively standard.Source: 3) Performance of deep learning methods · p. 11
  • CNN features are more discriminative than LSTM features in the cross-dataset t-SNE visualization, while CNN and CNN+LSTM feature distributions are similar.Source: 3) Performance of deep learning methods · p. 11
  • CNN outperforms LSTM on Dataset #1 and Dataset #2, while jointly trained CNN+LSTM is not guaranteed to outperform either single-network method.The paper suggests that the more complex hybrid network may suffer from overfitting when trained from scratch.Source: 3) Performance of deep learning methods · p. 11
  • Freezing one network and training the other improves performance over single-network methods on Dataset #1 and Dataset #2.CNN+LSTMfix outperforms CNNfix+LSTM by approximately 0.6% on Dataset #1 and 0.3% on Dataset #2.Source: 3) Performance of deep learning methods · p. 11

2) Metric:

Authentication performance depends strongly on data alignment and model design. Vertically aligned inertial data generally benefits deep-learning methods, while the hybrid network outperforms stand-alone networks and exceeds 96% accuracy on OU-ISIR.

Metric:

  • Accuracy and ROC curves are used to evaluate and compare authentication methods.ROC curves plot true positive rate against false positive rate across varying thresholds.Source: 2) Metric · p. 12, Figure 9 · p. 13

Alignment

  • Vertical alignment improves deep-learning authentication accuracy by approximately 9% for CNN and LSTM, and 8% for CNN+LSTM and CNNfix+LSTM.The compared methods include CNN, LSTM, CNN+LSTM, and CNNfix+LSTM.Source: 3) Performance · p. 12

Model comparison

  • CNN+LSTM and CNNfix+LSTM substantially outperform stand-alone CNN or LSTM, demonstrating the effectiveness of fusion.CNNfix+LSTM also exceeds CNN+LSTM, possibly because the latter is harder to train from scratch.Source: 3) Performance · pp. 12–13

Traditional methods

  • 92.70% accuracy is achieved by Fourier on horizontally aligned data, versus 61.86% on vertically aligned data.Traditional methods perform much worse than deep-learning methods on vertically aligned data, although Fourier performs strongly on horizontal alignment.Source: 3) Performance · p. 12

OU-ISIR evaluation

  • Above 96% authentication accuracy is achieved by all three deep-learning methods on the OU-ISIR dataset.CNN+LSTM improves over the stand-alone CNN and LSTM models.Source: Table VIII · p. 12, 3) Performance · p. 13

VI. DISCUSSION

The discussion explains how the hybrid network represents inertial gait signals across spatial and temporal dimensions, while identifying limitations in identification transfer and participant diversity. The conclusions emphasize complementary sensors, effective CNN-LSTM feature extraction, and improved hybrid performance.

VI. DISCUSSION

  • The proposed method integrates CNN and RNN processing for inertial signals rather than image sequences.The CNN extracts two-dimensional features from gait time series, while the LSTM further processes retained temporal features.Source: §VI · p. 13
  • Six-axis accelerometer and gyroscope data are arranged so one-dimensional convolutions capture temporal structure and cross-axis relationships.The design assumes signals across accelerometer axes and between accelerometer and gyroscope measurements are related.Source: §VI · p. 13
  • The authentication network is not used for identification because it performs poorly on identification and may overfit with fewer training samples.Identification has about 20,000 training samples versus about 60,000 for authentication.Source: §VI · p. 13
  • The experiments are limited to datasets collected from 118 healthy persons, constraining evaluation on people with physical problems.The authors identify this population as a limitation of performance evaluation.Source: §VI · p. 13
  • Smartphone-based gait recognition is evaluated under unconstrained conditions without knowing when, where, or how users walk.A fully convolutional network partitions inertial data into walking and nonwalking sessions.Source: §VI · pp. 13–14, §VII · p. 14

VII. CONCLUSION

  • The conclusions report better accelerometer performance than gyroscope performance, complementary sensor benefits, and significantly improved hybrid-network results over stand-alone models.They also report benefits from step-separated data for LSTMs and vertically aligned data for authentication.Source: §VII · p. 14

Related papers

Loading 1811.00338v3…