PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning-Based Gait Recognition Using Smartphones in the Wild

Qin Zou, Yanling Wang, Qian Wang, Yi Zhao, Qingquan Li

arXiv:1811.00338v3cs.LGeess.SPstat.ML

TL;DR

The paper addresses gait recognition under unconstrained conditions, where existing studies largely use limited roads and specified walking speeds. It combines convolutional and recurrent networks to represent inertial gait features and reports improved performance over standalone models.

  • Problem

    Gait recognition in unconstrained or free environments remains challenging, while CNN–LSTM combinations for inertial feature extraction require further investigation.

  • Method

    A hybrid architecture combines a DCNN and an LSTM to extract spatial and temporal representations from smartphone inertial gait time series.

  • Results

    The proposed hybrid network obtained significantly improved performance over standalone CNN and LSTM models.

  • Takeaways & Limitations

    Smartphone inertial data can support gait recognition in the wild using walking-session segmentation and hybrid deep feature extraction.

  • Takeaways & Limitations

    The sequential CNN+LSTM authentication network did not perform well for identification, where training samples were much fewer than for authentication.

Abstract

from arXiv · show

Compared to other biometrics, gait is difficult to conceal and has the advantage of being unobtrusive. Inertial sensors, such as accelerometers and gyroscopes, are often used to capture gait dynamics. These inertial sensors are commonly integrated into smartphones and are widely used by the average person, which makes gait data convenient and inexpensive to collect. In this paper, we study gait recognition using smartphones in the wild. In contrast to traditional methods, which often require a person to walk along a specified road and/or at a normal walking speed, the proposed method collects inertial gait data under unconstrained conditions without knowing when, where, and how the user walks. To obtain good person identification and authentication performance, deep-learning techniques are presented to learn and model the gait biometrics based on walking data. Specifically, a hybrid deep neural network is proposed for robust gait feature representation, where features in the space and time domains are successively abstracted by a convolutional neural network and a recurrent neural network. In the experiments, two datasets collected by smartphones for a total of 118 subjects are used for evaluations. The experiments show that the proposed method achieves higher than 93.5\% and 93.7\% accuracies in person identification and authentication, respectively.

I. INTRODUCTION

Gait is an unobtrusive biometric that is difficult to conceal, and smartphones offer convenient, inexpensive inertial-data collection. This paper addresses the limited robustness of existing smartphone gait-recognition methods under unconstrained walking conditions by combining deep spatial and temporal feature learning.

  • Gait is unobtrusive and more difficult to conceal than many other biometric characteristics.
  • Accelerometers and gyroscopes are commonly integrated into smartphones, making inertial gait data convenient and inexpensive to collect.
  • Most existing smartphone gait-recognition methods require walking along a specified road and/or at a normal speed, limiting wider application.
  • The proposed architecture seamlessly integrates a DCNN and an LSTM to represent gait features across spatial and temporal domains.One-dimensional convolution kernels preserve time-series structure before LSTM processing.
  • Two datasets are collected for evaluating smartphone-based gait recognition in unconstrained conditions, with one dataset containing 20 subjects and the other 118 subjects.Six sub-datasets are constructed for quantitative evaluation and performance comparison.

B. Deep Learning for Gait Recognition

Prior deep-learning gait studies established the value of CNNs and recurrent models but largely used constrained data and left inertial-signal integration underexplored. The paper therefore targets unconstrained gait recognition with a hybrid CNN–LSTM approach and a segmentation network for extracting walking sessions.

  • CNNs abstract spatial or array-signal features, whereas RNNs process temporal information in time series.
  • LSTM networks provide a recurrent approach for extracting temporal features from time series.
  • Earlier gait-recognition datasets were mostly collected under limited road conditions and specified walking speeds.
  • A one-dimensional DCNN models walking-versus-nonwalking partitioning as time-series semantic segmentation and fuses hierarchical convolutional features.

C. Gait Cycle Segmentation

The method extracts walking data from continuous smartphone recordings and segments it into individual steps. It uses triaxial acceleration to reduce orientation sensitivity and identifies step boundaries from local maxima under empirical timing and amplitude rules.

  • Cycle-partitioned gait data can improve performance over non-cycle-partitioned data, motivating step-based segmentation.
  • The method divides extracted continuous gait data into separate steps after removing nonwalking sessions.
  • ACCo combines triaxial acceleration values to reduce the influence of smartphone orientation during gait-cycle segmentation.The three acceleration components are used to form the orientation-robust curve.
  • Step-separation points are selected at local maxima of ACCo, followed by automatic extraction using subject-specific thresholds and manual checking.
  • The step-separation rules require ACCo above 10m/s2 and consecutive maxima separated by 0.8s to 1.6s.

1) Problem formulation:

The formulation maps an inertial gait sequence to subject-membership probabilities, while the recognition network combines convolutional and recurrent feature extraction before classification.

  • 1) Problem formulation:: An inertial gait curve x with sampling length T is treated as the input for recognizing a subject’s identity.
  • 1) Problem formulation:: The candidate population is represented by n subjects, s=(s1, s2, ..., sn), and the output assigns each subject a probability P(si|x).
  • 1) Problem formulation:: Recognition selects the subject associated with the maximum output probability.
  • 2) Network structure:: CNN and LSTM feature extractors produce featcnn and featlstm, whose concatenation is passed to a fully connected classifier.
  • 2) Network structure:: An RNN generates hidden states across time and layers from the input sequence using recurrent, inter-layer, and bias parameters.
  • 2) Network structure:: The LSTM extends the recurrent unit with a cell, input gate, output gate, and forget gate for temporal information interaction.
  • 2) Network structure:: The CNN uses one-dimensional kernels and four convolutional layers with two max-pooling layers to abstract and downsample gait time-series features.

3) Loss Function:

The network produces subject probabilities, and cross-entropy training penalizes disagreement between predicted and target outputs to support convergence.

  • For each input sample, the network outputs o=(o1, o2, ..., on), where each oi=P(si|x) lies between 0 and 1.
  • The cross-entropy loss increases as the predicted probability differs more from the target output.
  • This loss property helps the network converge during training.

B. Neural Network for Authentication

Authentication is formulated as binary classification over pairs of gait sequences, using a CNN feature extractor with an LSTM-based authentication network under non-overlapping training and test subjects.

  • Authentication takes two gait sequences, xa and xb, as input and predicts whether they come from the same subject or different subjects.
  • The authentication output has two classes, labeled True for same-subject pairs and False for different-subject pairs.
  • A CNN trained on 98 classification subjects is frozen as a feature extractor, while an LSTM processes its output for authentication on 20 non-overlapping test subjects.
  • The experiments evaluate identification and authentication using smartphone-collected inertial gait data in unconstrained daily-life conditions.

1) Dataset #1:

The study constructs smartphone gait datasets from 20- and 118-subject collections using two-step or fixed-time segmentation, interpolation, and authentication pair formation.

  • 1) Dataset #1:: The 118-subject collection is segmented into continuous two-step samples and interpolated to a fixed length of 128.
  • 1) Dataset #1:: The 20-subject collection contains 49,275 two-step samples, with 44,339 for training and 4,936 for testing.
  • 1) Dataset #1:: A fixed-time 118-subject dataset uses 2.56-second windows sampled at 50Hz, with 1.28-second overlap and 29,274 total samples.
  • 1) Dataset #1:: The corresponding 20-subject fixed-time dataset has 39,314 samples without overlap, including 35,373 training samples and 3,941 test samples.
  • 1) Dataset #1:: The authentication dataset contains 74,142 paired samples from 118 subjects, split across 98 training subjects and 20 test subjects with balanced positive and negative classes.
  • 1) Dataset #1:: The app records accelerometer and gyroscope data at 50Hz while allowing the smartphone to be placed in a pocket, hand, or on a desk.

2) Identification experiment:

The identification experiments compare LSTM-, CNN-, and hybrid network structures across smartphone gait datasets and data settings. They evaluate classification performance using accuracy and include both same-subject and cross-subject dataset configurations.

  • The experiments compare LSTM-based, CNN-based, and CNN+LSTM-based networks for gait classification.
  • The evaluation uses multiple smartphone gait datasets, including datasets with 10 and 118 subjects and manually annotated walking/non-walking labels.Dataset #7 contains 577 samples from 10 subjects, while Dataset #8 contains 1,354 samples from 118 subjects.
  • Figure 7 compares single-layer, bidirectional, and double-layer LSTM networks across the evaluated configurations.
  • 90.22% accuracy is achieved on Dataset #7, whose training and test samples come from the same 10 subjects without sample overlap.The result measures separation of walking from non-walking data.
  • 85.57% accuracy is achieved on Dataset #8, where training and test samples come from different subjects.The reported result indicates generalization to subjects not represented in training.

1) Different LSTM networks:

The study evaluates three LSTM architectures across eight combinations of accelerometer and gyroscope inputs, using interpolated and time-fixed data. The best configuration uses double-layer LSTM with six-axis interpolated input.

  • The evaluated architectures are SL-LSTM, Bi-LSTM, and DL-LSTM, representing one hidden layer, bidirectional processing, and two hidden layers.
  • The experiments test eight channel combinations spanning six-axis, accelerometer-only, gyroscope-only, and mean-square-root inputs under interpolated and time-fixed sampling.
  • Accelerometer-based results exceed gyroscope-based results, which exceed mean-square-root results, indicating stronger gait-feature capture by the accelerometer.
  • Six-axis input produces the best results for both interpolated and time-fixed data, supporting complementary use of accelerometer and gyroscope data.
  • 91.88% classification accuracy is obtained by DL-LSTM with interp 6 data, which is selected for later experiments.

E. User-Identification Performance

User-identification experiments compare traditional signal-processing methods with several deep-learning architectures. Accuracy is the evaluation metric, with Fourier serving as the strongest traditional baseline described here.

  • Accuracy is used to evaluate person-identification methods because the task is formulated as classification.
  • The comparison includes Fourier, Wavelet, EigenGait, LSTM, CNN, CNN+LSTM, CNNfix+LSTM, CNN+LSTMfix, IdNet, and DeepConvLSTM.
  • CNN+LSTM is trained jointly from scratch, whereas CNNfix+LSTM and CNN+LSTMfix freeze one component and train the remaining components.
  • IdNet is a CNN-based smartphone inertial-data method trained from scratch on data from 50 subjects.
  • DeepConvLSTM combines convolutional and LSTM recurrent layers to learn feature representations and temporal dependencies.

2) Performance of traditional methods:

Traditional methods are evaluated against deep-learning methods for smartphone gait identification, while the experiments also examine feature separability and training strategies. Deep-learning methods generally achieve higher reported accuracy, and freezing one network improves performance over single-network methods.

  • 2) Performance of traditional methods:: 81.55% and 93.64% accuracy are achieved by Fourier on Dataset #1 and Dataset #2, respectively, making it the best traditional method on both datasets.The best results use interpolated six-axis inertial inputs.
  • 3) Performance of deep learning methods:: On the IdNet dataset, all seven deep-learning methods exceed 99.2% accuracy, with little difference because the walking style is relatively standard.
  • 3) Performance of deep learning methods:: CNN features are more discriminative than LSTM features in the cross-dataset t-SNE visualization, while CNN and CNN+LSTM feature distributions are similar.
  • 3) Performance of deep learning methods:: CNN outperforms LSTM on Dataset #1 and Dataset #2, while jointly trained CNN+LSTM is not guaranteed to outperform either single-network method.The paper suggests that the more complex hybrid network may suffer from overfitting when trained from scratch.
  • 3) Performance of deep learning methods:: Freezing one network and training the other improves performance over single-network methods on Dataset #1 and Dataset #2.CNN+LSTMfix outperforms CNNfix+LSTM by approximately 0.6% on Dataset #1 and 0.3% on Dataset #2.

2) Metric:

Authentication performance depends strongly on data alignment and model design. Vertically aligned inertial data generally benefits deep-learning methods, while the hybrid network outperforms stand-alone networks and exceeds 96% accuracy on OU-ISIR.

  • Metric:: Accuracy and ROC curves are used to evaluate and compare authentication methods.ROC curves plot true positive rate against false positive rate across varying thresholds.
  • Alignment: Vertical alignment improves deep-learning authentication accuracy by approximately 9% for CNN and LSTM, and 8% for CNN+LSTM and CNNfix+LSTM.The compared methods include CNN, LSTM, CNN+LSTM, and CNNfix+LSTM.
  • Model comparison: CNN+LSTM and CNNfix+LSTM substantially outperform stand-alone CNN or LSTM, demonstrating the effectiveness of fusion.CNNfix+LSTM also exceeds CNN+LSTM, possibly because the latter is harder to train from scratch.
  • Traditional methods: 92.70% accuracy is achieved by Fourier on horizontally aligned data, versus 61.86% on vertically aligned data.Traditional methods perform much worse than deep-learning methods on vertically aligned data, although Fourier performs strongly on horizontal alignment.
  • OU-ISIR evaluation: Above 96% authentication accuracy is achieved by all three deep-learning methods on the OU-ISIR dataset.CNN+LSTM improves over the stand-alone CNN and LSTM models.

VI. DISCUSSION

The discussion explains how the hybrid network represents inertial gait signals across spatial and temporal dimensions, while identifying limitations in identification transfer and participant diversity. The conclusions emphasize complementary sensors, effective CNN-LSTM feature extraction, and improved hybrid performance.

  • VI. DISCUSSION: The proposed method integrates CNN and RNN processing for inertial signals rather than image sequences.The CNN extracts two-dimensional features from gait time series, while the LSTM further processes retained temporal features.
  • VI. DISCUSSION: Six-axis accelerometer and gyroscope data are arranged so one-dimensional convolutions capture temporal structure and cross-axis relationships.The design assumes signals across accelerometer axes and between accelerometer and gyroscope measurements are related.
  • VI. DISCUSSION: The authentication network is not used for identification because it performs poorly on identification and may overfit with fewer training samples.Identification has about 20,000 training samples versus about 60,000 for authentication.
  • VI. DISCUSSION: The experiments are limited to datasets collected from 118 healthy persons, constraining evaluation on people with physical problems.The authors identify this population as a limitation of performance evaluation.
  • VI. DISCUSSION: Smartphone-based gait recognition is evaluated under unconstrained conditions without knowing when, where, or how users walk.A fully convolutional network partitions inertial data into walking and nonwalking sessions.
  • VII. CONCLUSION: The conclusions report better accelerometer performance than gyroscope performance, complementary sensor benefits, and significantly improved hybrid-network results over stand-alone models.They also report benefits from step-separated data for LSTMs and vertically aligned data for authentication.
Loading 1811.00338v3…