Source-linked AI summary
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Shibo Zhang, Yaxuan Li, Shen Zhang, Farzad Shahabi, Stephen Xia, Yu Deng, Nabil Alshurafa
TL;DR
Wearable HAR has advanced through low-power sensing and deep learning, but the field still lacks a consolidated account of its methods, challenges, and directions. This paper systematically reviews applications, datasets, sensors, and deep-learning approaches, finding broad performance advances while identifying data, privacy, deployment, and review-scope limitations.
Problem
Despite promising deep-learning results, wearable HAR still presents unresolved challenges, including data-acquisition and privacy problems that motivate further research.
Method
The paper conducts a systematic review of wearable HAR, categorizing applications, sensors, datasets, and deep-learning approaches while comparing their properties and selection criteria.
Results
Deep learning has pushed wearable HAR performance to an all-time high, while reviewed studies report accuracy improvements ranging from 1% to 32% for DBNs over traditional machine-learning methods.
Takeaways & Limitations
The review provides guidelines and insights for choosing suitable deep-learning approaches and highlights future opportunities for practical wearable HAR deployment.
Takeaways & Limitations
The review may omit papers and is subject to selection bias, while heterogeneity in devices, participants, and implementations can produce heterogeneous study results.
Abstract
from arXiv · showhide
Mobile and wearable devices have enabled numerous applications, including activity tracking, wellness monitoring, and human--computer interaction, that measure and improve our daily lives. Many of these applications are made possible by leveraging the rich collection of low-power sensors found in many mobile and wearable devices to perform human activity recognition (HAR). Recently, deep learning has greatly pushed the boundaries of HAR on mobile and wearable devices. This paper systematically categorizes and summarizes existing work that introduces deep learning methods for wearables-based HAR and provides a comprehensive analysis of the current advancements, developing trends, and major challenges. We also present cutting-edge frontiers and future directions for deep learning-based HAR.
1 Introduction
Wearables combine low-power sensing with deep learning to support HAR across diverse applications. This review surveys the field, categorizes major methods, compares their properties, and identifies ongoing challenges and future directions.
- Motivation: Wearables sense movement and physiological signals for applications spanning healthcare, entertainment, gaming, industry, and lifestyle.Examples of sensed signals include heart rate, temperature, blood pressure, and electrodermal activity.
- Motivation: Deep learning improves HAR performance and robustness by learning features directly from raw sensor data.The passage contrasts this with traditional methods that generally require manually engineered features.
- Contributions: The review also synthesizes current challenges, obstacles, trends, and opportunities for deep-learning-based wearable HAR.Its stated goal is to shed light on possible future directions for students and researchers.
- Scope: The review focuses on physical-activity recognition covering locomotion, activities of daily living, exercise, and factory work.It intentionally excludes other deep-learning applications such as ambient scene analysis, emotion recognition, and subject identification.
- Contributions: The paper surveys HAR applications, sensors, public datasets, and deep-learning methods while comparing algorithm accuracy, advantages, disadvantages, deployment, and model-selection criteria.The review categorizes relevant papers by algorithms including autoencoders, CNNs, and RNNs.
2 Methodology
The paper frames its review around three research questions and uses a systematic literature-selection process. It organizes HAR by sensing, application, deep-learning approach, and challenge dimensions.
- Research Questions: The review asks which HAR applications, sensors, and public datasets are used; which deep-learning approaches and trade-offs they present; and which challenges and opportunities remain.The authors present the review as an answer to these three questions.
- Review Process: The literature search used PRISMA procedures and keyword combinations covering HAR, wearables, deep learning, and major algorithm families.Searches combined two keyword groups using an AND rule and retrieved up to 200 results per combination.
- Review Process: The authors excluded image- or video-based HAR, environmental-sensor systems, and papers with minor algorithmic advances.These criteria were intended to focus the review on novel technical progress in wearable sensor-based HAR.
- Review Process: The review process narrowed 8400 initial Google Scholar results to 176 included papers.The selection is presented through a consort diagram.
- Limitations: The authors acknowledge possible selection bias, missed pioneering papers, and heterogeneity from devices, participant demographics, and implementation details.These factors can produce heterogeneity among study results within the reviewed literature.
- Taxonomy: The taxonomy organizes HAR literature across sensor, application, deep-learning approach, and challenge dimensions.The sensor dimension distinguishes physical and physiological sensors, while the approach dimension includes six major algorithm families.
3 Related Work
Earlier reviews established taxonomies and analyses of sensor-based HAR, but this paper responds to newer deep-learning developments. It emphasizes recent architectures and wearable on-body data for a broad introductory synthesis.
- Field Developments: Rapid advances in computing resources and deep-learning techniques have reshaped applied sensor-based HAR.The paper characterizes this period as one of substantial advancement and high performance.
- This Review: This review offers a gentle introduction while analyzing recent CNN, RNN, GAN, deep-reinforcement-learning, and hybrid architectures with their HAR trade-offs.It also distills the authors’ experience into a different view of challenges and opportunities.
- This Review: Compared with earlier work, the paper gives greater attention to cutting-edge methods applied to on-body sensory data, including GAN and DRL.The comparison is made specifically with a review centered on CNN, LSTM, and hybrid methods.
4 Human Activity Recognition Overview
Wearable HAR supports activity monitoring across lifestyle, healthcare, and human–computer interaction contexts. The overview connects recorded activity information and physiological monitoring with practical assessment and control applications.
- Applications: Commercial wearables span applications from fitness and lifestyle to medical and human–computer interaction.The cited database summarizes 582 commercial wearables registered since 2015.
- Physical Activity: Wearable activity data capture duration and intensity, supporting interpretation of physical activities such as sitting, walking, climbing stairs, jogging, and running.Physical activity is linked in the passage to chronic-disease risk and mental health.
- Physical Activity: Energy-expenditure estimation is an important motivation for personal activity tracking and can support self-reflection and behavior regulation.The passage connects these behaviors with interventions targeting obesity, diabetes, and cardiovascular disease.
- Healthcare: HAR contributes to healthcare and rehabilitation by helping capture information and monitor patient health, including Parkinson disease symptoms.Wearables can track, store, and share patient data with medical institutions.
- Human–Computer Interaction: In HCI, wrist-worn IMUs can detect wrist shaking to control smart devices without navigating screens and pressing buttons.The example uses shaking to skip a song.
4.2 Wearable Sensors
Wearable HAR relies on diverse low-power sensors distributed across devices and the body, with inertial sensing as a central modality and specialized sensors supporting physiological and movement recognition.
- Wearable platforms: Wearable HAR systems use devices ranging from smartphones and smartwatches to smart glasses, rings, gloves, armbands, necklaces, shoes, and E-tattoos.MEMS technology enables these devices to become smaller and lighter, reducing the burden of wearing them.
- Inertial sensors: An IMU combines an accelerometer, gyroscope, and sometimes magnetometer to capture linear motion, rotation, and magnetic fields.The accelerometer measures acceleration along x, y, and z axes, while the gyroscope measures roll, yaw, and pitch rates.
- Physiological sensors: ECG and PPG provide common wearable modalities for monitoring heart rate, with ECG measuring cardiac electrical activity and PPG using infrared light.ECG uses body-attached electrodes, whereas PPG relies on an infrared light sensor.
- Muscle sensors: EMG measures electrical activity from muscle movement and contractions through either surface electrodes or intramuscular sensors.Surface EMG places electrode arrays on the skin to measure muscle-generated electrical signals.
- Muscle sensors: MMG measures low-frequency muscle contractions and vibrations with a microphone or accelerometer rather than electrodes.Four-channel thigh MMG signals have been used to detect knee motion patterns for powered lower-limb prostheses.
- Other modalities: Additional HAR sensors include piezoelectric, barometric pressure, EEG, respiration, ultraviolet, GPS, and microphone sensors.These modalities measure quantities including pressure, temperature, brain activity, breathing, sun exposure, location, and sound.
4.3 Major Datasets
Wearable HAR research uses a broad set of mostly public datasets, primarily collected with IMU, GPS, or ECG sensors and covering physical, daily, gesture, breathing, and industrial activities.
- Dataset landscape: Major wearable-HAR datasets are ranked in Table 1 according to their annual Google Scholar citation counts.Most datasets are publicly available and support machine-learning and deep-learning evaluation.
- Dataset modalities: Most listed datasets use IMU, GPS, or ECG data to recognize physical or daily activities.The collection also includes datasets for hand gestures, breathing patterns, and car assembly-line activities.
- Public archives: The UCR-TSC archive aggregates datasets from various sensing modalities and grew from 16 datasets at release to 128 by October 2018.It had expanded to 85 datasets by 2015.
- Representative datasets: PAMAP2 contains recordings from 9 subjects performing 18 activities with 3 IMUs sampled at 100 Hz and a heart-rate monitor sampled at 9 Hz.Activities range from jumping to house cleaning.
- Representative datasets: Skoda and WISDM are also commonly used to train and evaluate HAR algorithms.Figure 4 presents inertial-sensor placements for 9 common datasets, including WISDOM, ActRecTut, UCI-HAR, SHO, PAMAP2, and Opportunity.
5 Deep Learning Approaches
Deep learning has improved wearable HAR by learning task-relevant representations from increasingly available data, while the review organizes major approaches and explains autoencoders as feature-learning tools.
- Field progress: Deep-learning approaches have outperformed traditional machine-learning approaches across a wide range of HAR tasks.The paper attributes this progress to increasing data availability, hardware acceleration, and algorithmic advancements.
- Autoencoders: Autoencoders learn features by encoding input x into a code z and decoding z into a reconstruction x′.The encoder and decoder use learnable weights and biases with nonlinear activation functions such as Sigmoid, tanh, or ReLU.
- Autoencoders: Autoencoders are commonly applied in wearable HAR for feature extraction, dimensionality reduction, denoising, and information retrieval.They may be used individually or in stacked architectures and are typically trained with mean squared error or mean squared error plus KL-divergence losses.
Encoder Decoder
Deep learning methods for wearable HAR span representation learning, denoising, privacy preservation, temporal modeling, reinforcement learning, and synthetic-data generation. Across these approaches, the reviewed studies report improved recognition, robustness, personalization, or privacy outcomes under specific datasets and settings.
- Autoencoder applications: Autoencoders learn latent representations that support feature extraction, domain adaptation, unseen-data detection, and null-class recognition in wearable HAR.They can also be combined with clustering for unsupervised recognition, achieving 96% weighted accuracy in one study.
- Autoencoder applications: DSTSAE denoises loose-clothing sensor signals, improving SNR from −12 dB to +18.2 dB and increasing gesture and locomotion F1-scores by 14.4% and 55.3%.The method outperformed traditional Kalman Filters in the reported experiments.
- Privacy preservation: Replacement and anonymizing autoencoders preserve activity-recognition utility while reducing detection of sensitive activities or user attributes.One framework reduced gender-classification accuracy from more than 90% on raw data to 50%, corresponding to random guessing.
- Deep belief networks: DBN performance varied with architecture and input representation: deeper models and spectrogram-based triaxial acceleration inputs outperformed shallower alternatives in reported studies.A tuned DBN reached 98.23% on WISDM, 91.5% on Daphnet, and 89.38% on Skoda; an RBM-based fusion architecture reached 81% on Skoda.
- Other deep learning paradigms: Other reviewed architectures combine convolutional feature extraction with recurrent temporal reasoning, while reinforcement learning and GANs support decision policies and semi-supervised synthetic-data generation.A CNN-LSTM study achieved 98.1% on SHL transportation-mode classification, and DRL studies reported 98.33% arm-movement accuracy and 98.02% locomotion-mode recognition accuracy.
6 Challenges and Opportunities
The review identifies four major challenge areas in wearable HAR: acquiring data and labels, developing models, and deploying models. It frames these areas as research questions concerning current methods and potential solutions.
- Data acquisition: Data acquisition poses challenges that require investigation of how data can be collected and what solutions can address those difficulties.
- Label acquisition: Label acquisition remains a distinct challenge, including questions about current methods for obtaining activity labels.
- Modeling: Modeling challenges concern how HAR models are developed and what potential solutions can improve modeling.
- Model deployment: Model deployment presents challenges and opportunities for applying HAR models in practical settings.
6.1 Challenges in Data Acquisition
Data acquisition in wearable HAR is constrained by the cost of collecting high-quality, representative data and by privacy concerns surrounding sensor inference.
- 6.1.1 The Need for More Data: Collecting raw data across many wearables is labor-intensive, making low-cost acquisition of massive, high-quality datasets a central need.There is no well-defined method for determining HAR training-set size; error-rate convergence as a function of dataset size is one proposed approach.
- 6.1.1 The Need for More Data: Synthetic augmentation, including noise injection, signal translation, Dynamic Time Warping, and GAN-based synthesis, can improve classification accuracy or address dataset imbalance.Conditional GANs have been used to generate irregularly sampled time series for unbalanced datasets.
- 6.1.2 Data Quality: Training quality depends strongly on data quality, which can be degraded by electromagnetic interference and sampling-schedule uncertainty.Data-cleaning and imputation algorithms are proposed alongside hardware improvements to replace or fill poor-quality data.
- 6.1.3 Privacy Protection: Privacy-preserving HAR must reduce sensitive individual-identifying information while retaining features common to activities or movements.Replacement autoencoders, anonymizing autoencoders, and federated learning are among the approaches discussed.
6.2 Challenges in Label Acquisition
Label acquisition for wearable HAR is difficult because sensor time series are hard to interpret directly and often require additional synchronized sensing, while annotation reduction methods seek to use unlabeled data more efficiently.
- 6.2.1 The Need for More Labels: Labeling activities from HAR sensor time series is difficult and often requires video or audio sensing to establish ground truth.Accurate synchronization between the auxiliary sensing source and wearable data is also necessary.
- 6.2.1 The Need for More Labels: Semi-supervised, weakly supervised, active-learning, and augmentation methods reduce the amount of manually annotated data required.Semi-supervised learning combines labeled and unlabeled data to learn more generalizable feature representations.
- 6.2.1 The Need for More Labels: Active learning selects unlabeled samples with low prediction confidence for human annotation while leveraging deep-learning classification.The approach establishes labels selectively and on the fly rather than annotating all available data.
- 6.2.2 In-field Experiments: Moving HAR evaluation from laboratories to in-field settings complicates ground-truth collection because fixed surveillance cameras may not cover subjects’ daily movements.Wearable cameras have been used to record activities outside laboratory settings.
6.3 Challenges in Modeling
Modeling challenges in wearable HAR include choosing suitable activity segments, recognizing semantically complex activities, generalizing beyond training data, and maintaining robustness across sensing conditions.
- 6.3.1 Data Segmentation: Static sliding windows may be too large for short activities or too small for long movements, motivating more adaptive segmentation methods.Reinforcement learning has been used to find more optimal activity segments and boost HAR performance.
- 6.3.2 Semantically Complex Activity Recognition: Complex activities such as eating remain difficult because they comprise varied movements, unlike simpler activities such as running.Proposed solutions decompose complex gestures into atomic gestures or hierarchically construct high-level activities from low-level ones.
- 6.3.3 Model Generalizability: Generalizability means performing well on unseen data, whereas overfitting means performing well on training data but poorly on new data.Generalizing to larger populations often requires substantial data and higher model complexity.
- 6.3.3 Model Generalizability: Model-agnostic approaches are presented as a way to improve HAR model generalizability.The stated aim is to improve generalizability without depending on a particular model family.
- 6.3.4 Model Robustness: Robustness and reliability are increasingly important, with multisensory systems combining signals from multiple sensor types.DeepFusionHAR combines handcrafted and deep-learning-extracted features to detect daily-life and sports activities.
6.4 Challenges in Model Deployment
Deploying deep-learning HAR on mobile and wearable platforms requires reducing inference cost, exploiting specialized hardware, and balancing flexibility with power efficiency across devices.
- 6.4.1 Mobile Platforms: Mobile HAR deployments use CPUs, DSPs, mobile GPUs, and lightweight CNN- or RNN-based systems to improve inference time, power consumption, or tolerance to noisy readings.These approaches target practical inference on resource-limited mobile platforms.
- 6.4.2 Reducing Model Complexity: Neural-network complexity can be reduced through lower-dimensional layer representations or binary weights instead of fixed-point or floating-point weights.Both strategies reduce computation on resource-limited mobile platforms.
- 6.4.3 Specialized Hardware: ASICs and low-power microcontrollers can execute the HAR pipeline with lower power consumption than mobile or GPU-based platforms, but ASICs are not reconfigurable for other tasks.Specialized hardware therefore trades energy efficiency against flexibility.
- 6.4.4 Distributed Deployment: The review identifies computation partitioning across cloud, mobile, and edge devices as an important opportunity for improving practical HAR deployment.Many existing works rely on a single platform, leaving cross-device partitioning insufficiently explored.
- 6.4.4 Distributed Deployment: Mobile DNN inference can scale across different available compute resources, supporting further study of optimal inference partitioning strategies.The exploration included phone CPU and DSP configurations and offloading computation to remote devices.
7 Conclusions
The paper concludes by synthesizing wearable HAR applications, sensors, datasets, deep learning advances, model-selection guidance, roadblocks, and future research directions.
- The review surveys real-life applications, mainstream sensors, and popular public datasets for wearable human activity recognition.
- It reviews deep learning advances in wearable HAR and compares approaches to provide guidance for selecting suitable methods.
- It organizes current roadblocks into data-wise, label-wise, and model-wise challenges, identifying potential opportunities for each.
- The review identifies open challenges and suggests future research avenues for deep learning-based wearable sensor HAR.