Source-linked AI summary
Deep Learning for Sensor-based Activity Recognition: A Survey
Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, Lisha Hu
TL;DR
Sensor-based HAR must infer high-level activities from low-level sensor readings despite conventional methods’ reliance on hand-crafted features and limitations for unsupervised and incremental learning. The paper surveys deep-learning approaches across sensor modality, deep model, and application, synthesizes their behavior across scenarios, and identifies challenges and feasible solutions for future research.
Problem
Conventional HAR relies on hand-crafted features and has undermined performance for unsupervised and incremental learning tasks.
Method
The paper surveys deep-learning HAR literature across sensor modality, deep model, and application, with detailed synthesis and proposed future challenges and solutions.
Results
Model performance depends on the scenario: CNN performs best in Skoda and UCI Smartphone protocols, while DBN and RNN outperform others in two OPPORTUNITY protocols.
Takeaways & Limitations
No single deep model is best in all situations, so model choice should follow the recognition scenario; hybrid models tend to outperform single models.
Takeaways & Limitations
Deep HAR remains far from online and mobile deployment because models are often trained offline on remote servers, limiting real-time and incremental-learning support.
Abstract
from arXiv · showhide
Sensor-based activity recognition seeks the profound high-level knowledge about human activities from multitudes of low-level sensor readings. Conventional pattern recognition approaches have made tremendous progress in the past years. However, those methods often heavily rely on heuristic hand-crafted feature extraction, which could hinder their generalization performance. Additionally, existing methods are undermined for unsupervised and incremental learning tasks. Recently, the recent advancement of deep learning makes it possible to perform automatic high-level feature extraction thus achieves promising performance in many areas. Since then, deep learning based methods have been widely adopted for the sensor-based activity recognition tasks. This paper surveys the recent advance of deep learning based sensor-based activity recognition. We summarize existing literature from three aspects: sensor modality, deep model, and application. We also present detailed insights on existing work and propose grand challenges for future research.
1. Introduction
Sensor-based HAR extracts high-level activity knowledge from sensor readings, but conventional methods depend on hand-crafted features and are limited for unsupervised and incremental learning. This survey reviews deep-learning HAR across sensor modality, deep model, and application, and identifies future challenges.
- Conventional pattern-recognition methods rely heavily on heuristic, hand-crafted feature extraction constrained by human domain knowledge.
- Deep learning automatically learns higher-level features through end-to-end neural networks, reducing manual feature-design effort.
- Deep-network structures are more feasible for unsupervised and incremental learning than conventional approaches.
- No specific prior survey had focused on the intersection of deep learning and HAR, motivating this survey.
- The survey organizes recent deep-learning HAR work by sensor modality, deep model, and application, then discusses benchmarks and grand challenges.
2. Background
Sensor-based HAR learns a model that maps sequences of sensor readings to activity sequences by minimizing prediction loss. Deep learning addresses conventional methods’ reliance on shallow hand-crafted features and labeled data by learning representations automatically and exploiting unlabeled data.
- 2.1. Sensor-based Activity Recognition: HAR models map sensor-reading sequences to predicted activity sequences, with training minimizing discrepancy from ground-truth activities.
- 2.1. Sensor-based Activity Recognition: Conventional HAR extracts manually designed statistics such as mean, variance, DC, and amplitude before training a recognition model.
- 2.2. Why Deep Learning?: Hand-crafted features depend on human experience or domain knowledge, increasing development time and reducing generalization across environments and tasks.
- 2.2. Why Deep Learning?: Shallow statistical features support low-level activities but are difficult to use for high-level or context-aware activities such as having coffee.
- 2.2. Why Deep Learning?: Conventional models often require extensive labeled data, whereas deep generative networks can exploit unlabeled samples for training.
- 2.2. Why Deep Learning?: Deep-learning pipelines jointly perform feature extraction and model building, automatically learning high-level representations for complex activity recognition.
3. Sensor Modality
Sensor modalities for HAR include body-worn, object, and ambient sensors, each capturing different aspects of human activity and context. Combining modalities can support more complex recognition, but object and ambient sensors face deployment and robustness constraints.
- 3. Sensor Modality: HAR sensor modalities are classified as body-worn, object, and ambient sensors.
- 3.1. Body-worn Sensor: Body-worn accelerometers, gyroscopes, and magnetometers capture movement and are widely used for daily-living and sports activities.
- 3.2. Object Sensor: Object sensors detect movement of specific objects and can provide fine-grained information for complex activity recognition.
- 3.2. Object Sensor: Object sensors are used less often than body-worn sensors because deployment is difficult, while combinations with other modalities are emerging for higher-level activities.
- 3.3. Ambient Sensor: Ambient sensors capture human–environment interaction through radar, sound, pressure, and temperature measurements.
- 3.3. Ambient Sensor: Ambient-sensor deployment is difficult, environmental effects can reduce robustness, and only certain activities can be inferred reliably.
- 3.4. Multiple Sensors: Combining sensor types, such as acceleration and acoustic information, can improve HAR accuracy and capture object movement alongside environmental state.
4. Deep Model
The survey reviews DNNs, CNNs, autoencoders, RBMs, RNNs, and hybrid models for HAR, emphasizing their feature-learning capabilities and design trade-offs. CNNs and hybrid architectures exploit signal structure, while unsupervised models support feature learning with model-selection limitations.
- Deep Neural Network: DNNs learn representations through multiple hidden layers; a 5-hidden-layer DNN performed automatic feature learning and classification with improved performance.Deeper networks can help with multidimensional HAR and complex activities, although situation-specific details remain important for fine-tuning.
- Convolutional Neural Network: CNNs exploit local dependency and scale invariance when classifying temporal, multidimensional HAR signals.Sensor time series require input adaptation into virtual-image forms, using model-driven or data-driven approaches.
- Convolutional Neural Network: Partial weight-sharing can improve CNN performance while accommodating signals that behave differently across units.Weight sharing is also described as an efficient way to speed training on a new task.
- Autoencoder: Autoencoders learn latent representations through encoding-decoding, and stacked autoencoders use learned features with labels to form classifiers.SAE-based HAR commonly uses greedy layer-wise pre-training followed by fine-tuning.
- Autoencoder: SAEs enable unsupervised feature learning, while their performance depends strongly on layer and activation-function choices.Adding sparse constraints was reported to improve HAR performance, but finding optimal configurations can be difficult.
- Restricted Boltzmann Machine: RBMs use visible and hidden layers, while DBNs stack RBMs and commonly feed into fully connected layers.Multimodal RBMs construct an RBM for each sensor modality before unifying their outputs.
- Recurrent and Hybrid Models: RNNs model temporal correlations, but relatively few HAR studies used them because learning speed and resource consumption were concerns.Hybrid CNN-RNN models combine CNN spatial feature extraction with RNN temporal modeling, and CNN plus recurrent dense layers outperformed CNN plus dense layers.
5. Applications
Deep learning HAR applications span daily living, sports, lifestyle, health, disease, and higher-level activity understanding. The survey also reviews benchmark datasets and identifies a need to incorporate richer behavioral and contextual information.
- Most surveyed applications recognize activities of daily living and sports, which body-worn sensors can readily capture.
- Lifestyle applications extend HAR to sleep and respiration, often requiring object and ambient sensors such as WiFi and sound.
- Health-oriented HAR has addressed Parkinson’s disease, trauma resuscitation, and paroxysmal atrial fibrillation using sensors corresponding to changed bodily movements or functions.
- Disease-related recognition requires attention to the association between disease and activity and to appropriate sensor placement, such as inertial sensors on shoes for gait freezing.
- Most high-level activity studies focus on body movements, while emotion and user state remain insufficiently represented.A smart-home study combined activity and environment signals but constrained recognition to body movements.
- The survey distinguishes self-collected data, which is tedious to process, from public HAR datasets commonly adopted by researchers.It presents several widely used public datasets in Table 3.
6. Summary and Discussion
The survey links successful HAR to suitable sensors, preprocessing, and scenario-dependent model selection. Its review indicates that no single deep model dominates all tasks, while hybrid models often perform better than single models.
- Sensor deployment and preprocessing: Accelerometers are the most common body-worn modality because they are widely available in mobile devices and recognize many daily activities.
- Sensor deployment and preprocessing: Semantic activities beyond simple body movements are better addressed by combining object and ambient sensors, for which open datasets remain scarce.
- Sensor deployment and preprocessing: Sensor placement should support common activities while object and ambient sensors should collect data naturally and non-invasively.
- Sensor deployment and preprocessing: Sliding windows divide raw signals into model inputs, while separate sensor modalities or axes can serve as channels to enhance representation.
- Model selection: RNN and LSTM suit short, naturally ordered activities, whereas CNN better handles long-term repetitive activities and multimodal signals.
- Model selection: Multi-layer RBM or SAE is preferred for more accurate recognition after pre-training and fine-tuning.
- Model selection: No model outperforms all others across situations; CNN leads on Skoda and UCI Smartphone, while DBN and RNN lead on two OPPORTUNITY protocols.
- Model selection: Hybrid models tend to outperform single models, including DeepConvLSTM on OPPORTUNITY 1 and Skoda.
7. Grand Challenges
The survey identifies deployment, labeling, sensing, feature-learning, and model-efficiency challenges for deep HAR. Proposed directions include mobile execution, unsupervised learning, richer sensing, and collaboration between complementary models.
- Online and mobile deep activity recognition: Online and mobile HAR remains limited because models are often trained offline on remote servers and only deployed to devices.This arrangement is neither real-time nor friendly to incremental learning.
- Online and mobile deep activity recognition: Reducing mobile-server communication costs and increasing device computing ability are proposed responses to deployment constraints.
- More accurate unsupervised activity recognition: Unsupervised HAR is urgent because deep-learning performance still relies heavily on expensive and time-consuming labeled samples.
- More accurate unsupervised activity recognition: Crowdsourcing and privacy-concerned labeling can help annotate otherwise unlabeled activities.
- Sensing and feature learning: Hybrid sensors provide elaborate information for fine-grained activity recognition, while contextual signals such as Wi-Fi, Bluetooth, and GPS can characterize situations.
- Sensing and feature learning: Combining human-crafted and deep features may improve performance and support more robust feature learning.
- Sensing and feature learning: Collaboration between deep and shallow models could combine accurate learning with lightweight execution, although parameter sharing remains unresolved.
- Opportunistic activity sensing: Opportunistic sensing requires deep-model backpropagation to be designed for noncontinuous activity signals.
8. Conclusion
The survey reviews deep-learning approaches for sensor-based HAR and organizes recent progress across sensor modality, deep model, and application. It concludes that deep learning reduces reliance on handcrafted features while automatically learning high-level sensor representations.
- Deep learning reduces dependence on human-crafted feature extraction and automatically learns high-level representations of sensor data for HAR.
- The survey organizes recent deep-learning HAR progress around sensor modality, deep model, and application.