PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities

Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, Yunhao Liu

arXiv:2001.07416v2cs.HCcs.LG

TL;DR

Sensor-based human activity recognition is useful but faces practical challenges including computational constraints, privacy risks, noisy data, and distribution discrepancy. This survey organizes deep learning methods by those challenges, reviews modalities and datasets, and concludes that current solutions remain incomplete while providing foundations for future research.

  • Problem

    Sensor-based activity recognition faces practical challenges including computational cost, privacy risks, noisy sensory data, interpretability, and distribution discrepancy across users, time, and sensors.

  • Method

    The paper surveys deep learning methods for sensor-based activity recognition, organizing them by challenges while covering sensor modalities, public datasets, and challenge-specific techniques.

  • Results

    The survey summarizes and analyzes current challenge-related methods, including approaches for multimodal extraction, semi-supervised learning, distribution discrepancy, efficient inference, and privacy.

  • Takeaways & Limitations

    Current research provides concrete foundations and guidance for addressing activity-recognition challenges, while the survey offers a rough guideline for research-topic selection and approach development.

  • Takeaways & Limitations

    Current research still lacks comprehensive and reliable solutions for the identified challenges.

Abstract

from arXiv · show

The vast proliferation of sensor devices and Internet of Things enables the applications of sensor-based activity recognition. However, there exist substantial challenges that could influence the performance of the recognition system in practical scenarios. Recently, as deep learning has demonstrated its effectiveness in many areas, plenty of deep methods have been investigated to address the challenges in activity recognition. In this study, we present a survey of the state-of-the-art deep learning methods for sensor-based human activity recognition. We first introduce the multi-modality of the sensory data and provide information for public datasets that can be used for evaluation in different challenge tasks. We then propose a new taxonomy to structure the deep methods by challenges. Challenges and challenge-related deep methods are summarized and analyzed to form an overview of the current research progress. At the end of this work, we discuss the open issues and provide some insights for future directions.

1 INTRODUCTION

Sensor-based human activity recognition is increasingly important because pervasive sensors support daily-life monitoring while avoiding some privacy concerns of cameras. The survey organizes recognition challenges and reviews deep learning methods developed to address them.

  • Motivation: Sensor-based systems use on-body or ambient sensors to monitor motion and activity, offering privacy advantages and broad deployment across smart devices and everyday objects.Sensors can be embedded in phones, watches, cars, walls, and furniture.
  • Challenges: Recognition faces challenges including indistinguishable activity features, scarce annotations, class imbalance, and variation across users, time, and sensors.Activities such as walking and running may share similar characteristics, while unexpected activities are difficult to collect and label.
  • Challenges: Practical systems must address computational cost, privacy risks, and interpretability of noisy sensory data.The system should fit portable devices, respond instantly, and identify which data supports or harms recognition.
  • Prior Methods: Earlier methods relied on carefully engineered heuristic features without universal approaches for extracting distinguishable activity representations.Common techniques included time-frequency transformations, statistical approaches, and symbolic representations.
  • Survey Scope: The survey focuses on challenge-specific deep learning models and techniques, extending beyond surveys organized mainly by neural-network structure.Examples include deep transfer learning and multimodal fusion.
  • Contributions: The paper proposes a challenge-method-analysis taxonomy that compares solutions, summarizes public datasets, and discusses open issues and future directions.Its stated goal is to provide a rough guideline for selecting research topics or developing approaches.

2 SENSOR MODALITY AND DATASETS

Sensor-based activity recognition draws on wearable, ambient, object, and other sensor modalities, each capturing different aspects of human behavior and context. The survey also catalogs public datasets by acquisition context, subjects, activities, sensor types, and challenge tasks.

  • Sensor Modality: The survey classifies sensor modalities into wearable, ambient, object, and other strategies because sensor modality crucially affects recognition performance.This taxonomy distinguishes sensors by what they capture and where they operate.
  • Wearable Sensors: Wearable sensors directly capture body movements and are commonly integrated into smartphones, watches, bands, and clothing.Accelerometers and gyroscopes provide multivariate motion-related time series from body-mounted locations.
  • Ambient Sensors: Ambient sensors capture interactions between people and their surroundings, support multi-occupant activity detection, and can enable indoor localization.WiFi-based recognition uses movement- and location-related interference with signal propagation.
  • Object Sensors: Object-sensor information is important for recognizing composite activities that involve continuous interaction with surrounding objects.Examples include drinking or eating, cooking, and playing.
  • Other Modalities: Other modalities, such as audio sensors, use signals modified by human movement to reflect motion information.Mobile-device speakers can transmit ultrasound while microphones receive the modified signals.
  • Datasets: Public datasets are summarized by acquisition context, subject and activity counts, sensor types, and potential challenge tasks.The survey defines daily living as common activities performed by subjects under instructions.

3 CHALLENGES AND TECHNIQUES

Sensor-based activity recognition faces challenges in extracting temporal and multimodal features from continuous, heterogeneous signals. Deep models address these challenges through temporal networks, multimodal fusion strategies, and hybrid architectures.

  • Temporal Feature Extraction: Temporal feature extraction is crucial because activities are continuous time-series signals containing smooth movements and abrupt transitions.Sensor sampling frequencies commonly range from tens to hundreds of Hz.
  • Temporal Feature Extraction: Deep models learn temporal features end to end using recurrent neural networks, temporal CNNs, and their variants.The paper highlights mutual promotion between feature learning and recognition during end-to-end training.
  • Temporal Feature Extraction: LSTM-based RNNs use sliding windows to segment streaming signals, while bidirectional and ensemble variants capture broader temporal dynamics.Sliding-window length and movement step require careful tuning; ensembles of deep LSTMs outperformed individual networks on three benchmark datasets.
  • Multimodal Feature Extraction: Multimodal recognition must learn both inter-modality correlations and intra-modality information from sensors such as accelerometers, gyroscopes, and magnetometers.Combining diverse sensing modalities can outperform using one sensor alone.
  • Multimodal Feature Extraction: The survey extends feature-fusion categories beyond CNN-only architectures to all deep learning architectures.This extension is intended to reveal additional insights and considerations about multimodal fusion.
  • Multimodal Feature Extraction: Fusion strategies range from early data fusion and sensor-specific branches to axis-specific processing, shared filters, and classifier ensembles.These strategies differ in when modalities interact and how independently each sensing source is processed.

3.2 Annotation Scarcity

Annotation scarcity limits supervised activity-recognition methods because labeled data are expensive, time-consuming, and error-prone. Deep unsupervised and semi-supervised approaches exploit unlabeled or synthetic data, but their supervision requirements and labeling risks remain important boundaries.

  • Motivation: Supervised activity-recognition methods require large labeled datasets, whose annotation is expensive, time-consuming, tedious, and vulnerable to noise.Noise can arise from sensors, segmentation, and variation in activities across people.
  • Unsupervised Learning: Unsupervised learning explores patterns and extracts features from unlabeled sensory data, but effective methods for high-dimensional heterogeneous data remain needed.Deep belief networks and autoencoders are described as prominent generative models for this purpose.
  • Unsupervised Learning: Unsupervised deep generative models cannot independently identify true activity labels without labeled ground-truth samples.Consequently, the surveyed applications are treated as semi-supervised when labeled and unlabeled data are jointly used.
  • Semi-supervised Learning: Semi-supervised activity recognition reduces labeled-data requirements by combining smaller labeled sets with larger unlabeled sets.Approaches discussed include manifold learning, co-training, active learning, and data augmentation.
  • Semi-supervised Learning: Co-training uses multiple classifiers trained on different data views to select high-confidence unlabeled samples for iterative relabeling and retraining.Automatic labeling can introduce errors, whereas correct labels can improve accuracy.
  • Semi-supervised Learning: Active learning selects informative unlabeled instances for manual annotation, iteratively updating the classifier with the newly acquired true labels.Uncertainty and diversity are common query strategies; deep variants also address discarded samples and outlier selection.
  • Data Augmentation: Synthetic-data augmentation generates fake activity data from limited real data, commonly using GANs composed of a generator and discriminator.Activity-specific and semi-supervised GAN variants are described, with some generation processes depending on activity labels.

3.3 Class Imbalance

Class imbalance arises when activity classes have unequal sample counts, while scarce or difficult-to-collect activities make supervised training data insufficient. Deep-learning approaches address this through data resampling, augmentation, loss redesign, and semi-supervised labeling strategies.

  • Data-level challenges: Supervised activity recognition requires substantial labeled data, but some activities, such as elderly falls, are difficult to obtain.Raw data collected in unconstrained settings further complicates training.
  • Data-level methods: Resampling majority classes can reduce training data and omit critical samples with distinctive characteristics.The passage contrasts this risk with augmentation of minority classes.
  • Data-level methods: Augmenting minority classes preserves original samples while improving model robustness.The passage identifies augmentation as an alternative to majority-class resampling.
  • Algorithmic methods: Using the F1-score instead of cross-entropy gives classes with different training counts equal consideration through recall and precision.The method modifies model training rather than directly balancing the dataset.
  • Algorithmic methods: Semi-supervised labeling can itself create uneven class counts, motivating methods designed for imbalance in small labeled datasets.The imbalance may emerge as unlabeled samples are progressively assigned labels.

3.4 Distribution Discrepancy

Sensor-based activity recognition commonly faces distribution discrepancy because training and test data are not identically distributed. The survey organizes this discrepancy by users, time, and sensor deployment, and reviews transfer-learning and model-adaptation strategies for these settings.

  • Challenge overview: Training and test data often violate the i.i.d. assumption because sensory distributions differ across users, time, and sensor deployments.The survey also distinguishes homogeneous and heterogeneous discrepancy by the character of the change.
  • Transfer learning: Transfer learning transfers classification ability from a source domain with annotated data to a target domain with a different configuration.It is presented as particularly effective for distribution discrepancy problems.
  • Discrepancy across users: Different users produce different motion patterns, so data distributions vary even when they perform the same activity.User-specific training and testing can yield high accuracy but is impractical for deployment.
  • Discrepancy with time: Long-term streaming systems experience changing sensory distributions over months or years, creating time-related discrepancy.The initial model must accommodate future data rather than a fixed distribution.
  • Discrepancy with time: Concept drift is a distribution shift between source and target domains that may occur abruptly or gradually.The survey states that incremental training can continuously learn new activity concepts from incoming data.
  • Discrepancy with time: Concept evolution concerns new activities emerging after initial training, when application-phase annotation is restricted.The initial dataset cannot practically cover every activity, including rare events such as falls.
  • Open-set recognition: Open-set recognition identifies whether test samples belong to the target activity set, whereas concept-evolution methods must learn newly appearing activities.A GAN-based approach constructs a negative set for open-set classification.
  • Discrepancy across sensors: Sensor instances, types, positions, layouts, and environments can substantially alter collected signals and reduce recognition reliability.Hardware, software, signal propagation, and device-placement factors all contribute to deployment discrepancy.

3.5 Composite Activity

Most activity-recognition studies focus on simple, low-level activities, whereas composite activities combine actions and carry higher-level semantics. The survey describes unified and hierarchical recognition strategies, while noting that deep-learning research on composite activities remains limited.

  • Motivation: Existing research mainly recognizes repeated or single-posture activities such as walking, standing, and jogging.These activities have lower-level semantics than composite activities.
  • Motivation: Composite activities contain sequences of simple actions and include examples such as working, having dinner, and preparing food.Their higher-level semantics make recognition more challenging.
  • Challenges: Composite-activity recognition requires richer environments, diverse sensors, and carefully designed data-collection experiments.The survey characterizes this area as substantially less explored than simple-activity recognition.
  • Recognition strategies: One research stream builds unified models that recognize simple and composite activities together across locomotive, semantic, transitional, and postural categories.The cited activity set includes twenty-two simple and composite activities.
  • Recognition strategies: A hierarchical strategy treats composite activities as combinations of simple activities and can use multi-task learning for both levels.This approach is described as intuitive but remains underexplored with deep learning.
  • Recognition strategies: Conditional probabilistic models can infer simple-activity sequences and composite activities alternately, using each prediction to inform the other.The method extracts temporal correlations among simple actions for composite classification.

3.6 Data Segmentation

Fixed-size windows provide enough temporal context for sensor-based recognition but can contain activity transitions or insufficient information. The survey therefore reviews hierarchical segmentation and unsegmented multi-label prediction as alternatives.

  • Fixed-window segmentation: Fixed-size windows partition continuous sensor streams into model inputs so each segment can provide sufficient activity information.The ideal window contains one activity and receives one label.
  • Fixed-window segmentation: Larger windows provide richer information but increase transition risk, whereas smaller windows may not contain enough information.This creates a trade-off in empirical window-size selection.
  • Hierarchical segmentation: Hierarchical signal segmentation begins with a large window and progressively narrows it until a segment contains one activity.The method addresses the fixed-window trade-off through successive refinement.
  • Unsegmented recognition: Unsegmented activity recognition formulates recognition as set prediction and uses a multi-label architecture without explicit segmentation.The architecture predicts both the number of ongoing activities and each alternative activity's occurrence probability.

3.7 Concurrent Activity

Concurrent activity recognition treats simultaneous activities as a multi-label task because one sensor sample may correspond to multiple ground-truth labels. Deep methods either classify candidate activities independently or model activity combinations directly, with scalability limitations in both approaches.

  • Concurrent activities occur when a person performs multiple activities simultaneously, such as making a phone call while watching television.
  • Independent activity-specific classifiers use shared multimodal fusion features but incur substantially higher computational cost as the number of activities increases.
  • A multilayer LSTM can directly model the possibility of every activity combination, but its output dimension grows exponentially with the number of concurrent activities.

3.8 Multi-occupant Activity

Multi-occupant activity recognition addresses spaces where multiple people perform activities either independently in parallel or collaboratively. Wearable sensing can separate occupants more readily, whereas ambient sensing introduces a data-association challenge.

  • Multi-occupant activities are categorized as parallel activities performed independently and collaborative activities performed jointly.
  • With wearable sensors, parallel multi-occupant recognition can be decomposed into multiple single-occupant recognition tasks.
  • Ambient or object sensors require associating sensed signals with the occupants who generated them.
  • A two-occupant system combined wearable and ambient sensing with separate RBMs and downstream sequential, DBN, and MLP networks, achieving nearly 100% accuracy.
  • A multi-label RNN recognized each occupant separately, but used only ambient sensors and provided no specific solution to data association.

3.9 Computation Cost

Deep activity-recognition models can be too computationally demanding for real-time mobile deployment. Surveyed remedies combine handcrafted and learned features, simplify network structures, or quantize networks, while practical validation remains uneven.

  • AlexNet processes 61M parameters, uses 249MB of memory, and performs 1.5B high-precision operations per prediction.
  • Increasing neural-network depth and width improves performance but inevitably increases computational complexity, challenging real-time mobile recognition.
  • A spectrogram-based model with one CNN layer and two fully connected layers matched state-of-the-art accuracy across four benchmark datasets.
  • The same hybrid approach required milliseconds to tens of milliseconds per prediction on two smartphones and one on-node unit.
  • An SGRNN reduced complexity relative to LSTM and GRU in running time and model size, but still ran in hundreds of milliseconds without mobile real-world evaluation.
  • Network quantization constrains weights and activation outputs to discrete values such as -1 and +1 to reduce storage and computation requirements.

3.11 Interpretability

Interpretability is needed to identify which sensor modalities and time intervals drive activity-recognition decisions. Attention-based methods assign importance to input components, using either differentiable soft weighting or stochastic hard selection.

  • Sensor samples combine modalities and body positions, but only some contribute to particular activities and their importance can vary over time.
  • Interpretability methods seek to reveal the input factors behind decisions, such as the modality and time interval determining a walking prediction.
  • Interpretable deep learning focuses models on salient inputs and supports interpretation by visualizing extracted features.
  • Attention mechanisms weight input components so neural networks concentrate on subsets considered relevant to recognition.
  • Soft Attention: Soft attention assigns differentiable weights from 0 to 1, allowing gradients to propagate through the model.
  • Hard Attention: Hard attention assigns binary weights of 0 or 1 and uses stochastic selection because the selection policy lacks ground-truth labels.

4 FUTURE RESEARCH DIRECTION

Future work should address underexplored activity-recognition challenges and develop methods that reduce annotation requirements and recognize previously unseen activities. Although existing research provides foundations and guidance, comprehensive and reliable solutions remain unavailable.

  • Class imbalance, composite activities, and concurrent activities remain insufficiently explored challenges for future research.
  • Current research lacks comprehensive and reliable solutions for several activity-recognition challenges, despite providing foundations and guidance for future directions.
  • Independent unsupervised methods: Independent unsupervised methods could reduce the need for annotated samples, but existing deep unsupervised models mainly extract features rather than identify activities.
  • Independent unsupervised methods: Potential unsupervised approaches include deep unsupervised transfer learning and ontology-based, data-driven methods for inferring activity labels.
  • Identifying new activities: Recognizing novel activities requires models that learn new knowledge online without ground truth and produce features scalable to diverse activities.

5 CONCLUSION

The paper surveys deep learning for sensor-based human activity recognition by organizing methods around data modalities, datasets, challenges, and research directions. It aims to guide researchers in selecting topics and developing approaches.

  • The survey introduces sensory-data multimodality and public datasets for evaluating activity-recognition challenges.
  • It categorizes challenges by their reasons and analyzes how existing deep methods address them.
  • The paper discusses open issues and provides insights into future research directions for deep sensor-based activity recognition.
Loading 2001.07416v2…