A Survey of Human Activity Recognition in Smart Homes Based on IoT Sensors Algorithms: Taxonomies, Challenges, and Opportunities with Deep Learning

Damien Bouchabou; Sao Mai Nguyen; Christophe Lohr; Benoit Leduc; Ioannis Kanellos

Source-linked AI summary

A Survey of Human Activity Recognition in Smart Homes Based on IoT Sensors Algorithms: Taxonomies, Challenges, and Opportunities with Deep Learning

Damien Bouchabou, Sao Mai Nguyen, Christophe Lohr, Benoit Leduc, Ioannis Kanellos

arXiv:2111.04418v1Oct 18, 2021cs.HCcs.AIcs.LGeess.SP

TL;DR

Smart-home human activity recognition must support independent living despite complex, variable activities and annotation challenges. This survey develops a taxonomy, reviews algorithms and solutions, and identifies challenges, opportunities, and research directions, concluding that advances in datasets, computation, and deep learning can support more adaptive recognition.

Problem
Smart-home activity recognition must handle residents’ diverse lifestyles, abilities, and flexible daily activities, including complex activity sequences that current sequential algorithms cannot yet recognize reliably.
Method
The paper surveys smart-home human activity recognition using ambient sensors, proposes a taxonomy of algorithm components, and reviews current solutions, challenges, and opportunities.
Results
The survey finds that data-driven approaches can provide good recognition and become more adaptive, while deep learning can reduce their dependence on manually designed input features.
Takeaways & Limitations
Increasing datasets and computational power create opportunities to train more complex models and address outstanding smart-home recognition challenges.
Takeaways & Limitations
Annotation remains constrained by resident self-labeling errors, hypothesis-based post-processing, non-sequential activity flows, and inconsistent dataset taxonomies.

Abstract

from arXiv · show

Recent advances in Internet of Things (IoT) technologies and the reduction in the cost of sensors have encouraged the development of smart environments, such as smart homes. Smart homes can offer home assistance services to improve the quality of life, autonomy and health of their residents, especially for the elderly and dependent. To provide such services, a smart home must be able to understand the daily activities of its residents. Techniques for recognizing human activity in smart homes are advancing daily. But new challenges are emerging every day. In this paper, we present recent algorithms, works, challenges and taxonomy of the field of human activity recognition in a smart home through ambient sensors. Moreover, since activity recognition in smart homes is a young field, we raise specific problems, missing and needed contributions. But also propose directions, research opportunities and solutions to accelerate advances in this field.

1. Introduction

IoT-enabled smart homes use interconnected sensors and devices to support services such as health monitoring, assistance, energy management, and security. Human Activity Recognition (HAR) is needed to infer residents’ activities, but variability, privacy, and technical challenges remain.

Smart homes interconnect sensors, actuators, and household devices to provide health monitoring, daily assistance, energy management, and security services.
HAR monitors and analyzes residents’ behavior to deduce their activities, using either video-based or sensor-based systems.
Ambient sensor-based smart homes are considered less intrusive and more accepted than camera-based monitoring because camera systems raise privacy and data-sharing concerns.
Sensor-based solutions include wearable sensors, sensors on objects, and ambient sensors placed in the environment.
Smart-home HAR must address residents’ differing lifestyles, habits, abilities, and flexible activity performance through scalable and adaptive approaches.
The survey reviews recent algorithms, proposes a challenge-oriented taxonomy, summarizes deep-learning applications, and identifies open issues and future directions.

2. Pattern Classification

Smart-home HAR methods divide into knowledge-driven and data-driven approaches. Knowledge-driven systems formalize activities with expert rules, whereas data-driven systems learn from sensor data but require substantial, well-labeled training data.

HAR methods are broadly divided into Data-Driven Approaches (DDA), which learn from user-generated data, and Knowledge-Driven Approaches (KDA), which use expert knowledge and rules.
Knowledge-Driven Approaches (KDA): KDA models use schemas, rules, or networks to represent activities, temporal and spatial context, and relationships among entities.
Knowledge-Driven Approaches (KDA): KDA formalize activities semantically and logically but require extensive domain knowledge and are weak at handling uncertainty, adaptation, and changing settings.
Data-Driven Approaches (DDA): DDA methods include supervised and unsupervised probabilistic or statistical learning; supervised methods train on labeled data before classifying unknown data.
Data-Driven Approaches (DDA): DDA can handle noisy, uncertain, and incomplete sensor data without predefined domain knowledge, but supervised learning requires substantial clean, correctly labeled data.
KDA development is time-consuming and difficult to maintain, while DDA is more adaptive but needs more data and computation; deep learning can reduce dependence on manually designed features.

3. Features Extraction

Early HAR systems relied on handcrafted features extracted from segmented sensor windows. Later approaches incorporated temporal weighting, sensor relationships, prior-session information, and unsupervised probability features, with augmented handcrafted features sometimes matching deep-learning results.

Traditional HAR extracts carefully designed feature vectors from explicit activity sequences or segmented sensor windows before classification.
A baseline window feature vector can include first and last event times, window duration, and counts of sensor events, with size determined by the sensor count.For 34 sensors, the stated vector size is 34 + 3.
Temporal-dependency features weight sensor events by their distance from the last event, reducing the influence of temporally distant events.
Sensor-dependency features weight events according to mutual-information relationships with the segment’s final sensor event.
Past-session information can be incorporated through a two-part learning process when the previous activity label is unavailable to the classifier.
Unsupervised probability features learned with an HMM and Bayesian network produced results comparable to LSTM and CNN features, and significantly improved handcrafted-feature performance.

3.2. Automatic Features

Automatic feature extraction methods use deep models to learn representations from sensor data rather than relying entirely on heuristic handcrafted features. CNNs, autoencoders, and sequence-to-sequence models provide several routes for learning activity-discriminating representations.

Handcrafted features are heuristic, and no universal systematic extraction approach reliably captures distinctive human-activity characteristics.
Deep learning can automatically learn high-level features from raw signals through end-to-end training without human-designed feature guidance.
Convolutional Neural Networks (CNN): CNNs capture local dependencies, provide scale invariance to step differences or event frequency, and learn hierarchical representations for HAR.
Convolutional Neural Networks (CNN): A 1D CNN applied to raw data sequences achieved similar high results in the reported experiments.
Autoencoder Method: Autoencoders compress and reconstruct inputs while reducing dimensionality and ignoring noise, producing encoded representations that discriminate activities.
Autoencoder Method: A two-layer Stacked Denoising Autoencoder using binary or numerical sensor-window representations outperformed machine-learning algorithms on the Van Kasteren Dataset.

3.3. Semantics

Deep learning can automatically extract and classify features from smart-home sensor data, while semantic representations may support activity relationships and rapid adaptation. However, temporal-series analysis remains a challenge for pattern-recognition methods.

Autoencoders and CNNs can extract features and perform classification in end-to-end HAR models.
Semantic representations of activity relationships may improve recognition and enable techniques such as zero-shot learning for rapid adaptation.
Handcrafted features can perform well, but require extensive expert-led preprocessing that is time-consuming and reduces adaptability across environments.
Deep learning learns higher-quality features directly from raw data across datasets and supports supervised, unsupervised, and end-to-end processing.
Pattern-recognition methods developed for deep learning may be insufficient for sensor data represented as temporal series.

4. Temporal Data

Smart-home sensors generate event logs over time, but status-change reporting and differing trigger times produce sparse, irregularly sampled series. These temporal properties complicate activity recognition in real-world homes.

Smart-home sensors record residents’ environmental interactions as time-series event logs.
Because sensors usually report only status changes and have different triggering times, their recordings are scattered and irregularly sampled.
The segmentation literature addresses the temporal complexity of human activity data in real use cases.

4.1. Data Segmentation

Segmentation methods convert continuous sensor streams into activity-related windows, but they differ in timing assumptions, context representation, and suitability for real-time recognition. Window size and boundary detection remain central design issues.

4.1. Data Segmentation: Real-time segmentation is necessary for reactive services such as emergency fall detection.
4.1. Data Segmentation: Activity-window segmentation assigns sensor-event sequences to activity labels, but unlabelled data require detecting activity-change points.
4.1. Data Segmentation: Whole-segment prediction prevents the basic segmentation approach from operating in real time.
4.1.2. Time Windows (TW): Time windows use regular intervals, but window duration must balance missing relevant information against mixing multiple activities.
4.1.3. Sensor Event Windows (SEW): Sensor event windows contain equal numbers of events and use preceding events as context for labeling the final event.
4.1.4. Dynamic Windows (DW): Dynamic windows use offline rules to identify best-fit sensor groups before online classification assigns activity labels.
4.1.5. Feature Time Windows (FTW): Feature time windows encode short-, medium-, and long-term evolution for each binary sensor sequence using trapezoidal temporal segmentation.
4.1.6. Outlines: Table 1 summarizes and categorizes the segmentation techniques discussed in this section.

4.2. Time Series Classification

Smart-home HAR is a sequential pattern-recognition problem because ambient sensors produce irregular time series. Sequence models, especially LSTMs, capture temporal context and often outperform traditional approaches, although their gains are not uniformly significant.

4.2. Time Series Classification: Smart-home HAR requires methods for sequential data because sensor observations form irregularly sampled time series.
4.2. Time Series Classification: Markov models and RNNs model spatiotemporal or sequential information, with RNNs learning features from multidimensional time series.
4.2. Time Series Classification: RNN outputs depend on prior inputs and hidden context, but standard RNNs suffer from long-term dependency problems.
4.2. Time Series Classification: LSTM studies report higher classification performance than several traditional HAR and machine-learning approaches without handcrafted features.
4.2. Time Series Classification: LSTM performance is not significantly higher than some decision-tree, SVM, stochastic-gradient, and regression approaches.

4.3. Complex Human Activity Recognition

Complex human activities involve sequences of micro-actions whose order may vary, while activities can also overlap, interleave, or involve multiple residents. The survey highlights context-aware, hierarchical, and multi-label approaches as directions for recognizing these patterns.

4.3. Complex Human Activity Recognition: Sequential algorithms handle primitive activities but struggle with complex activities composed of multiple actions, such as cooking.The paper distinguishes simple activities from complex activities involving sequences of actions and interactions.
4.3.1. Sequences of sub-activities: Daily activities are compound sequences of micro-actions with recurring patterns but no strict constraints on composition or order.This compositionality motivates modeling activities beyond fixed gesture-like units.
4.3.1. Sequences of sub-activities: Context-aware algorithms use multi-timescale representations or attention to capture longer-term dependencies in sensor streams.These methods incorporate the context surrounding sensor activations rather than treating events independently.
4.3.1. Sequences of sub-activities: NLP-inspired embeddings could model sensor-event context and order across hierarchical activity levels, analogous to words, sentences, and paragraphs.The survey draws a direct parallel between sensor events and words, micro-activities and sentences, and compound activities and paragraphs.
4.3.2. Concurrent and interleaved activities: Activities may be concurrent or interleaved, motivating multi-label formulations and models such as ontology-based methods and CNN-LSTM structures.The cited approaches address activities that occur simultaneously or in overlapping schedules.
4.3.3. Multiple residents: Multiple residents create additional ambiguity because sensors may reflect shared activities or different activities performed simultaneously.The survey describes multi-resident recognition as a multi-label classification problem.
4.3.3. Multiple residents: A hybrid 1D CNN-LSTM structure performed best among the evaluated architectures for multi-resident activity recognition.The comparison included MLP, CNN, LSTM, GRU, and hybrid neural architectures.

4.4. Outlines

The survey compares recent smart-home activity-recognition methods and identifies unresolved challenges beyond single-resident recognition of simple activities. It points to resident-wise temporal segmentation as a needed direction.

4.4. Outlines: Table 2 summarizes and compares recent activity-recognition methods for smart homes.The table is presented as an overview of methods studied in the field.
4.4. Outlines: LSTM performs well for irregular time series involving one resident and simple activities, but concurrent, interleaved, idle, and multi-resident activities remain unsolved.The survey frames these cases as more difficult multi-class classification problems.
4.4. Outlines: Activity-recognition algorithms should segment sensor streams for each resident, with time-series adaptations of image-segmentation methods such as U-Net suggested as inspiration.The proposed direction targets multi-resident activity streams.

5. Data Variability

Smart-home activity data vary over time within a household and across households because residents, habits, configurations, and sensor deployments differ. The survey discusses lifelong learning, transfer learning, and meta-learning as adaptation strategies.

5.1. Temporal Drift: Temporal drift occurs when residents’ behaviors and habits change, making newly captured data differ from the original training data.The model is trained on an initial portrait of activities, but the statistical properties of later data change over time.
5.1. Temporal Drift: Lifelong learning could continuously update HAR models with new data, potentially including occasional user-provided labels.The proposed interactive setting requests labeled data from users periodically to support continued adaptation.
5.2. Variability of Settings: Models learned in one household transfer poorly to another because house configurations, sensor equipment, family compositions, and habits differ.Sensor location, number, and type can influence recognition performance across homes.
5.2. Variability of Settings: Transfer learning can adapt pretrained models to different data distributions, while meta-learning aims to solve new household tasks using few training samples.The survey notes that meta-learning had not yet been applied to HAR despite its relevance to household variability.

6. Datasets

The survey examines real and synthetic smart-home datasets, emphasizing their trade-offs, collection burdens, sensor-placement constraints, and annotation difficulties. It concludes that existing datasets enable evaluation only under particular conditions and remain imperfect representations of real homes.

6. Datasets: Laboratory datasets lack sufficient activity variety and realism, motivating public datasets recorded in real homes with volunteer residents.Competitions such as EvAAL-AR and UCAmI Cup support comparison under shared conditions.
6. Datasets: Real dataset production is tedious because sensor-equipped homes, volunteer actors, annotation, and post-processing require substantial effort.These practical demands make recording campaigns difficult to manage.
6. Datasets: The survey analyzes real and synthetic datasets to compare the advantages and disadvantages of the two approaches.The section explicitly frames its purpose as understanding the trade-offs between these dataset types.
6.1. Sensor Type and Positioning Problem: Sensor choice and placement affect privacy, activity coverage, and robustness when rooms or furnishings change.A sensor positioned for a bed, for example, may lose information if the bed is moved.
6.1. Sensor Type and Positioning Problem: Different homes use different sensor counts, layouts, and topologies, creating coverage, synchronization, and deployment challenges.Examples range from 14–21 sensors in the Kasteren dataset to 236 sensors in Orange4Home.
6.3. Real and Synthetic Datasets: Real datasets offer variability and representativeness, but strategic sensor placement and unusually large sensor sets may not reflect actual homes.The survey cautions that these design choices help inference while departing from real-world sensor equipment.
6.2. Synthetic Datasets: Synthetic datasets enable rapid evaluation of sensor configurations and large-scale data generation without real homes or volunteers.They also provide more precise annotations than diary, smartphone, or voice-recording methods.
6.3. Outlines: Resident self-annotation and post-processing can introduce errors and uncertainty, while few datasets represent concurrent or interleaved activities.Different annotation taxonomies also complicate comparisons across datasets.

7. Evaluation Methods

The survey reviews evaluation methods for smart-home HAR and shows that dataset construction, preprocessing, class balance, and temporal splitting can strongly affect reported performance and generalization. Method suitability depends on the activity representation and temporal dependencies.

Evaluation Principles: Using the same data for training and testing causes overfitting, so evaluation requires separate training and test data.A model can memorize observed labels without predicting unseen samples usefully.
Evaluation Principles: Dataset preprocessing and evaluation methodology can bias final results, making algorithm comparisons sensitive to experimental choices.The survey specifically highlights biases induced by preprocessing and evaluation-method selection.
Dataset and Labelling Issues: Grouping semantically close activities improves comparability across datasets but reduces activity-label granularity.Examples include grouping preparing breakfast, lunch, dinner, and snacks under “preparing a meal.”
Dataset and Labelling Issues: The “Other” class can represent about 50% of a dataset, combining unlabeled events or multiple activities and creating class imbalance.Suppressing this class is one response, while isolating its sequences could support later annotation and discovery of activities.
Dataset and Labelling Issues: Missing labels and duplicated events in datasets such as CASAS Milan require cleaning before algorithm formation, with results depending on data quality.The survey notes duplicated timestamps, sensors, values, and activity labels.
Validation Methods: Overlapping sliding windows distributed across training and test sets can make them too similar, inflating performance and weakening generalization.The survey considers this setup especially problematic when windows overlap strongly.
Validation Methods: Leave-One-Out validation by day uses one day for testing and the others for training, but the test size may be too small to demonstrate smart-home generalization.Each day becomes the test day in turn, allowing broad training coverage but limited test significance.
Validation Methods: K-fold cross-validation is suitable for offline, pre-segmented activities when segment time dependence is ignored, whereas other methods are needed when temporal links matter.Leave-One-Out Cross-Validation can process small datasets but treats days as independent.

8. General Conclusion and Discussions

The survey organizes smart-home HAR around classification, feature extraction, and time-series analysis while reviewing deep-learning progress and unresolved deployment barriers. It concludes that ambient-sensor HAR remains short of real-world readiness and identifies language-based, adaptive, hardware, privacy, and security directions.

Taxonomy and Challenges: Ambient-sensor HAR must handle complex, concurrent, multi-occupant activities, data drift, and variability caused by house configuration, sensor choice, and sensor location.These sources of variability make collected data sensitive to the deployment environment.
Taxonomy and Challenges: The survey’s taxonomy identifies classification, automatic feature extraction, and time-series analysis as the main components of real-use HAR algorithms.The components must recognize patterns from raw data while integrating temporal analysis.
Deep Learning: CNNs have addressed pattern recognition and feature extraction, while LSTMs have improved sequence analysis, but sequence challenges remain largely unresolved.The survey reports equivalent performance levels for CNN- and LSTM-based approaches while noting continuing sequence-analysis difficulties.
Conclusion: The surveyed state of the art is still far from real-use deployment because datasets have shortcomings and evaluation metrics need standardization for fair comparison.The authors connect deployment readiness to evaluation procedures that reflect real-use requirements.
Opportunities: The survey proposes NLP-inspired sequence and semantic techniques to address sparse data, data drift, long-term context, and composite activity analysis.Semantic encodings could relate unfamiliar data to known activities, while language-model ideas motivate long-context sequence analysis.
Opportunities: Meta-learning and interactive learning are identified as possible routes for adapting to changes in habits, users, and sensor sets.The paper presents this adaptation problem as currently unsolved but potentially addressable by these research directions.
Deployment Constraints: Full smart-home service deployment also depends on hardware development and user acceptability, beyond the HAR algorithms themselves.The survey therefore treats algorithmic progress as only one part of deployment.
Deployment Constraints: Real homes cannot generally match datasets equipped with large sensor sets, so systems must optimize hardware under budget, house-configuration, and inhabitant constraints.The authors expect improved IoT accuracy, autonomy, and cost to improve accessibility but still reject lavish sensor deployments as realistic.