PaperlayerView exact revision on arXiv

Source-linked AI summary

Ensembles of Deep LSTM Learners for Activity Recognition using Wearables

Yu Guan, Thomas Ploetz

arXiv:1703.09370v1cs.LGcs.AIcs.CV

TL;DR

Wearable HAR remains challenged by noisy, erroneous, ambiguous, and imbalanced data, as well as limitations of frame-based processing. The paper develops ensembles of deep LSTM learners using modified, data-subset-based training. Across Opportunity, PAMAP2, and Skoda, the approach outperforms the state of the art and is positioned as robust for realistic HAR scenarios, while deployment implementation remains unaddressed.

  • Problem

    Wearable HAR still faces noisy or erroneous sensing, imbalanced classes, difficult annotation, and frame-based processing that can neglect sequential context.

  • Method

    The paper trains diverse deep LSTM learners through modified probabilistic subset selection and combines them into ensemble classifiers.

  • Results

    The ensemble approach statistically significantly outperforms the state of the art on the Opportunity, PAMAP2, and Skoda benchmark datasets.

  • Takeaways & Limitations

    The results indicate increased robustness for challenging real-world wearable HAR involving imbalanced class distributions, background data, and noisy or faulty sensor readings.

  • Takeaways & Limitations

    The paper focuses on basic methodology and does not yet address implementation issues relevant to field deployments or products.

Abstract

from arXiv · show

Recently, deep learning (DL) methods have been introduced very successfully into human activity recognition (HAR) scenarios in ubiquitous and wearable computing. Especially the prospect of overcoming the need for manual feature design combined with superior classification capabilities render deep neural networks very attractive for real-life HAR application. Even though DL-based approaches now outperform the state-of-the-art in a number of recognitions tasks of the field, yet substantial challenges remain. Most prominently, issues with real-life datasets, typically including imbalanced datasets and problematic data quality, still limit the effectiveness of activity recognition using wearables. In this paper we tackle such challenges through Ensembles of deep Long Short Term Memory (LSTM) networks. We have developed modified training procedures for LSTM networks and combine sets of diverse LSTM learners into classifier collectives. We demonstrate, both formally and empirically, that Ensembles of deep LSTM learners outperform the individual LSTM networks. Through an extensive experimental evaluation on three standard benchmarks (Opportunity, PAMAP2, Skoda) we demonstrate the excellent recognition capabilities of our approach and its potential for real-life applications of human activity recognition.

1. INTRODUCTION

Deep learning offers strong feature-learning and classification capabilities for wearable HAR, but real-world sensing remains difficult because data are noisy, incomplete, ambiguous, and imbalanced. The paper addresses these challenges with sample-wise deep LSTM ensembles trained using diverse data subsets.

  • Challenges in wearable HAR: Real-world wearable HAR data are noisy, incomplete, difficult to annotate, and often imbalanced, creating problems during training and inference.These issues include missing or erroneous sensor readings, difficult ground-truth collection, and over-represented background classes.
  • Deep learning for HAR: Deep learning reduces reliance on manually engineered features while providing strong classification capacity for activity recognition.The paper presents deep learning as attractive because models can learn rich representations directly from input data.
  • Sequential modeling gap: Conventional HAR commonly analyzes isolated, overlapping sensor windows rather than exploiting the full sequential structure of sensor streams.The paper identifies frame-based inference as a mismatch for models designed to analyze sequential data.
  • Proposed framework: The framework uses deep recurrent LSTM networks for sample-wise prediction and combines multiple learners into ensemble classifiers for realistic wearable HAR.This extends prior work beyond the sliding-window paradigm and targets improved robustness in challenging applications.
  • Proposed framework: A probabilistic subset-selection training procedure creates diverse LSTM learners by assuming that some sensor data are problematic, then iterating the selection to build ensembles.The procedure resembles Bagging and is intended to address noisy, ambiguous, or erroneous training data.

2. BACKGROUND

The background motivates recurrent LSTMs for wearable HAR because temporal context matters beyond isolated frames, then distinguishes this paper’s higher-level ensembles of complete LSTM networks from ordinary deep architectures. Its novelty is data-driven diversity among strong LSTM learners rather than prescribed activity-specific variants.

  • Deep learning for HAR: HAR methods in ubiquitous computing include deep CNNs, recurrent networks such as LSTMs, and combinations of convolutional representation learning with sequence modeling.The background frames CNNs and LSTMs as the two major deep-learning approaches used for wearable HAR.
  • Temporal modeling: Frame-based processing treats sensor frames as statistically independent, potentially limiting recognition when temporal dependencies and complex activities matter.Sequential models are presented as more appropriate when analysis must extend beyond frame boundaries.
  • Ensemble learning: Prior HAR work primarily optimized individual LSTM models, whereas this paper combines multiple complete networks into classifier ensembles.The ensemble operates at a higher abstraction level than the aggregation of nodes within one deep network.
  • LSTM models: LSTM units use forget, input, cell, and output mechanisms to address vanishing or exploding gradients in recurrent networks.The paper describes LSTMs as recurrent models with principally infinite memory and gate-specific parameters.
  • Research hypothesis: The paper hypothesizes that combining diverse strong LSTM learners improves robustness and recognition results in challenging ubiquitous-computing scenarios.LSTMs are selected as base learners because prior work reports strong performance on difficult HAR benchmarks.
  • Ensemble learning: The proposed ensembles derive diversity from data-driven training rather than prescribing base learners for specific activity variants.This contrasts with related video-action ensembles whose learner diversity is tied to predefined activity variants.

3. ENSEMBLES OF DEEP LSTM LEARNERS FOR ACTIVITY RECOGNITION

The paper develops robust HAR using ensembles of deep LSTM learners, targeting noisy, incomplete, imbalanced, and temporally complex wearable-sensing data. Its modified training procedures and learner fusion are designed to improve robustness and recognition performance.

  • Wearable HAR data are noisy and may contain faulty readings, while ground-truth annotation is difficult to obtain.
  • Frame-based analysis can lose temporal context, limiting recognition of complex activities.
  • The framework combines multiple deep recurrent LSTM networks into an ensemble meta-classifier for more robust activity recognition.
  • 3.2.1 Epoch-wise Bagging: Epoch-wise bagging randomizes mini-batch sizes and frame lengths to create diverse LSTM learners without relying on fixed training parameters.
  • 3.2.2 Alternative Loss Function for LSTM learners: The method also explores F1-score loss, a cost-sensitive global loss intended to address data imbalance alongside cross entropy.
  • 3.2.2 Alternative Loss Function for LSTM learners: Mini-batch training with F1 loss only approximates the true whole-dataset gradient toward an optimal F1 score.
  • 3.3 Combining Multiple LSTMs into HAR Ensembles: Fusing the best-performing, typically least-correlated LSTM learners substantially improves model performance.

4. EXPERIMENTAL EVALUATION

The evaluation uses standard benchmark datasets and follows a documented methodology for training, testing, and dataset-specific reporting. The experiments are organized to support comparison with prior work.

  • The framework is evaluated on three benchmark datasets considered standard and widely used in HAR research.
  • Table I reports the configuration of mini-batch size and frame length for model training and testing.
  • The evaluation section describes training and evaluation methodology, dataset details, and the resulting performance analysis.

4.1 Model Training, and Evaluation Protocol

The protocol standardizes model configurations and emphasizes realistic, comparable evaluation settings. Results are averaged across repeated runs to account for randomness in ensemble creation.

  • Experiments use raw sensor data without feature extraction, apart from potential subsampling and modality fusion.
  • The LSTM configuration uses two layers of 256 units, dropout probability 0.5, ADAM, and learning rate 0.001.
  • The evaluation protocol targets meaningful, practically relevant, and comparable results while avoiding unrealistically over-optimistic scenarios.
  • The F1 metric is defined per activity class using true-positive, false-positive, and false-negative prediction counts.
  • Performance is reported as averaged F1 scores with standard deviations over 30 repetitions per model configuration.
  • Results use sample-wise prediction rather than frame-wise prediction, with literature results converted where necessary for comparability.

4.2 Datasets

The evaluation uses three benchmark datasets with continuous wearable-sensor recordings representing diverse activity-recognition tasks. Their class distributions and activity durations expose imbalance, background-data complexity, and substantial variability relevant to automated recognition.

  • Dataset selection: Three widely used benchmarks provide continuous inertial-measurement-unit recordings from participants wearing sensors at different body positions.The datasets represent diverse but typical human-activity-recognition tasks.
  • Dataset challenges: The benchmarks exhibit class imbalance, diverse background activity, and substantial variability in activity durations, complicating automated recognition.Opportunity is especially imbalanced and strongly biased toward the NULL class; background data cannot generally be removed through simple filtering because it is diverse.
  • Opportunity: Opportunity contains annotated domestic-activity recordings from four participants wearing sensors at 12 on-body positions.The dataset focuses on kitchen routines and everyday-life activities.
  • PAMAP2: PAMAP2 was collected in a scripted setting with nine participants performing 12 daily-living and sportive activities using multimodal body-worn platforms.Recorded signals include inertial measurements, temperature, and heart rate.
  • Skoda: Skoda represents manufacturing activity recognition, covering assembly-line workers’ manual quality checks in a car-production environment.The dataset includes manipulative gestures such as checking the boot, opening vehicle components, and turning the steering wheel.

4.3 Results

Across three benchmark tasks, LSTM ensembles consistently outperform individual learners, with combined CE+F1 training providing the strongest gains on challenging or imbalanced data. Performance generally plateaus at ten base learners, while benefits vary with dataset homogeneity and class difficulty.

  • Ensemble size: Ensembles of LSTM learners outperform single models consistently across all three recognition tasks.The authors attribute this to learners focusing on different aspects of challenging sample data.
  • Ensemble size: Performance generally plateaus at M = 10 base learners, with no statistically significant M = 20 improvement except for Opportunity under F1 loss.Increasing learner count improves results for both loss functions before this plateau.
  • Loss-function diversity: Opportunity gains further when CE and F1 learners are combined, whereas PAMAP2 does not benefit from loss-function diversity.The additional variation significantly increases Opportunity performance but adds no comparable benefit for the more homogeneous PAMAP2 task.
  • Class-wise results: The combined CE(10)+F1(10) ensemble produces the largest gains for challenging classes and almost never harms activity-class performance.This pattern is reported as especially encouraging for real-world scenarios.
  • Class-wise results: Class improvements are strongest for difficult, diverse activities, while classes with very small sample counts benefit less because learners capture less variability.Confusion matrices identify the NULL class as the largest source of confusion when present.
  • Benchmark comparison: Dataset conditions shape the ensemble benefit: Opportunity shows substantial improvement, PAMAP2 moderate significant gains, and Skoda no substantial boost despite significance.The authors associate Skoda’s limited gain with its single-worker setting and already high baseline specialization.

5. DISCUSSION

The paper develops ensembles of deep LSTM networks for robust, sample-wise wearable activity recognition in noisy and imbalanced real-life settings. Experiments across three standard benchmarks show improved recognition performance and potential for deployment, while implementation for field products remains future work.

  • 5. DISCUSSION: The framework combines multiple deep LSTM learners into ensembles for sample-wise activity recognition in challenging real-life wearable-sensing scenarios.The approach targets noisy and imbalanced datasets and uses a modified training procedure to derive diverse learners.
  • 5. DISCUSSION: The modified training procedure specifically targets problematic data situations while retaining the underlying LSTM architecture and supporting replication with standard open-source toolkits.The method selects subsets of data during training to build diverse base learners; the supplied discussion emphasizes unchanged learner anatomy and straightforward system integration.
  • 5. DISCUSSION: The method has potential for future HAR applications including activity prediction for time-critical inference.The supplied discussion links this potential to the recognition approach’s sample-wise operation.
  • 5. DISCUSSION: The approach significantly improves recognition accuracy and robustness across Opportunity, PAMAP2, and Skoda, with statistically significant gains over the state of the art.The reported evaluation highlights substantial gains in challenging scenarios such as Opportunity and little detrimental effect where single models already perform highly.
  • 5. DISCUSSION: The paper focuses on basic methodology rather than implementation issues relevant to field deployments or products.The authors note that ensembles increase the parameter space, while arguing that pruning and wearable implementation remain accessible.

A. REDUCING LOSS IN LSTM ENSEMBLES

This section introduces the loss-reduction analysis for LSTM ensembles, framing cross-entropy loss for sequential predictions and unseen test data. It establishes the setup used to compare individual learners with fused models.

  • A. REDUCING LOSS IN LSTM ENSEMBLES: At each time step, an LSTM uses the current signal and previous hidden and cell states to make a sample-wise prediction.The sequential inputs are x_t, h_t−1, and c_t−1.
  • A. REDUCING LOSS IN LSTM ENSEMBLES: The section represents the correct class with a binary vector before expressing cross-entropy in vector form.The vector has one entry for the correct class and zero entries for the others.
  • A. REDUCING LOSS IN LSTM ENSEMBLES: The analysis examines how LSTM loss, specifically cross-entropy, is reduced for unseen test data.Cross-entropy is described as the negative logarithmic loss of the predicted probability for the target class.

Average Loss of M Individual Learners

This section defines the expected cross-entropy loss of an individual LSTM learner over N test samples and then averages that loss across M models.

  • Average Loss of M Individual Learners: For N test samples, the expected cross-entropy loss L_m is defined for the m-th LSTM model.The expectation is taken over test-data loss at time steps t = 1, 2, ..., N.
  • Average Loss of M Individual Learners: The average loss L_avg is formed by averaging the losses of M individual models.This quantity provides the individual-learner baseline for comparison with score-level fusion.

Loss when fusing M models

This section defines cross-entropy for a score-level-fused ensemble and compares it with the average loss of its individual LSTM learners. The formal result favors fusion, especially for diverse models.

  • Loss when fusing M models: The ensemble’s cross-entropy L_fusion is defined from score-level fusion of multiple individual learners at each time step.The average fused loss is then computed over the N test samples.
  • Loss when fusing M models: The difference between individual-model and ensemble loss is defined as Δ = L_avg − L_fusion.This quantity measures the loss reduction associated with fusion on the test data.
  • Loss when fusing M models: L_avg ≥ L_fusion, so the fused ensemble’s average loss is always less than or equal to the average loss of individual learners.The result follows from the inequality of arithmetic and geometric means.
  • Loss when fusing M models: Fusing M diverse, less-correlated LSTM models improves prediction probabilities for the true classes.The identical-learner case is presented separately, while diversity is identified as the general source of ensemble benefit.
Loading 1703.09370v1…