PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Predictive Coding for Human Activity Recognition

Harish Haresamudram, Irfan Essa, Thomas Ploetz

arXiv:2012.05333v1cs.LG

TL;DR

HAR needs effective representations when labeled sensor data are scarce, even though unlabeled wearable recordings are easier to collect. The paper introduces CPC to learn temporally informed representations from unlabeled data and reports improved recognition in limited-label settings, while identifying sensor-placement transfer and real-to-virtual domain shift as open boundaries.

  • Problem

    Wearable sensor data are easy to record but difficult to annotate, and existing representation learning methods often do not explicitly model their temporal structure.

  • Method

    CPC uses autoregressive prediction of multiple future latent timesteps to learn long-term temporal representations from unlabeled sensor data for downstream HAR.

  • Results

    CPC pre-training improves HAR recognition performance when only small amounts of labeled training data are available.

  • Takeaways & Limitations

    The learned representations can be integrated into standard activity-recognition chains and support more economical use of annotations.

  • Takeaways & Limitations

    Effective transfer to applications with sensor placements different from the source dataset remains an open challenge.

Abstract

from arXiv · show

Feature extraction is crucial for human activity recognition (HAR) using body-worn movement sensors. Recently, learned representations have been used successfully, offering promising alternatives to manually engineered features. Our work focuses on effective use of small amounts of labeled data and the opportunistic exploitation of unlabeled data that are straightforward to collect in mobile and ubiquitous computing scenarios. We hypothesize and demonstrate that explicitly considering the temporality of sensor data at representation level plays an important role for effective HAR in challenging scenarios. We introduce the Contrastive Predictive Coding (CPC) framework to human activity recognition, which captures the long-term temporal structure of sensor data streams. Through a range of experimental evaluations on real-life recognition tasks, we demonstrate its effectiveness for improved HAR. CPC-based pre-training is self-supervised, and the resulting learned representations can be integrated into standard activity chains. It leads to significantly improved recognition performance when only small amounts of labeled training data are available, thereby demonstrating the practical value of our approach.

1 INTRODUCTION

HAR commonly relies on labeled sensor data, but wearable sensors make unlabeled recordings easier to collect than annotations. The paper therefore models temporal structure with CPC to improve representations when labels are limited.

  • Wearable sensor recordings are straightforward to collect, whereas obtaining activity annotations is comparatively difficult.
  • Existing unsupervised and self-supervised methods learn useful representations, but most do not explicitly target the temporality of body-worn sensor data.
  • CPC predicts multiple future latent timesteps so representations encode global structure between temporally separated parts of sensor streams.
  • The proposed CPC pre-training is integrated into standard activity-recognition chains and supports fine-tuning with limited labeled data.

2 RELATED WORK ON REPRESENTATIONS OF SENSOR DATA IN HUMAN ACTIVITY RECOGNITION

Prior HAR representation learning spans handcrafted features, supervised models, autoencoders, self-supervision, and contrastive approaches. These methods motivate learning from unlabeled sensor data while highlighting the importance of task design and temporal context.

  • Feature extraction in HAR has shifted from handcrafted representations toward learned features integrated with end-to-end recognition models.
  • Unsupervised HAR work includes RBMs, autoencoders, convolutional autoencoders, and self-supervised pre-training used as representation-learning baselines.
  • Self-supervised learning defines pretext tasks from data itself so their supervisory signals can support downstream recognition.
  • Contrastive learning compares positive and negative examples to encourage representations that distinguish semantically similar from dissimilar data.
  • Wearable HAR has also used transformation prediction and masked-timestep reconstruction, including transformer-based reconstruction for transfer and fine-tuning.

3 SELF-SUPERVISED PRE-TRAINING WITH CONTRASTIVE PREDICTIVE CODING

The CPC framework pre-trains representations from unlabeled sensor windows and then uses them in a standard HAR classification chain. It predicts future latent representations with autoregressive context and contrastive learning.

  • CPC pre-training segments unlabeled sensor data into overlapping windows and learns representations before classifier-based activity recognition.
  • A nonlinear encoder maps sensor windows to latent representations, while an autoregressive model summarizes past latents into a context representation.
  • Future latent timesteps are scored through separate linear transformations, avoiding direct modeling of the high-dimensional sensor distribution.
  • InfoNCE training identifies one positive future sample among N−1 negative samples from a proposal distribution.
  • Implementation samples a timestep within each window, predicts subsequent steps, and draws negatives from the same timestep in other batch windows.
  • The downstream classifier is a three-layer feedforward network whose final softmax layer predicts the activity classes.

4 HUMAN ACTIVITY RECOGNITION BASED ON CONTRASTIVE PREDICTIVE CODING

The study evaluates CPC-based representations within a standard HAR framework using multiple real-world movement datasets. Its evaluation covers locomotion-oriented activities and participant-based data splits.

  • CPC pre-training is integrated into the standard Activity Recognition Chain and evaluated through extensive experiments on limited annotated data.
  • The evaluation focuses primarily on locomotion activities such as walking, running, and sitting across four benchmark datasets.
  • Training, validation, and test partitions are generally formed by randomly assigning participants rather than individual sensor windows.
  • The datasets include smartphone- and wearable-sensing studies with varied participants, sensor configurations, activity inventories, and collection platforms.

4.2 Implementation Details

The framework uses raw, normalized sensor data segmented into overlapping windows, then applies CPC pre-training and classifier training with tuned optimization settings.

  • Raw accelerometer and gyroscope data are downsampled to 30 Hz, normalized using training-split statistics, and left unfiltered.
  • Sensor streams are segmented into 1-second windows with 50% overlap between consecutive windows.
  • CPC pre-training runs for 150 epochs with Adam optimization, tuned learning rates, and prediction horizons k∈{2, 4, 8, 12, 16}.
  • The classifier is trained for 150 epochs using cross-entropy loss, tuned learning rates, and decay by 0.8 every 25 epochs.

4.3 Performance Metric

The study evaluates representation quality with test-set mean F1-score and examines both general activity-recognition performance and limited-label settings.

  • Test-set mean F1-score is the primary metric because the datasets have substantial class imbalance.
  • Mean F1-score averages class-specific F1 values, with each class F1 computed from its precision and recall.
  • The experiments first assess the raw discriminative quality of representations learned during self-supervised pre-training.
  • They then compare learned weights with end-to-end training when annotated data are limited, reflecting practical post-deployment labeling constraints.

5 RESULTS AND DISCUSSION

CPC provides effective temporal representations for HAR, outperforming unsupervised baselines across most datasets and improving performance when labeled data are scarce. Its gains over supervised or random-weight alternatives vary by dataset and label availability.

  • 5.1 Activity Recognition: CPC-based pre-training outperforms all compared unsupervised approaches on Mobiact, Motionsense, and USC-HAD, while matching masked reconstruction on UCI-HAR.
  • 5.1 Activity Recognition: Relative to DeepConvLSTM, CPC improves performance by approximately 4% on Motionsense and over 7% on USC-HAD.
  • 5.2 Semi-Supervised Learning on Limited Labeled Data: The limited-label evaluation pre-trains on the entire unlabeled dataset, freezes the learned encoder and autoregressive weights, and trains a randomly initialized classifier.
  • 5.2 Semi-Supervised Learning on Limited Labeled Data: On Mobiact, CPC exceeds DeepConvLSTM by over 5% throughout the tested label range and exceeds random frozen features by over 15% above ten samples per class.
  • 5.2 Semi-Supervised Learning on Limited Labeled Data: On Motionsense, CPC surpasses DeepConvLSTM beyond one labeled sample per class, reaching approximately 10% maximum improvement at 50 samples.
  • 5.2 Semi-Supervised Learning on Limited Labeled Data: On UCI-HAR, CPC exceeds DeepConvLSTM even with one labeled sample per class, with the difference peaking above 10% at ten samples per class.
  • 5.2 Semi-Supervised Learning on Limited Labeled Data: The largest unlabeled dataset, Mobiact, shows the strongest improvement over random weights, suggesting that more unlabeled data may improve representation quality.

6 DISCUSSION

The paper argues that predicting multiple future sensor timesteps with contrastive learning targets temporal characteristics directly in the representation and integrates into standard HAR pipelines.

  • CPC predicts multiple future timesteps with contrastive learning to obtain discriminative features from temporal movement structure.
  • The pre-training scheme operates at the feature-extraction stage, allowing integration into traditional activity-recognition chains.

6.1 Analyzing Contrastive Predictive Coding

The experiments show that CPC performance depends on encoder design, prediction horizon, and which pre-trained weights are transferred. Effective settings avoid trivial prediction tasks, predict multiple future timesteps, and can benefit from using only a subset of learned weights.

  • 6.1.1 Choice of encoder: 13% lower F1-score results from using a kernel size of 9 instead of kernel size 3.Larger convolutional overlaps make future predictions easier and reduce the discriminative ability of the learned representations.
  • 6.1.2 Number of Future Steps to Predict: Predicting approximately 400ms ahead performs best for Mobiact, Motionsense, and UCI-HAR, while USC-HAD peaks at approximately 270ms.Performance generally improves as the prediction horizon increases, then declines beyond the dataset-specific peak.
  • 6.1.3 Which Pre-Trained Weights Should be Used?: For Mobiact, learned g_enc≤2 weights achieve a mean F1-score of 85.22%, improving over fully supervised training.For UCI-HAR, learned g_enc≤2 weights achieve a mean F1-score of 82.58%, while Motionsense and USC-HAD benefit from the default frozen configuration.
  • 6.1.1 Choice of encoder: Encoder architecture significantly affects CPC representations and downstream activity recognition.Fully connected and 1D convolutional encoders with kernel sizes 3 or 5 perform best among the evaluated choices.
  • 6.1.3 Which Pre-Trained Weights Should be Used?: Using a subset of learned weights can improve activity recognition relative to using all learned weights.The preferred transferred representation depends on the activities under study, so freezing both encoder and autoregressive weights can be suboptimal.

6.2 Practical Value of CPC for Human Activity Recognition Tasks

CPC extends self-supervised representation learning for wearable-sensor HAR by modeling temporal structure through multiple future predictions. The framework is flexible in its prediction setup, architecture, and negative sampling, and unlabeled pre-training can outperform fully supervised training when labeled data are limited.

  • 6.2.1 Incorporating Temporality at Representation Level results in Improved Recognition Performance: CPC learns global time-series structure by predicting multiple future timesteps rather than reconstructing only local sensor information.This provides richer representations for body-worn sensor data than approaches that do not explicitly use temporal characteristics.
  • 6.2.2 CPC is Flexible and Generic: CPC permits alternative target-prediction directions, network architectures, and negative-sampling strategies.The paper cautions that encoder, target, and negative-sample choices must avoid making the contrastive task trivial.
  • 6.2.1 Incorporating Temporality at Representation Level results in Improved Recognition Performance: CPC pre-training on unlabeled data can exceed fully supervised training when only limited labeled data are available.The approach addresses the scarcity of large-scale annotated wearable-sensor datasets while exploiting the relative ease of collecting unlabeled data.

6.3 Research Agenda: Representation Learning for Human Activity Recognition

The research agenda emphasizes learning generic movement representations from abundant unlabeled data, then fine-tuning them for task-specific HAR. Key challenges include adapting representations across sensor placements and addressing domain shift in virtual movement data.

  • Unlabeled datasets such as the UK Biobank can provide generic representations for fine-tuning on task-specific labeled data.
  • Representations must remain effective when target applications use sensor placements different from those in the source dataset.This is especially important for niche scenarios requiring unique body locations.
  • Virtual movement data from mocap and video offer scale and diverse placements but introduce artifacts that can reduce recognition performance relative to real IMU data.Representation learning that handles this domain shift could improve recognition.

7 CONCLUSION

The paper presents temporally focused learned representations for HAR and shows that CPC improves recognition in realistic settings, particularly when labeled data are limited. Its results support using readily collected unlabeled sensor data to derive effective representations for recognition systems.

  • Feature extraction is central to HAR with body-worn movement sensors, motivating learned alternatives to hand-crafted features.
  • The approach learns representations that explicitly capture sensor-data temporality and integrates them into the standard activity recognition chain.
  • CPC-learned features outperform previous unsupervised representation-learning approaches across benchmark scenarios and are on par with supervised approaches.
  • The findings indicate that unlabeled sensor data can support effective representations because such data are readily collected from ubiquitous mobile and body-worn sensing platforms.
Loading 2012.05333v1…