PaperlayerView exact revision on arXiv

Source-linked AI summary

ColloSSL: Collaborative Self-Supervised Learning for Human Activity Recognition

Yash Jain, Chi Ian Tang, Chulhong Min, Fahim Kawsar, Akhil Mathur

arXiv:2202.00758v1cs.LGcs.AIcs.HC

TL;DR

Large labeled HAR datasets are difficult to collect, motivating approaches that learn from unlabeled sensor data. ColloSSL uses time-aligned data from multiple devices as natural transformations for contrastive learning. Across most experiment settings it outperforms supervised and semi-supervised methods, including with one-tenth of the labeled data in most settings.

  • Problem

    Large-scale labeled HAR data are difficult to collect, while robust models need useful features from sensor data.

    Source: §1 · p. 2

  • Method

    ColloSSL learns representations from unlabeled, time-aligned inertial data across devices by treating their views as natural transformations for contrastive learning.

    Source: §1 · p. 3, §9 · p. 25

  • Results

    ColloSSL outperforms fully supervised and semi-supervised techniques in most experiment settings and outperforms fully supervised baselines with one-tenth of the labeled data in most settings.

    Source: §9 · p. 25

  • Takeaways & Limitations

    The authors report that ColloSSL learns well-separable features and suggest the framework can be applied to other sensing modalities and applications in future work.

    Source: §9 · p. 25

    This work focuses on contrastive SSL with positive and negative samples; exploring newer negative-free methods for TSMDS is left to future work.

    Source: §8 · p. 25

Abstract

from arXiv · show

A major bottleneck in training robust Human-Activity Recognition models (HAR) is the need for large-scale labeled sensor datasets. Because labeling large amounts of sensor data is an expensive task, unsupervised and semi-supervised learning techniques have emerged that can learn good features from the data without requiring any labels. In this paper, we extend this line of research and present a novel technique called Collaborative Self-Supervised Learning (ColloSSL) which leverages unlabeled data collected from multiple devices worn by a user to learn high-quality features of the data. A key insight that underpins the design of ColloSSL is that unlabeled sensor datasets simultaneously captured by multiple devices can be viewed as natural transformations of each other, and leveraged to generate a supervisory signal for representation learning. We present three technical innovations to extend conventional self-supervised learning algorithms to a multi-device setting: a Device Selection approach which selects positive and negative devices to enable contrastive learning, a Contrastive Sampling algorithm which samples positive and negative examples in a multi-device setting, and a loss function called Multi-view Contrastive Loss which extends standard contrastive loss to a multi-device setting. Our experimental results on three multi-device datasets show that ColloSSL outperforms both fully-supervised and semi-supervised learning techniques in majority of the experiment settings, resulting in an absolute increase of upto 7.9% in F_1 score compared to the best performing baselines. We also show that ColloSSL outperforms the fully-supervised methods in a low-data regime, by just using one-tenth of the available labeled data in the best case.

1 INTRODUCTION

Large labeled HAR datasets are difficult to collect, while time-synchronized devices worn by one user offer a source of unlabeled data that can serve as natural transformations. ColloSSL uses this opportunity for contrastive feature learning and is evaluated against supervised and semi-supervised baselines.

1 INTRODUCTION

  • Labeling large-scale HAR datasets is difficult because sensor streams resist visual inspection, leaving many collections small and controlled.The resulting models may not generalize to real-life situations.Source: §1 · p. 2
  • Time-synchronized devices capture the same activity from different perspectives, allowing their sensor data to act as natural transformations for self-supervised learning.For example, a smartphone at the thigh and a smartwatch at the wrist record running from distinct placements.Source: §1 · pp. 2–3
  • ColloSSL learns features from unlabeled inertial data across multiple user-worn devices, then uses those features for downstream HAR classification.The approach extends prior work [46] [57] by leveraging cross-device natural transformations for contrastive learning.Source: §1 · p. 3
  • The framework introduces device selection, contrastive sampling, and a multi-device loss, and is evaluated on three multi-device HAR datasets against supervised and semi-supervised baselines.The paper reports stronger recognition accuracy and labeled-data efficiency across its evaluation.Source: §1 · p. 3

2 RELATED WORK

Prior HAR work addresses labeled-data scarcity with semi-supervised and self-supervised approaches, but many SSL methods use one device or similar data distributions. ColloSSL instead uses unlabeled data across multiple devices during training, without requiring all devices at inference.

2 RELATED WORK

  • Labeled sensor time-series are difficult to collect in diverse free-living settings, limiting the quality and breadth of supervised HAR data.Sensor types, placements, use cases, and target activities vary across deployments.Source: §2 · p. 4
  • Earlier HAR self-supervision used transformation prediction or contrastive learning, but cited approaches focused on similar distributions or a single sensor.Prior methods include signal-transformation pretraining [46] and SimCLR-based time-series transformations [58].Source: §2 · pp. 4–5
  • Unlike labeled sensor-fusion methods, ColloSSL learns features from unlabeled data across devices during training and can infer independently on each device.Fusion approaches concatenate multiple streams and assume that all sensor data streams are labeled.Source: §2 · p. 5

3 PRELIMINARIES

The preliminaries introduce time-synchronous multi-device systems (TSMDS) and contrastive self-supervised learning. They describe the setting, its assumptions, and how contrastive learning uses paired and contrasting examples to learn features.

3 PRELIMINARIES

  • The preliminaries frame TSMDS and contrastive self-supervised learning as the setting and learning approach used in the paper.The section introduces the setting conceptually before its later mathematical formalization.Source: §3 · p. 5, §3.1 · p. 5, §3.2 · p. 6

3.1 Time-Synchronous Multi-Device Systems

  • TSMDS devices observe the same phenomenon simultaneously and record time-aligned data; the HAR setup assumes matching or resampleable sampling rates and time alignment.The paper acknowledges perfect alignment as a strong assumption and reports robustness to moderate temporal misalignment.Source: §3.1 · pp. 5–6

3.2 Primer on Contrastive Self-Supervised Learning

  • Contrastive learning brings anchor and positive embeddings closer while pushing anchor and negative embeddings apart, with the learned features used for downstream classification.The chosen perturbations matter: cited work reports ImageNet top-1 accuracy falling from 56.3% with crop and coloring to 2.6% with rotation.Source: §3.2 · p. 6

4 MOTIVATION AND CHALLENGES FOR COLLABORATIVE SELF-SUPERVISED LEARNING

Time-aligned data from multiple devices provide natural views of the same activity, offering an alternative to manually designed transformations. ColloSSL must choose useful devices and samples for its contrastive objective.

4 MOTIVATION AND CHALLENGES FOR COLLABORATIVE SELF-SUPERVISED LEARNING

  • Data from different device placements can serve as natural transformations because body-part motion changes the sensor perspectives on the same activity.Body parts differ in translation and rotation characteristics, such as the wrist having more rotational freedom than the chest.Source: §4 · p. 6
  • Device Selection must avoid pairing anchors with distributions too far away, and account for distribution shifts across activities.The proposed data-driven strategy uses MMD to select positive and negative devices dynamically.Source: §4 · p. 7, Figure 2 · p. 7
  • Contrastive Sampling uses time-synchronized samples from positive devices and time-asynchronized samples from negative devices.Multi-View Contrastive Loss accepts an arbitrary number of positive and negative samples to compute the training objective.Source: §4 · pp. 7–8

5 COLLABORATIVE SELF-SUPERVISED LEARNING

ColloSSL is formulated for time-aligned, unlabeled sensor datasets collected across devices, with a small labeled dataset available for an anchor device. Its objectives are to pretrain a feature extractor on the unlabeled data and train an HAR classifier from labeled anchor-device examples.

5.1 Problem Statement

  • The TSMDS setting provides pre-segmented, time-aligned unlabeled sensor windows from multiple devices, alongside a small labeled dataset for an anchor device.Each sample stacks three-axis accelerometer and three-axis gyroscope data into a six-dimensional vector.Source: §5.1 · p. 8
  • ColloSSL uses the unlabeled datasets from all devices to train a feature extractor that generates embeddings for anchor-device data.Source: §5.1 · p. 8
  • The pretrained feature extractor supplies embeddings for labeled anchor-device samples, which are used to train an HAR classifier.Source: §5.1 · p. 8

5.2 Solution Overview

The solution first uses unlabeled, time-aligned data for multi-view contrastive learning, then uses labeled anchor-device data to train an HAR classifier.

5.2 Solution Overview

  • The pipeline samples time-aligned unlabeled data across devices, selects positive and negative devices and samples, then computes a Multi-view Contrastive Loss from their embeddings.Source: §5.2 · p. 8, Figure 3 · p. 9
  • The contrastive-learning steps repeat until the loss converges, producing a pretrained feature extractor.Source: §5.2 · p. 9
  • The pretrained feature extractor and labeled anchor-device data are then used to train an HAR classifier.Source: §5.2 · p. 9

5.3 Feature Extractor

ColloSSL uses a temporal convolutional feature extractor, with a consistent architecture chosen to support fair comparisons across techniques.

5.3 Feature Extractor

  • The feature extractor is a temporal one-dimensional CNN, chosen to keep the model architecture consistent across techniques for fair baseline comparisons.Source: §5.3 · pp. 9–10
  • Its architecture has three Conv1D layers with 32, 64, and 96 filters and kernel sizes 24, 16, and 8, respectively, all with stride 1.Source: §5.3 · p. 10
  • The model takes 100 × 6 windows sampled at 50 Hz over two seconds and produces a 96 × 1 feature embedding.Source: §5.3 · p. 10

5.4 Device Selection and Contrastive Sampling

Device Selection and Contrastive Sampling address which devices and samples provide positives and negatives for contrastive learning. The approach uses distribution distances and temporal alignment to guide those choices without access to class labels.

5.4 Device Selection and Contrastive Sampling

  • A good positive should share the anchor’s class and come from a device with a similar data distribution; time alignment is assumed to imply matching labels across devices.Source: §5.4 · p. 10
  • Useful negatives should be different-class samples near the anchor in embedding space, but without labels the method cannot strictly enforce the different-class condition.Near negatives can provide a stronger supervisory signal, while a same-class negative pushed away could harm classification.Source: §5.4 · pp. 10–11

5.4.1 Device Selection.

  • For Device Selection, pairwise Maximum Mean Discrepancy (MMD) ranks candidate devices: the closest device is the positive, while all candidates contribute negative samples weighted inversely by MMD.Using all candidates as negatives hedges against a device producing a same-class negative; just one closest positive device performed best in experiments.Source: §5.4.1 · p. 11

5.4.2 Contrastive Sampling.

  • Contrastive Sampling selects time-synchronized counterparts from positive devices, which are treated as same-label positives for the anchor.Source: §5.4.2 · p. 12
  • For negatives, it selects samples from other time steps because they are more likely to differ in class; a batch size of 512 reduces the chance of label overlap.Different time steps do not guarantee different labels.Source: §5.4.2 · p. 12

5.4.3 Summary.

  • The algorithms first choose positive and negative devices, then choose samples from those devices; a device may serve in both roles, using synchronized samples as positives and asynchronous samples as negatives.Source: §5.4.3 · p. 12

5.5 Multi-view Contrastive Loss

Multi-view Contrastive Loss extends standard contrastive loss to handle multiple positive and negative samples, training the feature extractor to bring anchor and positive embeddings closer while pushing negative embeddings away.

5.5 Multi-view Contrastive Loss

  • Anchor, positive, and negative samples are passed through the feature extractor to produce embeddings for the loss.The selected embeddings come from positive and negative devices, while the anchor has its own embedding.Source: §5.5 · p. 12
  • Multi-view Contrastive Loss extends standard contrastive loss to support multiple positive and negative samples.This makes the objective compatible with contrastive learning across multiple devices.Source: §5.5 · p. 12
  • Cosine similarity, negative-device weights assigned according to (1), and temperature parameterize the objective.The loss is minimized for each batch using stochastic gradient descent.Source: §5.5 · p. 13
  • Loss minimization trains the extractor to increase anchor–positive cosine similarity and decrease anchor–negative similarity.The resulting features are intended for downstream classification tasks.Source: §5.5 · p. 13
  • For downstream HAR training, the extractor is mostly frozen and paired with a classification head trained on a small labeled dataset from the anchor device.Only its last convolution layer is unfrozen; the head has a 1024-unit ReLU hidden layer and an output layer matching the number of labels.Source: §5.5.1 · p. 13

6 EVALUATION

Across three multi-device HAR datasets, ColloSSL outperforms the strongest baseline at full label availability and often matches or exceeds full-data supervised performance with much less labeled data. Its learned embeddings are class-separable, and its saliency patterns resemble those of fully supervised models.

6.1 Experimental Setup

  • Evaluation used time-aligned inertial data from RealWorld, Opportunity, and PAMAP2, with leave-one-group-out testing on a single anchor device.The feature extractor is trained on unlabeled data from the other groups, then fine-tuned with labeled anchor-device data and tested on the held-out group.Source: §6.1 · pp. 13, 15

6.2 Performance of ColloSSL in a Low-data Regime

  • With 10% or 25% of labeled data, ColloSSL beat full-data Supervised-single on 15 of 18 devices; it used the fewest labels on 13 and led low-data F1 on 14.The remaining comparisons include cases of comparable performance and one PAMAP2 ankle-device case where ColloSSL used 100% of labeled data.Source: §6.2 · p. 16, Table 2 · p. 16
  • Across labeled-data fractions, ColloSSL generally outperformed baselines, though some poor-performing baselines are not visible in Figure 4.The figure plots performance at different percentages of labeled anchor-device data; Table 2 provides in-depth results.Source: Figure 4 · p. 17, §6.2 · p. 17

6.3 Comparison of ColloSSL Recognition Performance with Baselines

  • Across all datasets, ColloSSL improves macro F1 by around 7.9% over the best baseline, Supervised-single, with 100% labeled data.Table 3 reports mean and standard deviation of macro F1 scores by anchor device.Source: §6.3 · p. 18, Table 3 · p. 18

6.4 Embedding Similarity and Data Saliency

  • Two-dimensional t-SNE plots show that untuned ColloSSL embeddings are already class-separable, with separation improving after fine-tuning.The learned clustering is largely comparable to that of the fully supervised model trained with all labeled data.Source: §6.4 · pp. 19–20, Figure 5 · p. 19
  • ColloSSL and fully supervised models focus on similar data regions when making predictions.The saliency maps show similar color-intensity patterns for the two approaches.Source: §6.4 · pp. 20–21, Figure 6 · p. 20

7 ANALYSIS

The analysis tests ColloSSL’s component choices and robustness under practical multi-device conditions, reporting improvements from its sampling and weighting strategies alongside resilience to device, sensor, and timing variation.

7.1 Analysis of the Device Selection Strategy

  • ColloSSL outperforms both device-selection baselines; random positives can mismatch anchor distributions, while random negatives may violate N1.The alternatives are random positive and negative selection, and closest-positive with random-negative selection.Source: §7.1 · p. 21

7.2 Analysis of the Contrastive Sampling Approach

  • Asynchronous negative sampling improves performance over synchronous sampling, whose same-class negatives can violate N1.Positive samples remain synchronous in both settings to avoid violating P1.Source: §7.2 · p. 21, Table 5 · p. 22

7.3 Analyzing the Role of Weights in Multi-view Contrastive Loss

  • Weighted negatives improve performance, supporting the hypothesis that they push closer negative embeddings farther from the anchor (N2).The ablation compares the weighted loss with a version assigning every device equal weight.Source: §7.3 · pp. 21–22, Table 6 · p. 22

7.4 Robustness to Sensor Heterogeneity

  • ColloSSL handles sensor heterogeneity and outperforms fully supervised and multi-task SSL baselines; low heterogeneity also improves performance.The experiment induces scale-factor and bias errors in RealWorld IMU data.Source: §7.4 · p. 22, Table 7 · p. 23

7.5 Robustness to Missing Devices

  • At most 1% performance drop is observed as device availability changes.The experiment assumes the anchor device is always available and varies availability of the other devices.Source: §7.5 · p. 23, Figure 7 · p. 23

7.6 Robustness to Temporal Misalignment

  • Within ±0.006 macro F1, performance is unchanged for synchronization errors ≤0.5 seconds; above 0.5 seconds, changes are about ±0.01.These comparisons cover waist and shin anchor devices.Source: §7.6 · p. 24, Table 8 · p. 24

7.7 Generalizability of the Feature Extractor

  • Less than 1% F1 decrease occurs when the feature extractor is pretrained without the evaluated device.The result provides an early indication of transferability to unseen devices for fine-tuning.Source: §7.7 · p. 24, Table 9 · p. 24

8 DISCUSSION AND LIMITATIONS

The discussion weighs ColloSSL’s offline training cost and raw-data requirements against its single-device runtime operation, and identifies boundaries for privacy and extension beyond inertial HAR.

Training Cost of ColloSSL

  • ColloSSL requires more training time than fully supervised learning because it pre-trains on unlabeled data, but training is offline and adds no data-collection cost.The paper says this has no adverse implications for mobile or wearable system resources.Source: §8 · p. 24

Runtime System Cost

  • At runtime, the trained model operates on one device, with expected inference latency and energy costs matching a supervised HAR model.This is presented as an expectation about system costs.Source: §8 · p. 25

ColloSSL as a general framework for learning in TSMDS settings

  • Applying ColloSSL beyond motion data requires redesigning device selection, contrastive sampling, and group contrastive loss for each modality’s signals, behavior, and environment.Audio and vision are proposed as future TSMDS settings to explore.Source: §8 · p. 25

Data Privacy

  • ColloSSL requires raw data from multiple devices, which manufacturers may not want to send to a centralized cloud for privacy or commercial reasons.The paper proposes trusted-edge training or federated self-supervised learning as possible responses.Source: §8 · p. 25

Extension to newer SSL algorithms

  • The work uses contrastive SSL with positive and negative samples; negative-free SSL methods are left for future exploration in TSMDS.The discussion notes that such methods have been proposed and can outperform contrastive methods such as SimCLR.Source: §8 · p. 25

9 CONCLUSION

ColloSSL learns representations from unlabeled, time-aligned inertial data collected across body-worn devices by treating device views as natural transformations. The paper reports broad benchmark gains and data efficiency, while positioning the framework for future use beyond inertial HAR.

9 CONCLUSION

  • ColloSSL uses time-aligned data from multiple body-worn devices as natural transformations, contrasting selected cross-device samples against an anchor to create supervision from unlabeled data.The pipeline learns a representation for downstream HAR.Source: §9 · p. 25
  • ColloSSL outperforms fully supervised and semi-supervised techniques in most experiment settings and can beat fully supervised baselines using one-tenth of labeled data in most settings.The conclusion also reports well-separable features and visualizes saliency maps to illustrate predictions.Source: §9 · p. 25

Related papers

Loading 2202.00758v1…