PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Self-Supervised Learning of Multi-Sensor Representations for Embedded Intelligence

Aaqib Saeed, Flora D. Salim, Tanir Ozcelebi, Johan Lukkien

arXiv:2007.13018v1cs.LGstat.ML

TL;DR

Privacy, bandwidth and annotation costs make centralized supervised learning difficult for sensor data from edge devices. The paper trains temporal networks to distinguish aligned raw signals and wavelet scalograms through a self-supervised task in federated settings. Its representations are competitive with fully supervised networks, and few-label fine-tuning improves F-score over training from scratch.

  • Problem

    Privacy, bandwidth limits and annotation costs constrain centralized supervised learning from the abundant sensor data produced by distributed IoT devices.

    Source: §I · pp. 1–2

  • Method

    The method trains a deep temporal network with a contrastive task that determines whether raw sensor signals and their wavelet scalograms are aligned.

    Source: §I · pp. 2–3

  • Results

    SCN was competitive with fully supervised networks, and fine-tuning with five or ten labeled instances per class improved F-score by 5%-6% over training from scratch.

    Source: §I · p. 3

  • Takeaways & Limitations

    The learned model can serve as a semantic feature extractor or initialization for adapting to an end task with few labeled instances.

    Source: §I · p. 2

    Federated experiments randomly partitioned training data because the evaluated datasets had few users, potentially producing unbalanced IID clients without extreme heterogeneity.

    Source: B. Quality assessment of the learned… · p. 7

Abstract

from arXiv · show

Smartphones, wearables, and Internet of Things (IoT) devices produce a wealth of data that cannot be accumulated in a centralized repository for learning supervised models due to privacy, bandwidth limitations, and the prohibitive cost of annotations. Federated learning provides a compelling framework for learning models from decentralized data, but conventionally, it assumes the availability of labeled samples, whereas on-device data are generally either unlabeled or cannot be annotated readily through user interaction. To address these issues, we propose a self-supervised approach termed \textit{scalogram-signal correspondence learning} based on wavelet transform to learn useful representations from unlabeled sensor inputs, such as electroencephalography, blood volume pulse, accelerometer, and WiFi channel state information. Our auxiliary task requires a deep temporal neural network to determine if a given pair of a signal and its complementary viewpoint (i.e., a scalogram generated with a wavelet transform) align with each other or not through optimizing a contrastive objective. We extensively assess the quality of learned features with our multi-view strategy on diverse public datasets, achieving strong performance in all domains. We demonstrate the effectiveness of representations learned from an unlabeled input collection on downstream tasks with training a linear classifier over pretrained network, usefulness in low-data regime, transfer learning, and cross-validation. Our methodology achieves competitive performance with fully-supervised networks, and it outperforms pre-training with autoencoders in both central and federated contexts. Notably, it improves the generalization in a semi-supervised setting as it reduces the volume of labeled data required through leveraging self-supervised learning.

I. INTRODUCTION

Privacy, bandwidth and annotation costs make learning from distributed sensor data difficult, while prior self-supervised work had not studied diverse IoT sensing modalities. This paper introduces a wavelet-based signal–scalogram task for learning representations from unlabeled sensor inputs in central and federated settings.

I. INTRODUCTION

  • IoT sensor data are difficult to use for supervised learning because privacy, bandwidth limits, annotation costs and device diversity constrain data collection and centralization.The paper notes that training often relies on labeled datasets in a central environment.Source: §I · p. 1
  • Prior self-supervised methods covered modalities such as audio, visual and textual data, but the authors identify a lack of study on sensor modalities including EEG, accelerometers and blood volume pulse.The introduction also describes federated learning research as focused on supervised tasks with readily acquired annotations.Source: §I · p. 2
  • The authors propose combining self-supervision and federated learning to learn representations from unlabeled, private and diverse sensory data.They present this as relevant to embedded, personalized machine-learning tasks.Source: §I · p. 2
  • SCN learns from raw sensor signals and their wavelet scalograms by using a contrastive objective to distinguish aligned from misaligned pairs.The task is designed for central and federated training without human labeling.Source: §I · pp. 2–3
  • SCN achieved performance competitive with fully supervised networks; fine-tuning with five or ten labeled instances per class improved F-score by 5%-6% over training from scratch.The paper also reports that SCN performed better than transferring supervised features between related tasks.Source: §I · p. 3

II. BACKGROUND AND RELATED WORK

The section defines representations as network outputs from an input-mapping function and frames the goal as learning general-purpose features for downstream tasks. It positions the work at the intersection of self-supervision and federated learning for distributed user data.

II. BACKGROUND AND RELATED WORK

  • A representation or feature is a vector produced by applying the network mapping function F to an input, using an intermediate or penultimate layer.The network is parameterized by θ and maps inputs from X into outputs in Z.Source: §II · p. 3
  • The goal is to learn general-purpose representations that make subsequent tasks easier to solve.Source: §II · p. 3
  • The work combines self-supervision with federated learning to pursue on-device learning from distributed user data that cannot be centralized due to privacy concerns and data growth.Source: §II · p. 3

A. Self-supervised Learning

Self-supervision learns from unlabeled inputs by optimizing a surrogate task with pseudolabels rather than strong class annotations. The paper applies this approach to sensor time series by contrasting raw inputs with wavelet-derived views.

A. Self-supervised Learning

  • Self-supervised learning uses pretext tasks and data-derived pseudolabels to train a network with a classification, regression or metric loss, without strong class annotations.Source: A. Self-supervised Learning · p. 3
  • Prior self-supervised methods include image, audio, video and text tasks, while the section notes little attention to sensor modalities such as EEG, IMUs and blood volume pulse.Source: A. Self-supervised Learning · p. 3
  • For sensor time series, the paper uses a contrastive objective between raw inputs and complementary views acquired through a wavelet transform.The approach is described as learning representations from edge sensor data.Source: A. Self-supervised Learning · pp. 3–4

B. Wavelet Transform

Wavelet transforms represent signals with time-localized frequency information, while federated learning trains from distributed client data and aggregates local models. In this setting, the paper addresses the difficulty of learning from decentralized sensor data when labeled examples are costly or unavailable.

B. Wavelet Transform

  • The wavelet transform provides time-localized information about signal variations, with larger windows trading improved frequency resolution for reduced time resolution.The passage notes that short-time Fourier transforms could also be used, potentially alongside segmentation.Source: B. Wavelet Transform · p. 4

C. Federated Learning

  • Federated learning aggregates locally trained models from remote devices to form a unified model while keeping the data distributed.Federated Averaging is identified as a typical method for accumulating client updates after local training.Source: C. Federated Learning · p. 4
  • Federated learning aggregates locally trained client models into a global model, but existing techniques assume labels are available or reliably obtainable.The paper frames self-supervised representation learning from unannotated, decentralized multi-sensor data as a step toward addressing this gap.Source: C. Federated Learning · pp. 4–5

III. APPROACH

The approach targets the difficulty of obtaining well-curated multi-sensor data by learning from raw signals and their corresponding scalograms. Its auxiliary task contrasts aligned and unaligned pairs to provide a self-supervised training objective.

III. APPROACH

  • The proposed auxiliary task distinguishes aligned raw-signal and scalogram pairs from unaligned pairs to learn multi-sensor representations.The motivation includes device diversity, environmental and interpersonal variation, privacy, and annotation cost.Source: §III · p. 5
  • The method uses a wavelet-derived scalogram as a complementary view of the raw signal for its correspondence task.The paper relates its rationale to cross-view learning in audio-visual work.Source: §III · p. 5
  • The approach generates training labels from signal-scalogram correspondence, allowing the network to train without human involvement.The paper describes this as an endless stream of labeled data in the absence of semantic labels.Source: §III · p. 5

A. Scalogram-signal Correspondence Learning

Scalogram-signal correspondence learning trains a multimodal network to match raw sensor segments with their wavelet-derived scalograms. Positive pairs are synchronized views; randomly mismatched scalograms provide negative pairs for contrastive learning.

A. Scalogram-signal Correspondence Learning

  • The method segments each time series into fixed-length windows with overlap and generates a scalogram for each segment using a specified wavelet transform.The resulting signal and scalogram representations are synchronized pairs.Source: A. Scalogram-signal Correspondence… · p. 5
  • Matching signal-scalogram pairs receive label 1, while randomly selected scalograms form label-0 negatives; training uses equal numbers of positive and negative instances.Negative scalograms may come from the same modality or, optionally, a different modality.Source: A. Scalogram-signal Correspondence… · p. 5
  • Scalograms are absolute squared continuous Morlet wavelet-transform values, with the mother wavelet's central frequency denoted by w0.Source: A. Scalogram-signal Correspondence… · p. 5
  • The contrastive objective pulls embeddings of positive pairs together and pushes different pairs apart, aligning complementary views and samples across sensory modalities.The authors say this label-free correspondence learning may yield representations invariant to sensor noise, amplitude or scale variations, and user-specific differences.Source: A. Scalogram-signal Correspondence… · p. 5

B. Network Architecture

The scalogram contrastive network has separate signal and scalogram feature pathways with modality-specific and fusion layers. Federated training dispatches models to clients, aggregates their updates, and uses the resulting global model for the end task.

B. Network Architecture

  • The network uses distinct signal and scalogram components, each with modality-specific layers and a fusion layer for joint embeddings.Only signal-network features are retained for evaluation; the scalogram network is discarded after pre-training.Source: B. Network Architecture · pp. 5–6
  • The scalogram pathway uses 2D convolutions and the signal pathway uses 1D convolutions, with additional pre-training layers discarded after self-supervised learning.The architecture passage specifies modality-specific and shared convolution layers for both pathways.Source: B. Network Architecture · p. 6
  • Federated training samples typically 10 clients per round for five local epochs, over 30–50 rounds, using randomly partitioned training data.The authors note that existing datasets have few users and that larger user pools could aid future evaluation; the server aggregates client models for the end task.Source: C. Implementation Details · p. 6, Figure 3 · p. 6

IV. EXPERIMENTS

The experiments assess representation quality, low-label learning, transfer, and robustness across five datasets spanning sleep, activity recognition, WiFi sensing, and physiological stress detection.

IV. EXPERIMENTS

  • Representations are evaluated with frozen-feature linear classification in central and federated settings, plus low-data, transfer, and cross-validation tests.The evaluation also examines robustness to subject variation through user-split cross-validation.Source: §IV · p. 6

A. Datasets and Preprocessing

  • The evaluation covers five datasets across sleep staging, human activity recognition, WiFi sensing, and physiological stress detection.The datasets include EEG/EOG, smartphone accelerometer and gyroscope, WiFi channel-state information, and wearable physiological signals.Source: A. Datasets and Preprocessing · p. 7
  • Sleep signals are segmented into 30-second epochs, while smartphone activity signals use 400-sample windows with 50% overlap.WiFi CSI is resampled from 1 kHz to 500 Hz across 90 channels; WESAD uses 30-second segments and per-subject normalization.Source: A. Datasets and Preprocessing · p. 7
  • Across all cases, the authors use a random 70%/30% dataset split based on users, with no user-data overlap; 20% of training data serves as validation for hyperparameter tuning and model selection.Source: A. Datasets and Preprocessing · p. 7

B. Quality assessment of the learned features with separability analysis

The feature assessment finds that SCN representations outperform autoencoder pretraining, with federated results close to centralized end-to-end models. Encoder features also outperform the network’s penultimate-layer projection for downstream classification.

B. Quality assessment of the learned features with separability analysis

  • Federated training uses random partitions to address the datasets’ low number of users, limiting evaluation of more heterogeneous non-IID settings.The authors identify large user pools with greater device variety as future work because labeled end-task data are needed to assess feature quality.Source: B. Quality assessment of the learned… · p. 7
  • On HHAR, SCN achieves F-score 82.7 versus 73.0 for the supervised baseline, and federated results are close to centralized end-to-end models.Across evaluated datasets, SCN surpasses autoencoder pretraining; with a federated classifier, its performance is mainly consistent with the centralized classifier.Source: B. Quality assessment of the learned… · p. 8, Table III · p. 8, Table II · p. 8
  • SCN’s t-SNE embedding shows distinct clusters in an unsupervised representation, with class colors added only afterward for interpretation.The embedding uses a random subset of test subjects, and t-SNE does not use class labels.Source: Figure 4 · p. 7, B. Quality assessment of the learned… · p. 8
  • The encoder output e yields significantly better downstream classification than the penultimate projection h(e).The authors suggest the final layer may learn auxiliary-task-specific features that are less useful for end tasks.Source: Figure 5 · p. 7, B. Quality assessment of the learned… · p. 8

C. Improving generalization in low-data regime and transfer as evaluation

The authors test SCN pretraining for few-label fine-tuning and transfer across activity datasets. Self-supervised pretraining outperforms the supervised baseline in the evaluated low-data settings, and transferred SCN features improve recognition relative to supervised representations.

C. Improving generalization in low-data regime and transfer as evaluation

  • Self-supervised pretraining yields better results than the supervised baseline across the evaluated few-label settings.SCN is pretrained on all unlabeled data, then fine-tuned using 5, 10, 20, or 40 labeled instances per class; the reported average F-scores use 100 independent repetitions.Source: C. Improving generalization in… · p. 8, Figure 6 · p. 9, Table IV · p. 9
  • Transferred SCN embeddings improve activity recognition relative to transferred supervised embeddings on both evaluated datasets.The evaluation uses HHAR and MobiAct; the reported comparison is qualitative in the supplied passage.Source: C. Improving generalization in… · p. 8, Table V · p. 9
  • With transferred SCN initialization and 5–40 labeled target examples per class, F-score improves by 2%–3% over the supervised network.The models are fine-tuned end-to-end using unlabeled data from a different source for pretraining.Source: C. Improving generalization in… · p. 8

D. Network robustness against subject variation with cross-validation

User-split cross-validation assesses robustness to changes in training subjects, using leave-one-subject-out for three datasets and other protocols for two. SCN outperforms autoencoder pretraining, with reported advantages over supervised and hand-designed baselines on selected datasets.

D. Network robustness against subject variation with cross-validation

  • Sleep-EDF, HHAR, and WESAD use leave-one-subject-out cross-validation; MobiAct and WiFi-CSI use 10-fold stratified cross-validation.The latter protocols are used because MobiAct has many users and WiFi-CSI lacks subject IDs.Source: D. Network robustness against subject… · p. 9
  • On Sleep-EDF, SCN achieves mean kappa 0.83 versus 0.77 for the supervised network and 0.76 reported in; on WESAD, F-score is 75.7 ± 0.13 versus 66.33 ± 0.36 for hand-designed features.The authors report that SCN also significantly outperforms autoencoder pretraining across the cross-validation evaluation.Source: D. Network robustness against subject… · p. 10, Table VII · p. 10
  • Direct comparisons with existing methods on the other datasets are not feasible because reported metrics and sensing modalities differ.The authors conclude that the results indicate self-supervised learning can be used for sensor modeling at scale and combined with active learning methods.Source: D. Network robustness against subject… · p. 10

V. CONCLUSIONS

The paper proposes wavelet-based self-supervision for learning representations from unlabeled multi-sensor data in federated settings. It reports improved low-data generalization and describes on-device learning and future evaluation directions.

V. CONCLUSIONS

  • Self-supervision improves generalization when only a few labeled instances are used to fine-tune the network for a downstream task.The paper also compares learned features with fully supervised networks and autoencoder pretraining in central and federated settings.Source: §V · p. 10
  • Wavelet-generated scalograms provide a complementary view for a scalogram-signal correspondence task whose labels can be extracted from unlabeled data.The approach is designed for federated training on widely distributed, unannotated data.Source: §V · p. 10
  • The method is evaluated on public datasets spanning EEG, blood volume pulse, and IMU streams, with feature quality assessed using a linear classifier on an end task.The evaluation compares against fully supervised networks and autoencoder pretraining in both central and federated settings.Source: §V · p. 10
  • The approach is presented as a way to learn general-purpose deep feature extractors on-device without transmitting the actual data to a server.The paper identifies larger datasets, architecture search, non-IID federated settings, and adversarial robustness as future research directions.Source: §V · p. 10
Loading 2007.13018v1…