PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Federated Learning on IoT Data

Yuchen Zhao, Payam Barnaghi, Hamed Haddadi

arXiv:2109.04833v2cs.LG

TL;DR

The paper addresses modality heterogeneity and scarce client labels in federated learning with a multimodal, semi-supervised framework that aggregates client autoencoders. Its experiments report improved classification with multiple modalities and decent cross-modal F1 scores.

  • Problem

    Federated learning faces modality heterogeneity across clients, while real-world clients may lack labels for local training.

    Source: A. Federated learning · p. 2, A. Heterogeneity beyond data… · p. 10

  • Method

    The framework uses client autoencoders to learn shared or correlated multimodal representations and multimodal FedAvg to aggregate them, with server auxiliary labels for classification.

    Source: §I · p. 2

  • Results

    Introducing data from more modalities into federated learning improves classification performance, and classifiers trained with server labels from one modality achieve decent F1 scores on other modalities.

    Source: §I · p. 2

  • Takeaways & Limitations

    Only part of the clients need to be multimodal for aligning modalities, supporting use in federated systems with varied device setups.

    Source: §VII · p. 11

    The framework currently assumes all multimodal clients use the same autoencoder type, an assumption that may need revision for differing data distributions or computational capabilities.

    Source: C. Utilizing mixed FL clients · p. 11

Abstract

from arXiv · show

Federated learning is proposed as an alternative to centralized machine learning since its client-server structure provides better privacy protection and scalability in real-world applications. In many applications, such as smart homes with Internet-of-Things (IoT) devices, local data on clients are generated from different modalities such as sensory, visual, and audio data. Existing federated learning systems only work on local data from a single modality, which limits the scalability of the systems. In this paper, we propose a multimodal and semi-supervised federated learning framework that trains autoencoders to extract shared or correlated representations from different local data modalities on clients. In addition, we propose a multimodal FedAvg algorithm to aggregate local autoencoders trained on different data modalities. We use the learned global autoencoder for a downstream classification task with the help of auxiliary labelled data on the server. We empirically evaluate our framework on different modalities including sensory data, depth camera videos, and RGB camera videos. Our experimental results demonstrate that introducing data from multiple modalities into federated learning can improve its classification performance. In addition, we can use labelled data from only one modality for supervised learning on the server and apply the learned model to testing data from other modalities to achieve decent F1 scores (e.g., with the best performance being higher than 60%), especially when combining contributions from both unimodal clients and multimodal clients.

I. INTRODUCTION

Federated learning protects local data, but IoT clients may have different modalities and device setups that existing approaches do not accommodate well. The paper proposes multimodal federated learning for mixed client types and reports improved classification performance, including cross-modality testing with labels from one modality.

I. INTRODUCTION

  • IoT clients may have sensory, visual, or audio data and differ in whether they hold one modality or several, motivating federated learning that supports varied device setups.Separate services for each modality do not capture the potential benefits of combining modalities shown in centralized machine learning.Source: §I · p. 1
  • Existing multimodal federated approaches either require aligned data and labels across modalities or upload local representations that may expose private data.These constraints leave federated learning with arbitrary client modalities and labels from only one modality as an open challenge.Source: §I · p. 1
  • The proposed framework uses multimodal autoencoders to extract shared or correlated representations and multimodal FedAvg to aggregate models from unimodal and multimodal clients.The framework uses alignment information available in multimodal clients’ local data.Source: §I · p. 2
  • With unlabelled client data, the server uses an auxiliary labelled dataset to train a classifier from representations produced by the global autoencoder.The reported setup uses labelled data from one modality for supervised learning and applies the model to testing data from other modalities.Source: §I · p. 2
  • Introducing data from more modalities into federated learning improves classification performance, and combining unimodal and multimodal clients further improves F1 scores.The framework also supports applying models trained with server labels from one modality to testing data from other modalities.Source: §I · p. 2

II. RELATED WORK

The related work reviews federated learning under limited client labels and heterogeneous clients, then connects multimodal representation learning to the challenge of differing data modalities. It frames the paper’s methodology as addressing that modality heterogeneity.

A. Federated learning

  • Canonical federated learning typically assumes labelled client data, while IoT users may lack time or expertise to annotate it; prior work addresses this with pseudo-labeling or representations learned from unlabelled data.This paper follows the representation-learning direction and extends it to multiple modalities.Source: A. Federated learning · p. 2

B. Heterogeneity in federated learning

  • Prior federated-learning research mainly addresses distributional or model heterogeneity, while modality heterogeneity remains under-addressed.Existing multimodal approaches either upload local representations, risking privacy, or require aligned multimodal data for clients, server labels, and testing.Source: B. Heterogeneity in federated learning · pp. 2–3

C. Multimodal deep learning

  • Multimodal deep learning uses representations that connect modalities, including autoencoders and methods that map modalities to highly correlated representations.Prior applications suggest modalities can complement one another in representation learning; this paper applies that approach to federated learning.Source: C. Multimodal deep learning · p. 3

III. METHODOLOGY

  • The methodology introduces the framework design, techniques for extracting representations from multimodal data, and aggregation of models trained on unimodal and multimodal clients.The overview positions these components as a response to differing client modalities.Source: §III · p. 3

A. Framework overview

The framework accommodates unlabelled client data from either one modality or multiple modalities, using local alignment on multimodal clients to connect representations. A global model then supports supervised classifier training with an auxiliary labelled dataset on the server.

A. Framework overview

  • The framework supports clients with either one modality or multiple modalities, assuming that multimodal clients have alignment information between their local data streams.For example, activity can be recorded at the same time by a smartwatch accelerometer and an IP camera.Source: A. Framework overview · p. 3
  • Clients learn representations from unlabelled data: multimodal clients learn shared or correlated representations, while unimodal clients learn representations from one modality.The server aggregates both types of local models with multimodal FedAvg into a global model.Source: A. Framework overview · p. 3
  • The server encodes an auxiliary labelled dataset from either modality with the global model, then trains a classifier on the resulting labelled representations.Clients receive the global model and classifier for local classification.Source: A. Framework overview · p. 3

B. Learning to extract representations

The section introduces autoencoders as a way to extract hidden representations from unimodal or multimodal data. These representations can be used with labels to train a downstream classifier.

B. Learning to extract representations

  • The framework introduces canonical autoencoders for unimodal data and multimodal autoencoders for shared or correlated hidden representations.The global autoencoder’s hidden representation is used with labels to train a classifier.Source: B. Learning to extract representations · p. 4
  • An autoencoder maps input data X to a hidden representation h, then decodes h into a reconstruction X′.The encoder is f and the decoder is g.Source: Figure 2 · p. 4

1) Autoencoders:

The section describes autoencoders and two multimodal variants that learn representations from aligned data. Split autoencoders learn shared representations, while canonically correlated autoencoders retain separate representations and maximize their correlation.

1) Autoencoders:

  • An autoencoder encodes unlabelled input X into h and decodes h into reconstruction X′, with training minimizing the reconstruction error.The stated assumption is that a small reconstruction error means h contains useful information from the input.Source: 1) Autoencoders · pp. 4–5

2) Split autoencoders:

  • For matching modality pairs, a SplitAE uses one modality’s encoder to produce h and two decoders to reconstruct both modalities.Its compound objective minimizes the sum of the two modality-specific losses.Source: 2) Split autoencoders · p. 5, Figure 3 · p. 5

3) Deep canonically correlated autoencoders:

C. Multimodal federated averaging

Multimodal FedAvg combines modality-specific updates from unimodal and multimodal clients. Local training updates only the encoder-decoder pairs corresponding to each client’s available modalities.

C. Multimodal federated averaging

  • A multimodal client updates encoder-decoder pairs for both modalities, while a unimodal client updates its own pair and freezes the other.Clients may be selected whether they are unimodal or multimodal.Source: C. Multimodal federated averaging · pp. 5–6, Figure 4 · p. 6
  • Multimodal FedAvg aggregates autoencoders from both unimodal and multimodal clients, using only their updated model parts.The global multimodal autoencoder is formed from local model contributions.Source: C. Multimodal federated averaging · p. 6, Figure 5 · p. 6
  • The weight α controls multimodal clients’ contribution; increasing it gives their updates more weight because they help align modalities.The passage states that this alignment helps unimodal clients benefit from data in another modality.Source: C. Multimodal federated averaging · p. 6

IV. EVALUATION

The evaluation uses simulations on datasets spanning sensory, depth-camera, and RGB-camera data. Its stated questions concern multimodal gains, cross-modality classification, and the contribution of combining unimodal and multimodal clients.

IV. EVALUATION

  • The evaluation uses simulations on multimodal datasets that include sensory, depth-camera, and RGB-camera data.The passages state the evaluation scope but do not report results.Source: §IV · p. 6
  • One research question asks whether introducing multiple modalities into federated learning improves performance.This is posed as an evaluation question, not a reported finding.Source: §IV · p. 6
  • The evaluation also asks whether a classifier trained with labels from one modality works on testing data from other modalities.A further question compares learning from both unimodal and multimodal clients against learning only from multimodal clients.Source: §IV · p. 6

A. Datasets

The evaluation uses three human-activity-recognition datasets; the supplied dataset details describe Opportunity and mHealth sensor modalities and their train/test sampling protocols.

A. Datasets

  • Three human activity recognition datasets were used because HAR often relies on multimodal data.Source: A. Datasets · p. 6

1) Different sensory modalities:

  • Opportunity includes 18 short-term activities; experiments use accelerometer and gyroscope data, with 118k test samples and 525k training samples.Clients receive randomly sampled sequences sized at 1/15 of the training data.Source: 1) Different sensory modalities · pp. 6–7
  • mHealth experiments use accelerometer, gyroscope, and magnetometer data, testing each pair with leave-one-subject-out splits.One participant is held out for testing; the other nine train, and a client's sampled sequence is 1/9 of the training data.Source: 1) Different sensory modalities · p. 7

2) Sensory-Visual modalities:

The experiments pair RGB, depth, or accelerometer data with simulation settings and client baselines. MmFL converges faster than UmFL across the three datasets and is no worse on UR Fall.

2) Sensory-Visual modalities:

  • UR Fall provides 70 RGB-and-depth video clips paired with accelerometer data; RGB frames become ResNet-18 features, while supplied features represent depth.Testing uses 1/10 of the data; each client's sampled sequence is 1/9 of the training data.Source: 2) Sensory-Visual modalities · p. 7

B. Simulation setup

  • Each replicate runs up to 100 communication rounds, selecting 10% of clients for local training before five cloud-training epochs.Local training uses two epochs; the server's labelled dataset matches one client's local-data size, and experiments use 64 replicates.Source: B. Simulation setup · p. 7

1) Baselines:

  • For Q1, 30 multimodal clients with two labelled modalities are compared with 30 unimodal clients and one labelled modality on the UmFL test modality.Source: 1) Baselines · p. 7
  • For Q2, Abl trains unimodal autoencoders without aligned multimodal local training; it is compared with 30 multimodal clients using one labelled modality.Source: 1) Baselines · p. 7
  • For Q3, the study compares multimodal-only updates with updates that also use unimodal clients to assess Mm-FedAvg.Source: 1) Baselines · p. 7
  • On UR Fall, MmFL has higher or equal converged F1 than UmFL; it converges faster on all three datasets.Source: Figure 6 · p. 8

2) Models:

  • Local time-series autoencoders use LSTMs, while server-side supervised learning uses an MLP followed by LogSoftmax.The mHealth classifier adds a dropout layer with rate 0.5 before the MLP.Source: 2) Models · p. 8

C. Metrics

Across the reported datasets, multimodal learning improves F1 or convergence speed relative to unimodal approaches. The evaluation aggregates class-weighted sequence F1 scores across repeated simulations.

C. Metrics

  • Classifier F1 is the average across sequences of each sequence's class-weighted F1, evaluated every other communication round and averaged over 64 replicates.The better-performing SplitAE or DCCAE is retained for each dataset.Source: C. Metrics · p. 8

V. RESULTS

  • Using multiple modalities yields higher classifier F1 than using one modality, and MmFL improves F1 or convergence speed over baselines in most cases.Source: §V · p. 8, Figure 7 · p. 9

A. Multimodal data improve F1 scores

  • Opp and mHealth show similar converged F1 but faster MmFL convergence; on UR Fall, MmFL raises F1 for Acce or RGB tests and converges faster for Depth.UR Fall's sensory-visual modality pairing is more heterogeneous than the sensory-sensory pairings in Opp and mHealth, yet the representations can still be aligned.Source: A. Multimodal data improve F1 scores · pp. 8–9
  • The framework can use separately collected unimodal server labels rather than requiring aligned multimodal labelled data, unlike the cited early-fusion approach [14].The authors suggest local alignment information can support scaling FL across modalities.Source: A. Multimodal data improve F1 scores · p. 9

B. Labels can be used across modalities

MmFL transfers label information from a labelled modality to other modalities through shared or related representations. Mixed-client training can further improve F1 or convergence compared with using only multimodal clients.

B. Labels can be used across modalities

  • With one labelled modality, MmFL achieves higher F1 than baselines across the reported datasets, and cross-modality test scores can approach or exceed unimodal scores.On Opp, Gyro labels yield Acce-test F1 close to the Acce-labelled unimodal scheme; on UR Fall, Depth labels yield higher Acce and RGB test F1 than those modalities' own labels.Source: B. Labels can be used across modalities · p. 10

C. Training on mixed clients

  • Mixed-client schemes improve F1 slightly on two mHealth experiments and on UR Fall Acce–Depth, while Opp shows faster convergence.The results support using unimodal clients when multimodal clients are limited, with greater weights for multimodal contributions.Source: C. Training on mixed clients · p. 10

VI. DISCUSSIONS

The discussion frames modality heterogeneity as important for scaling FL across varied IoT clients, and describes sharing labels across modalities and combining unimodal with multimodal clients. The conclusion reports improved classification F1 scores from multiple modalities and cross-modal testing using labels from one modality.

A. Heterogeneity beyond data distributions

  • The paper focuses on modality heterogeneity; other forms, including data-distribution and DNN-structure heterogeneity, remain outside its scope and are planned for future study.The discussion notes that addressing modality heterogeneity can make FL scalable across modalities and increase available data.Source: A. Heterogeneity beyond data… · pp. 10–11

B. Sharing label information across modalities

  • Label information from one modality can serve other modalities, potentially reducing server annotation costs and reliance on privacy-intrusive devices such as cameras.The discussion gives sensory data as a modality that may be difficult to annotate directly and visual data as easier to annotate.Source: B. Sharing label information across… · p. 11

C. Utilizing mixed FL clients

  • Mm-FedAvg combines updates from unimodal and multimodal clients, and mixed contributions with greater weight for multimodal clients achieve higher F1 scores than multimodal clients alone.The current framework assumes multimodal clients use the same autoencoder type; the authors suggest investigating more flexible averaging and dynamic weighting.Source: C. Utilizing mixed FL clients · p. 11

VII. CONCLUSIONS

  • The proposed multimodal and semi-supervised framework supports clients with different local modalities and device setups.The authors plan to investigate applications beyond multimodal human activity recognition.Source: §VII · p. 11
  • Multiple modalities improve FL classification F1 scores, and models trained with labels from one modality achieve decent F1 scores on testing data from other modalities.Only part of the clients need to be multimodal to align modalities.Source: §VII · p. 11

Related papers

Loading 2109.04833v2…