PaperlayerView exact revision on arXiv

Source-linked AI summary

FedMultimodal: A Benchmark For Multimodal Federated Learning

Tiantian Feng, Digbalay Bose, Tuo Zhang, Rajat Hebbar, Anil Ramakrishna, Rahul Gupta, Mi Zhang, Salman Avestimehr, Shrikanth Narayanan

arXiv:2306.09486v2cs.DCcs.LG

TL;DR

Multimodal federated learning lacks benchmarks despite privacy-sensitive applications spanning several fields. FedMultimodal provides an end-to-end benchmark across five applications and ten datasets, and its evaluations find that label noise is more damaging than missing modalities while attention fusion often outperforms concatenation.

  • Problem

    Multimodal FL applications remain underexplored, and existing FL benchmarks do not target multimodal applications despite their real-world use and privacy concerns.

    Source: §1 · p. 1

  • Method

    FedMultimodal benchmarks ten datasets across five applications with an end-to-end FL pipeline and emulates missing modalities, missing labels, and erroneous labels.

    Source: §1 · p. 2

  • Results

    Label noise causes larger performance decreases than missing modalities, while attention fusion outperforms concatenation on most datasets and FedOpt generally gives better baselines.

    Source: §4.2 · p. 7, §4.6 · p. 9

  • Takeaways & Limitations

    The benchmark establishes reproducible results across multimodal FL applications and evaluates robustness to three forms of data corruption.

    Source: §6 · p. 9

    FedMultimodal does not yet cover applications such as medical image analysis, autonomous driving, and virtual reality, and its supported model range is limited.

    Source: §5 · p. 9

Abstract

from arXiv · show

Over the past few years, Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges through collaborative training. In the Federated Learning algorithm, the clients submit a locally trained model, and the server aggregates these parameters until convergence. Despite significant efforts that have been made to FL in fields like computer vision, audio, and natural language processing, the FL applications utilizing multimodal data streams remain largely unexplored. It is known that multimodal learning has broad real-world applications in emotion recognition, healthcare, multimedia, and social media, while user privacy persists as a critical concern. Specifically, there are no existing FL benchmarks targeting multimodal applications or related tasks. In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities. FedMultimodal offers a systematic FL pipeline, enabling end-to-end modeling framework ranging from data partition and feature extraction to FL benchmark algorithms and model evaluation. Unlike existing FL benchmarks, FedMultimodal provides a standardized approach to assess the robustness of FL against three common data corruptions in real-life multimodal applications: missing modalities, missing labels, and erroneous labels. We hope that FedMultimodal can accelerate numerous future research directions, including designing multimodal FL algorithms toward extreme data heterogeneity, robustness multimodal FL, and efficient multimodal FL. The datasets and benchmark results can be accessed at: https://github.com/usc-sail/fed-multimodal.

1 INTRODUCTION

Federated learning keeps raw user data on edge devices, addressing privacy concerns associated with centralized training, but multimodal FL lacks benchmarks for fair comparisons. FedMultimodal addresses this gap with a reproducible benchmark and framework that includes robustness assessments.

1 INTRODUCTION

  • Centralized training typically sends human-centered data to remote servers, raising privacy concerns because the data can reveal sensitive environments and contexts.In FL, clients train locally and send model updates rather than raw data.Source: §1 · pp. 1–2
  • Existing FL benchmarks span areas such as computer vision, NLP, audio, and healthcare, while multimodal FL studies use separate experimental setups that hinder fair comparison.The paper describes earlier multimodal FL evaluations as challenging to compare fairly and effectively with existing state-of-the-art methods.Source: §1 · p. 2
  • FedMultimodal covers five application scenarios with ten representative datasets and provides systematic results for fair algorithm comparisons.The scenarios are emotion recognition, multimodal action recognition, human activity recognition, healthcare, and social media.Source: §1 · p. 2
  • The open-source end-to-end simulation framework supports data partitioning, feature processing, multimodal training, and customization across popular FL optimizers.Supported optimizers include FedAvg [46], FedProx, FedRS, SCAFFOLD [35], and FedOpt [57].Source: §1 · p. 2
  • Its robustness module emulates missing modalities, missing labels, and erroneous labels, which prior multimodal FL work had assessed only to a limited extent.These corruptions are intended to simulate real-world deployment scenarios.Source: §1 · p. 2

2 MULTIMODAL DATASETS AND TASKS

FedMultimodal organizes ten publicly available multimodal datasets across five application areas, with task-specific datasets and selections described for each area.

2 MULTIMODAL DATASETS AND TASKS

  • The benchmark includes ten publicly available datasets across five tasks: emotion recognition, multimedia action recognition, human activity recognition, healthcare, and social media classification.The dataset overview is intended to support accessibility and reproducibility.Source: §2 · p. 2, Table 2 · p. 3

2.1 Emotion Recognition (ER)

  • Emotion recognition uses MELD and CREMA-D, with MELD represented by audio and transcripts and CREMA-D by audio-visual clips.MELD is restricted to its four most-sampled emotions; CREMA-D contains clips from 91 actors portraying six emotions.Source: §2.1 · p. 3

2.2 Multimodal Action Recognition (MAR)

  • Multimodal action recognition uses UCF101 and Moments in Time, with the latter divided into MiT10 and MiT51 label subsets.UCF101 experiments use videos with audio and video for 51 labels; MiT subsets use the 10 or 51 most frequent labels, with frames subsampled for computational constraints.Source: §2.2 · p. 3

2.3 Human Activity Recognition (HAR)

  • Human activity recognition uses accelerometer and gyroscope data as separate modalities in UCI-HAR and KU-HAR.KU-HAR experiments retain six activities shared with UCI-HAR and add jumping and running.Source: §2.3 · p. 3

2.4 Healthcare

  • The healthcare task is multi-label ECG diagnosis using PTB-XL, with electrode readings grouped into two modalities.The dataset contains over 20,000 clinical 12-lead recordings from 18,885 patients; ECG data are sampled at 100 Hz.Source: §2.4 · p. 3

2.5 Social Media (SM)

  • Social media tasks address hateful content and crisis information, using multimodal datasets with paired image-text or visual-textual content.Hateful Memes has 10,000 image-text pairs; CrisisMMD comprises 18.1k tweets with paired visual and textual information.Source: §2.5 · pp. 3–4

3 END-TO-END MULTIMODAL FEDERATED LEARNING FRAMEWORK

FedMultimodal provides an end-to-end multimodal federated-learning pipeline with components for data partitioning, feature processing, models, fusion, optimizers, and noise emulation. It evaluates robustness to missing modalities, missing labels, and erroneous labels.

3 END-TO-END MULTIMODAL FEDERATED LEARNING FRAMEWORK

  • The framework combines six components, including non-IID partitioning, feature processing, multimodal models, fusion, federated optimizers, and emulation of three real-world noise factors.The noise factors are missing modalities, missing labels, and erroneous labels.Source: §3 · p. 4

3.1 Non-IID Data Partitioning

  • Datasets with natural client identifiers use speaker, participant, or clinical-site partitions; others use Dirichlet splits with α=0.1 for high and α=5.0 for low heterogeneity.UCI-HAR participant data is further split using a Dirichlet distribution because its original label distribution is IID.Source: §3.1 · p. 4

3.2 Feature Processing

  • Feature processing uses pretrained backbones and mobile-friendly extraction, selecting MobileNetV2 for visual features, MobileBERT for text, MFCCs for audio, and raw data for other modalities.The other raw modalities include accelerometer, gyroscope, and ECG readings.Source: §3.2 · pp. 4–5

3.3 Multimodal Models

  • Models prioritize lightweight solutions for resource-constrained edge devices, using Conv+RNN or RNN-only encoders, late fusion, and two dense prediction layers.The Conv+RNN encoder handles audio, accelerometer, gyroscope, and ECG inputs; the RNN-only encoder is used otherwise.Source: §3.3 · p. 5

3.4 Fusion Schemes

  • Fusion compares pooled modality embeddings concatenated together with temporal outputs combined by lightweight attention, which can mask missing modalities.Table 4 benchmarks performance for concatenation-based and attention-based fusion.Source: §3.4 · pp. 5–6, Table 4 · p. 6

3.5 Federated Optimizers

  • The benchmark supports FedAvg, FedProx, FedRS, and FedOpt to evaluate federated optimizers whose multimodal-task efficacy remains unexplored.FedOpt is noted as having state-of-the-art performance across multiple unimodal applications.Source: §3.5 · p. 6

3.6 Real-world Noise Factor Emulator

  • Noise emulation includes client-level missing modalities, partially labeled data, and erroneous labels generated through a transition matrix.Each modality's availability follows a Bernoulli distribution, with an equal missing rate used in the experiments.Source: §3.6 · p. 6

4 EXPERIMENTS AND DISCUSSION

FedMultimodal benchmarks multimodal FL across fusion choices, modality configurations, and label corruption. Results show that performance depends on fusion and corruption type, with erroneous labels especially damaging and unimodal models often competitive.

4.1 Experimental Details

  • Video and text use RNNs while other modalities use Conv-RNNs; experiments use batch size 16, one local epoch, and five-seed repeats or five-fold cross-validation.Training runs for 200 epochs except MiT10 and MiT51, which run for 300; FedOpt server learning rates and FedProx proximal terms are searched over stated ranges.Source: §4.1 · pp. 6–7

4.3 Uni-modality vs. Multi-modalities

  • Multimodal learning outperforms unimodal learning in most scenarios, while the performance gap is within 5% on most datasets.Figure 3 compares these approaches on datasets with natural or high-heterogeneity partitions.Source: Figure 3 · p. 7, §4.3 · pp. 7–8

4.4 Impact of Missing Modalities

  • Below 30% missing modalities, performance changes are not substantial; at 50%, declines become substantial, although half of models decrease by under 10%.Figure 4 presents relative changes across missing rates; the reported impact is larger for CrisisMMD, and the authors flag missing modalities for future HAR evaluation.Source: Figure 4 · p. 7, §4.4 · p. 8

4.2 Overall Performance

  • Attention-based fusion outperforms concatenation on most datasets, particularly under high data heterogeneity.It underperforms on two synthetic datasets at α=5.0; FedOpt generally gives stronger baselines but needs additional server learning-rate tuning.Source: §4.2 · p. 7
  • The benchmark reports the highest performance scores on HAR tasks and below-30% best-model performance on CrisisMMD.The authors suggest social-media features from pretrained models may be unrepresentative; MiT51 is also challenging, while reducing labels in MiT10 yields moderate performance.Source: §4.2 · p. 7

4.5 Impact of Missing Labels

  • Missing labels cause less performance decline than missing modalities, with decreases under 10% on most datasets except KU-HAR.Below a 50% missing-label ratio, decreases are minor across datasets; CrisisMMD performs worse at 30% than at 10%, which the authors conjecture may reflect labeling subjectivity.Source: Figure 5 · p. 8, §4.5 · p. 8

4.6 Impact of Erroneous Labels

  • At 30% erroneous labels, over half of datasets decline by more than 10%; at 50%, declines reach 20% in those datasets.Figure 7 compares the three corruption types at 30%, where multimodal FL is more susceptible to label noise than to missing modalities or labels.Source: Figure 6 · p. 8, Figure 7 · p. 9, §4.6 · p. 9

5 LIMITATIONS AND FUTURE WORK

FedMultimodal’s scope and fusion options are limited, while its benchmark is intended to support further work on multimodal heterogeneity, label scarcity, and privacy risks.

Scale of Datasets and Models

  • The selected datasets span varied scenarios, sizes, and client counts, but medical image analysis, autonomous driving, and virtual reality are not covered, and supported models remain limited.The included models are selected with FL use cases and edge-device computational limits in mind.Source: §5 · p. 9

Scale of Modality Fusion Schemes

  • FedMultimodal currently supports concatenation and attention for modality fusion, which remains an open problem in FL.The authors call for more advanced fusion schemes.Source: §5 · p. 9

Data Heterogeneity

  • Multimodal FL lacks extensive research on data heterogeneity; the benchmark offers opportunities to study it and future work may explore knowledge-transfer approaches.The cited knowledge-transfer approaches are proposed as a future direction within multimodal FL.Source: §5 · p. 9

Label Scarcity

  • FedMultimodal can emulate missing labels, supporting experiments on label scarcity and study of self-supervised and semi-supervised learning in multimodal FL.The authors identify these learning approaches as research directions enabled by missing-label experiments.Source: §5 · p. 9

Privacy Leakage

  • Although sharing model updates is more private than sharing raw data, FL remains susceptible to membership, reconstruction, attribute, and label inference attacks.The authors identify multimodal privacy leakage and techniques such as differential privacy and secure aggregation as future research directions.Source: §5 · p. 9

6 CONCLUSION

The paper presents FedMultimodal as a framework and benchmark for multimodal federated learning. It reports results across five applications and ten datasets, including robustness benchmarks for three data corruptions.

6 CONCLUSION

  • FedMultimodal is a new framework for enabling federated learning in multimodal applications.Source: §6 · p. 9
  • The paper establishes reproducible benchmark results for five multimodal FL applications covering ten datasets.Source: §6 · p. 9
  • The benchmark assesses model robustness to missing modalities, missing labels, and noisy labels across each task.Source: §6 · p. 9
Loading 2306.09486v2…