Source-linked AI summary
Multimodal Federated Learning via Contrastive Representation Ensemble
Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, Jingjing Liu
TL;DR
Existing multimodal federated learning methods restrict server and client architectures through single-modality aggregation, limiting heterogeneous data use under privacy constraints. CreamFL transfers public-data representations, aggregates them with global-local cross-modal ensembling, and uses inter- and intra-modal contrasts to address modality and task gaps; it outperforms competing FL methods on reported image-text retrieval results.
Problem
Existing multimodal federated learning methods aggregate at the single-modality level, restricting server and client architectures despite growing heterogeneous private multimodal data needs.
Method
CreamFL transfers representations of public data, applies global-local cross-modal representation aggregation, and uses inter- and intra-modal contrastive regularization for heterogeneous clients.
Results
CreamFL outperforms FedAvg, FedIoT, reamFL+IoT, FedMD, FedET, and FedGEMS on the reported R@1 sum score, reaching 132.88%.
Takeaways & Limitations
CreamFL supports larger server models with heterogeneous client modalities and architectures while communicating knowledge on public data without revealing private data or models.
Takeaways & Limitations
Clients are assumed to use small local models that output representations matching the global model's representation dimension.
Abstract
from arXiv · showhide
With the increasing amount of multimedia data on modern mobile systems and IoT infrastructures, harnessing these rich multimodal data without breaching user privacy becomes a critical issue. Federated learning (FL) serves as a privacy-conscious alternative to centralized machine learning. However, existing FL methods extended to multimodal data all rely on model aggregation on single modality level, which restrains the server and clients to have identical model architecture for each modality. This limits the global model in terms of both model complexity and data capacity, not to mention task diversity. In this work, we propose Contrastive Representation Ensemble and Aggregation for Multimodal FL (CreamFL), a multimodal federated learning framework that enables training larger server models from clients with heterogeneous model architectures and data modalities, while only communicating knowledge on public dataset. To achieve better multimodal representation fusion, we design a global-local cross-modal ensemble strategy to aggregate client representations. To mitigate local model drift caused by two unprecedented heterogeneous factors stemming from multimodal discrepancy (modality gap and task gap), we further propose two inter-modal and intra-modal contrasts to regularize local training, which complements information of the absent modality for uni-modal clients and regularizes local clients to head towards global consensus. Thorough evaluations and ablation studies on image-text retrieval and visual question answering tasks showcase the superiority of CreamFL over state-of-the-art FL methods and its practical value.
1 INTRODUCTION
Multimodal federated learning must support privacy-preserving collaboration across heterogeneous modalities, models, and tasks. CreamFL addresses these constraints by transferring and aggregating public-data representations while regularizing modality and task gaps.
- Motivation: Multimodal FL must absorb heterogeneous private data while supporting larger models and diverse multimodal tasks at the edge.Existing FL commonly assumes clients’ private data share one modality, limiting multimodal collaboration.
- Limitations: Existing multimodal FL methods use homogeneous modality-specific models, constraining global-model capacity and worsening client model drift.Naive modality-dependent aggregation does not adequately address drift between clients.
- Limitations: Logit-based ensemble knowledge transfer is difficult to extend to multimodal tasks that operate on fused cross-modal representations.CreamFL instead communicates representations, which are described as contextual and applicable to more complex tasks than logits.
- CreamFL: CreamFL trains a larger server model from heterogeneous uni-modal and multimodal clients by communicating private knowledge through public data.The framework uses representation-level ensemble knowledge transfer without revealing private models or data.
- CreamFL: CreamFL’s global-local cross-modal contrastive aggregation filters drifting representations and selects candidates that better match paired representations across modalities.The strategy contrasts local representations with global representations and with representations from another modality.
- CreamFL: Inter-modal and intra-modal contrasts address modality and task gaps by supplementing absent-modality information and steering local models toward global consensus.Uni-modal clients gain cross-modality information, while intra-modal regularization addresses divergent task targets.
2 RELATED WORK
Related work applies knowledge distillation and federated aggregation to heterogeneous or multimodal settings, but prior approaches lack CreamFL’s representation-level, cross-modal selection strategy.
- Federated knowledge distillation: Some federated knowledge-distillation methods support heterogeneous clients or larger server models, but differ in whether they train a server model or select client contributions.FedGKT does not select among clients, while other methods use ensemble transfer.
- Federated knowledge distillation: Prior larger-server approaches selectively aggregate logits using statistics such as variance or entropy, which are not directly suited to fused multimodal representations.Multimodal tasks often operate at the fused cross-modality representation level.
- Multimodal federated learning: Earlier multimodal FL methods extend FedAvg, combine datasets without a communication phase, or assign manually tuned higher weights to multimodal clients.The weighting strategy only applies when both uni-modal and multimodal clients are present.
- Inter-modal contrastive learning: Inter-modal contrastive learning commonly treats aligned image-text pairs as positives and unaligned pairs as negatives in multimodal self-supervised learning.CreamFL repurposes contrastive learning to evaluate representation quality across heterogeneous clients and regularize local training.
- Inter-modal contrastive learning: CreamFL differs by using contrast as a representation-quality metric and as regularization to bridge modality gaps and restrict global-local discrepancy.These uses target heterogeneity between federated clients rather than only multimodal pretraining.
3 FEDERATED MULTIMODAL LEARNING
CreamFL trains a larger global multimodal model from heterogeneous clients by exchanging public-data representations rather than private models or data. Inter- and intra-modal contrasts regularize local training, while selective representation aggregation addresses modality, task, and local-model drift.
- Framework overview: CreamFL supports uni-modal and multimodal clients with heterogeneous architectures by transferring representations of public data to train a larger global model.Clients may choose small models based on their local data, task, and resources, provided their outputs share the global representation dimension.
- Server aggregation: After regularized local training, clients transmit modality-specific public-data representations, which the server selectively aggregates before distilling them into the server model.The server can filter out representations that drift too far and favor representations that align with paired data, improving multimodal representation learning.
- Local regularization: Inter-modal contrast uses paired global representations from another modality to guide uni-modal clients toward a shared multimodal representation space.For image clients, paired global text representations encourage alignment with the corresponding image while separating unrelated text representations.
- Heterogeneity challenges: The framework addresses modality gap from absent modalities and task gap from clients pursuing different objectives, both of which can produce incompatible local representations.Examples include image-only clients lacking text information and image classification clients training toward targets different from cross-modal retrieval clients.
- Local regularization: Intra-modal contrast guides local representations toward global consensus while contrasting them with representations from the previous local model to mitigate task-induced drift.Unlike MOON, this contrast uses public data as a bridge rather than operating on private data.
- Server aggregation: Representation aggregation must evaluate representation quality because client representations may be biased or maliciously attacked.The framework therefore focuses on selecting which representations contribute more during ensemble transfer.
4 EXPERIMENTS
CreamFL is evaluated across heterogeneous multimodal federated-learning clients on retrieval and VQA, with ablations examining aggregation, regularization, communication, and model drift. It consistently outperforms the compared baselines while improving communication efficiency and reducing representation drift.
- Main Results: CreamFL improves image-text retrieval over all baselines across both retrieval directions and all evaluated settings.The evaluation covers Recall@1, Recall@5, and Recall@10 on 1K and 5K MS-COCO test sets.
- Main Results: 132.88% R@1 sum exceeds FedAvg at 114.03% and FedIoT at 111.19%, while also improving communication efficiency.The comparison evaluates both model performance and communication efficiency.
- Main Results: 132.88% R@1 sum exceeds FedMD, FedET, and FedGEMS at 128.33%, 129.11%, and 129.70%, respectively.These baselines include aggregation and local regularization components.
- Main Results: CreamFL reaches 62.12% VQA accuracy versus 60.23% for the best baseline, a 1.89% gain.The result is reported for the visual question answering task.
- Model Drift: CreamFL pulls representations from different clients closer, mitigating the larger drift between multimodal and unimodal clients caused by modality and task gaps.The visualization compares one multimodal client with two image clients and reports smaller within-modality drift than cross-modality drift in the vanilla case.
5 CONCLUSION
CreamFL enables larger server models and heterogeneous clients in multimodal federated learning without disclosing private data or models. It transmits representations and combines cross-modal aggregation with contrastive regularization to mitigate model drift.
- CreamFL supports larger server models and heterogeneous clients while communicating knowledge on public data without disclosing private data or models.
- The framework transmits representations between servers and clients rather than relying on modality-specific model aggregation.
- A global-local cross-modal ensemble strategy and inter- and intra-modal contrastive regularization address representation aggregation and local model drift.
A.1 REGARDING THE USE OF PUBLIC DATA
The framework uses public data for knowledge transfer, but real-world deployment requires attention to legal, copyright, permission, compliance, and user-awareness requirements.
- Legal and copyright issues surrounding public data must be carefully considered in real-world applications.
- Commercial use of publicly released data requires obtaining permission.
- Companies may collect their own public data, while applications should address compliance and user awareness through opt-in participation.
A.2 DATASETS
The evaluation uses public multimodal data from COCO and private image, text, and multimodal datasets to represent different client modalities.
- A random subset of COCO containing 50,000 image-text pairs serves as the public multimodal dataset.
- CIFAR100, AG NEWS, and Flicker30k serve as private datasets for image, text, and multimodal clients, respectively.
- CIFAR-100 contains 50,000 colored training images across 100 classes, with 500 images per class.
- AG NEWS contains 120,000 training sentences from four classes.
A.3 DISCUSSIONS ABOUT REPRESENTATION ENSEMBLE DISTILLATION
Representation ensemble distillation remains under-explored because heterogeneous clients can provide representations of varying quality. This challenge becomes more important as federated learning moves toward large server models trained from diverse data and architectures.
- Representation ensemble distillation is under-explored, and prior work trains students to mimic transformed teacher representations without actively selecting teachers.
- Active teacher selection is important in federated learning because heterogeneous clients have different data distributions and representation quality.
- Representation aggregation is especially important for federated learning with foundation models, whose servers must learn universal representations from diverse data and architectures.
A.4 LARGER DATA CAPACITY
CreamFL remains effective when evaluated with 100,000 public image-text pairs, while communication cost increases linearly with the public-data size.
- 100,000 public image-text pairs test CreamFL under larger data capacity.The pairs are a random subset of MSCOCO.
- CreamFL remains effective in the larger-scale setup.
- Communication cost increases linearly with the number of public data samples.Improving performance with fewer public samples is identified as an important future direction.
A.5 TRADE-OFF BETWEEN COMMUNICATION AND PERFORMANCE
CreamFL offers a better communication–performance trade-off than the compared frameworks, while its visualizations show reduced model drift in multimodal representations.
- CreamFL exhibits a better communication–performance trade-off than the compared frameworks across server-model architectures.The comparison varies the server architecture from ResNet101 to ResNet50 and ResNet34.
- Text clients exhibit model drift, although the drift between modality-identical text clients is smaller than their gap with multimodal clients.
- CreamFL reduces the image-representation drift observed between multimodal clients.In the visualization, previously separated representations are pulled together under CreamFL.
- Federated learning supports collaborative model training without compromising privacy, a concern emphasized for mobile and IoT systems.