PaperlayerView exact revision on arXiv

Source-linked AI summary

Prototype-guided Bilateral Alignment Multimodal Federated Learning

Tianchi Liao Tianchi_Liao, Lele Fu, Sheng Huang, Qing Hu, Hong-Ning Dai, Chuan Chen

arXiv:2609.38925v1cs.AIcs.LG

TL;DR

MFL methods often assume homogeneous models and balanced modality distributions, despite heterogeneous architectures and missing modalities in practical settings. MFedPBA combines feature-space alignment with entropy-weighted aggregation of logit prototypes. The paper reports improved performance across diverse modality settings and benchmarks.

  • Problem

    Existing MFL methods often assume homogeneous client architectures and balanced modality distributions, while heterogeneous encoders and missing modalities complicate knowledge transfer.

    Source: §1 · p. 1

  • Method

    MFedPBA aligns heterogeneous feature spaces with contrastive learning and GW distance, and aggregates naturally aligned logit prototypes using entropy-based weights.

    Source: §1 · p. 2

  • Results

    MFedPBA consistently outperforms state-of-the-art baselines across multimodal benchmarks and diverse modality-distribution settings, and mitigates negative transfer under imbalanced and missing-modality settings.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The authors conclude that bilateral alignment supports knowledge transfer across heterogeneous feature spaces under imbalanced and missing-modality settings.

    Source: §7 · p. 9

    The analysis formalizes clients with modality subsets and inconsistent data distributions, bounding the stated setting to heterogeneous multimodal federated learning.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Multimodal federated learning (MFL) has emerged as a pivotal paradigm for leveraging distributed data to enhance model performance. However, existing methods predominantly rely on idealized assumptions of model homogeneity and balanced modality distributions, rendering them ill-suited for practical scenarios characterized by heterogeneous client architectures and severe modality imbalance. To address these challenges, we propose a \textbf{M}ultimodal \textbf{Fed}erated learning Prototype-guided Bilateral Alignment (MFedPBA) framework. MFedPBA facilitates robust knowledge synergy through a dual alignment mechanism: (i) at the feature level, it aligns heterogeneous feature spaces via a projection encoder optimized by contrastive learning and the Gromov-Wasserstein distance; (ii) at the decision level, it employs an entropy-weighted aggregation of naturally aligned logit prototypes. This novel design achieves robust MFL by jointly tackling heterogeneous feature spaces and collectively aggregating decisions. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines under conditions of model heterogeneity and modality imbalance.

1. Introduction

MFedPBA targets heterogeneous client architectures and imbalanced modality availability with feature- and decision-level alignment. The paper reports superior performance across diverse modality settings and provides non-convex convergence guarantees.

1. Introduction

  • Heterogeneous encoders can place same-class prototypes in incompatible spaces, so direct server aggregation may move global prototypes toward class boundaries and contaminate knowledge transfer.Such aggregation can reduce global-model performance below local-only training.Source: §1 · p. 1
  • Existing MFL often assumes balanced modalities, while practical clients may lack modalities or have skewed data, increasing information heterogeneity beyond bimodal or shared-modality settings.The paper frames this as a second alignment challenge alongside heterogeneous architectures.Source: §1 · pp. 1–2
  • MFedPBA aligns heterogeneous feature spaces using contrastive learning and GW distance, then aggregates naturally aligned logit prototypes with entropy-based weights.The framework preserves heterogeneous local architectures while combining feature- and decision-level knowledge.Source: §1 · p. 2
  • The paper frames MFedPBA as addressing realistic model heterogeneity and modality imbalance, and reports theoretical convergence guarantees in non-convex settings.The stated setting departs from conventional homogeneous assumptions.Source: §1 · p. 2
  • MFedPBA consistently outperforms state-of-the-art baselines across multimodal benchmarks and diverse modality-distribution settings.The introduction identifies experiments on multiple multimodal benchmarks.Source: §1 · p. 2

2. Related Work

Prior MFL work considers unimodal or missing-modality clients, but the related-work discussion identifies unresolved heterogeneity and scalability challenges. The paper positions learnable server prototypes and multimodal joint alignment as responses to those gaps.

2.1. Multimodal Federated Learning

  • Existing MFL work commonly studies unimodal clients or clients with missing modalities, while real deployments may require distinct modality-specific feature extractors.FedMBridge uses a topology-aware hypernetwork but incurs communication overhead that scales poorly with model complexity and modality count.Source: §2.1 · pp. 2–3
  • The paper characterizes prior methods as failing to address the combined challenges of inter-node modality imbalance and model heterogeneity, and proposes communication-efficient multimodal joint alignment.The cited scalability concern grows with model complexity and the number of modalities.Source: §2.1 · p. 3

2.2. Federated Prototype Learning

  • Federated prototypes are class-wise mean feature vectors used to abstract knowledge, support privacy-preserving sharing, and improve communication efficiency.Prior approaches include local regularization and server-side adaptive margin-enhanced contrastive learning.Source: §2.2 · p. 3
  • Because multimodal heterogeneity creates feature misalignment more severe than standard domain shifts, the paper proposes learnable server prototypes to distill client knowledge and align heterogeneous spaces.Naive aggregation does not bridge the divergent feature spaces produced by heterogeneous local models.Source: §2.2 · p. 3

3. Preliminary

The section formalizes multimodal federated learning with clients that may have different modality subsets, then motivates prototype sharing and Gromov–Wasserstein alignment for heterogeneous representations.

3.1. General multimodal FL Framework

  • Clients have modality-specific feature extractors and a classifier, and the global objective jointly optimizes local objectives while encouraging privacy-preserving knowledge sharing.Each client may hold only a subset of modalities, with data distributions, quantities, and modality configurations varying across clients.Source: §3.1 · p. 3

3.2. Prototype-based Federated Learning

  • Prototype-based federated learning exchanges class prototypes rather than model parameters, but heterogeneous feature extractors make direct feature-prototype aggregation difficult.Logit prototypes occupy a naturally aligned shared semantic space because each dimension corresponds to a class.Source: §3.2 · pp. 3–4

3.3. Gromov Wasserstein Distance

  • Gromov–Wasserstein distance compares internal pairwise-distance structures and seeks an optimal coupling, enabling comparison without direct feature-dimension correspondence.The passage formulates the comparison over metric measure spaces and notes that p is commonly 2.Source: §3.3 · p. 4

4. Methodology

MFedPBA collects feature and logit prototypes at the server, aligns heterogeneous feature representations, and returns global prototypes for client-side learning. Its server-side logit aggregation weights lower-entropy predictions more heavily, while local updates distill both prototype types.

4. Methodology

  • MFedPBA collects feature and logit prototypes from clients and constructs global prototypes at both levels on the server.The framework is presented as prototype-guided bilateral alignment.Source: §4 · p. 4, §4.1 · p. 4

4.1.1. Logit Prototype Aggregation

  • Lower-entropy logit prototypes receive higher aggregation weights because they indicate lower predictive uncertainty.Logit prototypes are directly aggregated because classifier outputs are described as naturally aligned across heterogeneous models.Source: §4.1.1 · pp. 4–5

4.1.2. Feature Prototype Learning

  • Modality-specific encoders and a shared decoder map feature prototypes into a comparable semantic space, with reconstruction, contrastive, and GW losses supporting alignment.The contrastive loss pulls different modalities together for the same class and separates different classes; GW alignment preserves inter-class relational structure without requiring strict dimensional alignment.Source: §4.1.2 · p. 5
  • The server uses entropy-regularized optimal transport solved by Sinkhorn, averages learned prototypes by modality, and sends global feature and logit prototypes to clients.The returned prototypes are intended to transfer complementary cross-modal knowledge to clients missing modalities.Source: §4.1.2 · p. 5

4.2. Client Local Update

  • Clients distill global feature and logit prototypes through feature alignment and KL-based logit alignment, while cross-entropy preserves local-domain discrimination.Feature alignment anchors local representations to global feature prototypes; logit alignment matches local classifier outputs to global logit prototypes.Source: §4.2 · p. 6

5. Convergence Analysis

The convergence analysis defines communication-round and local-iteration indices, develops client-loss bounds under stated assumptions, and reports a non-convex convergence rate for local models.

5. Convergence Analysis

  • The analysis defines t as the communication round and e as the local-iteration index, with E denoting the maximum number of local iterations.It introduces lemmas and theorems under assumptions to constrain client loss.Source: §5 · p. 6
  • The authors state that each client’s local model converges at a non-convex convergence rate.The rate statement is conditioned on a learning-rate bound in Theorem 5.4.Source: §5 · p. 6

6. Experiments

Experiments use four multimodal benchmarks and evaluate heterogeneous models across client modality-partition settings. MFedPBA consistently outperforms the compared methods across those settings.

6.1. Experimental Setup

  • Experiments use Caltech101, Reuters, NUS-WIDE, and YouTube, with a 75%/25% training-test split and heterogeneous models assigned across modalities and clients.The three data settings are M2 (two modalities per client), M1+ (at least one), and M1 (exactly one).Source: §6.1 · pp. 6–7

6.2. Experimental Result

  • MFedPBA consistently achieves superior test accuracy across four datasets and the M2, M1+, and M1 partition settings.Table 2 reports average performance across these distributions.Source: §6.2 · p. 7, Table 2 · p. 8
  • On Reuters, t-SNE visualizations show modality-separated features despite shared labels, while MFedPBA produces more compact, structured clusters than Local training.In several settings, FedProto underperforms Local training.Source: §6.2 · p. 7, Figure 5 · p. 8
  • To assess scalability, experiments extended to a larger number of clients, and results in Table 2 show MFedPBA consistently maintains superior performance in large-scale federated-learning settings.Source: §6.2 · p. 7
  • MFedPBA adds logit-prototype communication and server alignment work while limiting incremental client overhead and shifting primary computation to the server.The authors report acceptable client runtime per round on Reuters and characterize the additional communication overhead as negligible relative to methods transmitting model parameters.Source: §6.2 · pp. 7–8, Figure 6 · p. 9
  • Across four datasets under M2, MFedPBA exhibits stable convergence, whereas prototype-based baselines, particularly FedTGP, fluctuate during training.The authors associate baseline instability with erratic server prototype updates in multimodal scenarios.Source: §6.2 · p. 8, Figure 4 · p. 8
  • Removing any server-side module significantly reduces accuracy in the Caltech101 M2 ablation, with reconstruction-loss removal causing the largest drop.The authors attribute that decline to latent representation collapse and erroneous global knowledge reaching local training.Source: §6.2 · p. 9, Figure 7 · p. 9
  • Dimension and hyperparameter sweeps reveal trade-offs between representational capacity, trainability, and alignment strength.On YouTube, most evaluated methods are most stable and accurate at dD = 48; across most tested λ1 and λ2 choices, accuracy exceeds 46, with favorable settings reaching 48.9.Source: §6.2 · p. 9, Table 3 · p. 9, Figure 8 · p. 9

7. Conclusion.

The paper concludes that MFedPBA addresses model heterogeneity and modality imbalance through prototype-guided alignment for knowledge sharing across heterogeneous feature spaces. The appendix lists proofs and experimental details, and points to an algorithm summary.

7. Conclusion.

  • MFedPBA combines GW-based feature alignment with semantically aligned logit prototypes to address model heterogeneity and modality imbalance.The authors report that it mitigates negative transfer and knowledge contamination while improving generalization and robustness under imbalanced and missing-modality settings.Source: §7 · p. 9

A.1. Algorithm Framework

  • Algorithm 1 summarizes the proposed framework's main steps.Source: §A.1 · p. 13

A.2. Privacy Analysis

MFedPBA limits communication to feature and logit prototypes, which the paper argues improves privacy preservation relative to gradient- or parameter-sharing approaches. Its convergence analysis derives client-loss bounds and a non-convex convergence result under stated smoothness and gradient assumptions.

A.2. Privacy Analysis

  • Prototype-only communication is presented as more privacy-preserving than sharing gradients or parameters, while class-level aggregation and local encoders impede reconstruction and inference attacks.The paper also says server-side prototype optimization further reduces the probability of cross-client information leakage.Source: §A.2 · p. 13

B. Convergence Analysis

  • One communication round consists of client local updates followed by server prototype updates and redistribution.The analysis denotes the current communication round by t and local iterations by e, with E as the maximum local iterations.Source: B. Convergence Analysis · p. 14

B.1. Assumption

  • The convergence analysis assumes Lipschitz-smooth losses and bounded, unbiased stochastic gradients, including a bounded server optimization drift.These conditions are stated across assumptions B.1–B.5.Source: §B.1 · p. 15

B.2. Lemmas & Theorems

  • Under its assumptions, the analysis bounds client loss during local updates and after server feature-prototype learning and logit-prototype aggregation.Theorem B.9 states a non-convex convergence result for an arbitrary client and any ε > 0.Source: §B.2 · pp. 15–16

B.3. Proof of Lemma 5.1

The proof of Lemma 5.1 takes expectations over minibatch randomness and telescopes the local-update steps to obtain a bound on client loss during a communication round.

B.3. Proof of Lemma 5.1

  • Lemma 5.1 bounds an arbitrary client’s loss from the start of a communication round through its last local update.The proof takes expectations over the random batch and telescopes across E local steps.Source: §B.3 · p. 16
  • The proof concludes after deriving the expected-loss bound using the local stochastic-gradient variance term.The supplied proof passage includes a term involving σ² but does not state a readable complete bound.Source: §B.3 · p. 16
  • The analysis uses unbiased local stochastic gradients with bounded variance as assumptions for this lemma.The proof explicitly takes expectation over the random minibatch and then telescopes the local steps.Source: §B.3 · p. 16

B.4. Proof of Lemma 5.2

The proof bounds the loss change associated with server aggregation, then combines server-side and client-side changes to establish a one-round loss bound and a convergence condition.

B.4. Proof of Lemma 5.2

  • Because client model parameters stay fixed during aggregation, the cross-entropy loss cancels and the loss change comes from updated global prototypes in the regularization term.The proof analyzes the logit-prototype weighted aggregation and feature-prototype changes separately.Source: §B.4 · pp. 16–17

B.5. Proof of Theorem 5.3

  • Theorem 5.3 expresses one communication round’s total loss change as contributions from the server-side and client-side phases.Its bound uses Γ_d for client drift and Γ_s for server bias.Source: §B.5 · pp. 18–19

B.6. Proof of Theorem 5.4

  • Theorem 5.4 derives a sufficient condition on communication rounds for an arbitrary client to satisfy the stated ε convergence inequality.The proof sums the per-round relation and then solves for T.Source: §B.6 · pp. 19–20

C.1. Dataset Description

The experiments use four multimodal datasets and construct client partitions with varying modality availability and sample sizes. They also randomize modality-specific model architectures and compare against adapted federated-learning baselines.

C.1. Dataset Description

  • The evaluation uses Caltech101, Reuters, NUS-WIDE, and YouTube, spanning image, multilingual text-view, web-image/text, and synchronized video modalities.The dataset descriptions include 102 Caltech101 categories, 18,758 Reuters articles across six topics, eight NUS-WIDE categories, and 10 YouTube categories.Source: §C.1 · pp. 20–21
  • M1 assigns one modality per client, M2 assigns two randomly selected modalities, and M1+ gives each client at least one modality with client-specific composition.The number of clients equals the number of modalities in these configurations, and circle area in the visualizations represents sample size.Source: §C.1 · p. 22, Figure 9 · p. 22, Figure 10 · p. 22
  • The larger-client M1+ experiments use 50 Caltech101 clients, 100 Reuters clients, and 20 clients each for NUS-WIDE and YouTube, with non-IID Dirichlet partitioning at α = 0.5.The resulting distributions are visualized in Figure 12.Source: §C.1 · p. 22, Figure 12 · p. 24

C.2. Heterogeneous Model.

  • Modality-specific feature extractors are sampled from a candidate pool, with network depth and width randomized to create heterogeneous architectures.Repeated-module counts range from 2 to 5, and hidden dimensions are multiples of 32.Source: §C.2 · pp. 22–23, Table 4 · p. 23

C.3. Baseline Methods

  • The comparison includes prototype-based methods, multimodal federated-learning methods, and local training, with baselines reimplemented in a unified framework.Homogeneous multimodal baselines were adapted to the heterogeneous setting by changing uploads from full models to classifier parameters; multimodal local features are summed for Local, prototype methods, and MFedPBA.Source: §C.3 · p. 23

C.4. Parameter Setting Details

  • Experiments for all methods ran on two NVIDIA GeForce RTX 4090 GPUs with an AMD Ryzen 9 9950X 16-Core Processor.Method parameters are referenced in Table 5, but their values are not included in the supplied passage.Source: §C.4 · p. 24

D. Details of the Experimental Result

The experiments report test accuracy across four multimodal datasets and three partitioning schemes. In the larger-client M1+ evaluation, MFedPBA maintains superior performance and resilience.

D. Details of the Experimental Result

Loading 2609.38925v1…