PaperlayerView exact revision on arXiv

Source-linked AI summary

Coupling Perception and Reasoning in Federated Multimodal Graph Foundation Models

Zekai Chen, Xun Wu, Hailin Zhang, Xunkai Li, Yu Liu, Kairui Yang, Muyan Huang, Xuaner Chen, Rong-Hua Li, Guoren Wang

arXiv:2610.00277v1cs.LG

TL;DR

Federated multimodal graph foundation models need to adapt private decentralized graphs, but independent Encoder and GNN updates can lose their learned coordination during aggregation. FedCORE represents and optimizes their updates through a shared latent state. It improves over strong federated baselines, with a reported 2.27% average gain over the strongest non-FedCORE baseline.

  • Problem

    Federated adaptation of multimodal graph foundation models must coordinate Encoder and GNN updates, because independent module-wise aggregation can lose the relationship established during local training.

    Source: §1 · p. 1

  • Method

    FedCORE jointly optimizes a shared latent state that represents Encoder and GNN adaptations and communicates and aggregates that state during federation.

    Source: §1 · p. 2

  • Results

    2.27% is FedCORE’s reported average improvement over the strongest non-FedCORE baseline across multimodal graph benchmarks and downstream tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FedCORE preserves perception–reasoning coordination across local optimization and federated aggregation while consistently outperforming frozen-Encoder and independent adaptation strategies.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

Federated multimodal graph foundation models (GFMs) aim to adapt pretrained multimodal models to decentralized graph data, where each client owns a private multimodal graph and cannot share raw information. These models typically combine a multimodal Encoder that extracts semantic evidence from heterogeneous modalities and a graph neural network (GNN) that performs relational reasoning over graph structures. However, existing federated GFM adaptation methods mainly update graph-side modules while keeping the multimodal Encoder frozen, limiting adaptation to \emph{how information is propagated} while fixing \emph{what information is extracted}. Through empirical studies, we reveal that Encoder and GNN adaptations are not independent: Encoder adaptation is affected by graph relations, while cross-client module swapping reveals substantial pairing sensitivity between separately parameterized Encoder and GNN updates. Motivated by this observation, we propose \textbf{FedCORE}, a federated adaptation framework that represents Encoder and GNN updates through a shared low-dimensional latent state. FedCORE jointly optimizes this core from multimodal and structural signals and performs federated evolution directly in the shared state space, preserving compatibility between perception and reasoning adaptations. Extensive experiments demonstrate that FedCORE reduces the Encoder--GNN pairing gap from $30.6$ to $5.9$, corresponding to an $80.7\%$ reduction over independent joint adaptation.

1 INTRODUCTION

Federated multimodal graph foundation models must coordinate Encoder-based perception with GNN reasoning while adapting decentralized data without sharing raw information. The paper finds these adaptations are coupled and proposes FedCORE, which represents them through a shared latent state.

1 INTRODUCTION

  • Federated multimodal graph data cannot be centralized because of privacy, ownership, and communication constraints, motivating adaptation without sharing raw data.Multimodal GFMs combine pretrained Encoders with GNNs to model semantics and graph relations.Source: §1 · p. 1
  • Independent aggregation of jointly trained Encoder and GNN updates fails to preserve their learned relationship, weakening coordination after federation.The challenge is both enabling joint adaptation and preserving coordination during federated optimization and aggregation.Source: §1 · p. 1
  • Encoder adaptation is necessary, depends on graph relations, and is pairing-sensitive with GNN adaptation across clients.Controlled topology changes alter Encoder updates, and locally matched Encoder–GNN pairs outperform mismatched combinations on average.Source: §1 · p. 2
  • FedCORE jointly optimizes a shared latent core and decodes it into low-rank updates for Encoders, fusion layers, and GNNs.Only the shared core is communicated and aggregated, maintaining learned perception–reasoning coordination while retaining communication efficiency.Source: §1 · p. 2
  • 2.27% is FedCORE’s average improvement over the strongest non-FedCORE baseline across multimodal graph benchmarks and downstream tasks.The contribution passage also reports consistent gains over strong federated baselines.Source: §1 · p. 2

2 RELATED WORK

Prior multimodal graph foundation models combine multimodal perception and graph reasoning, but largely study centralized training. Federated adaptation work addresses decentralized settings yet generally treats representation learning and graph reasoning as independent components.

2.1 MULTIMODAL GRAPH FOUNDATION MODELS

  • Multimodal graph foundation models combine pretrained multimodal Encoders with GNNs to capture semantics and graph relations, with prior work spanning interaction, topology-aware learning, and structural grounding.The cited methods primarily consider centralized training with complete data.Source: §2.1 · p. 2

2.2 FEDERATED ADAPTATION OF GRAPH FOUNDATION MODELS

  • Federated graph foundation models and multimodal federated graph learning address decentralized adaptation, but existing approaches generally treat representation learning and graph reasoning as independent components.FedCORE instead models Encoder and GNN adaptations through a shared latent state to preserve their coordination throughout federated learning.Source: §2.2 · p. 2

3 EMPIRICAL STUDY

The empirical study tests whether graph-side capacity can replace Encoder adaptation, whether Encoder updates depend on graph relations, and whether independently learned Encoder–GNN updates remain compatible across clients. The findings show Encoder adaptation is needed, relationally conditioned, and sensitive to cross-client pairing.

3 EMPIRICAL STUDY

  • The empirical study evaluates Encoder necessity, dependence of Encoder adaptation on graph relations, and the need to coordinate Encoder–GNN updates during federation.Detailed protocols and metric definitions are provided in Appendix A.3.Source: §3 · p. 3

3.1 CAN STRONGER GNNS REPLACE ENCODER ADAPTATION?

  • Increasing GNN capacity improves graph-only adaptation but remains inferior to Encoder-adapted models, so graph reasoning cannot recover multimodal information missing from a frozen Encoder.The study increases trainable capacity for GCN, GraphSAGE, and GAT and compares against Encoder LoRA and full fine-tuning.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 DOES ENCODER ADAPTATION DEPEND ON RELATIONAL CONTEXT?

  • Encoder-LoRA updates change more under stronger graph interventions than under semantic-preserving perturbations, with mean update distance 0.088 versus 0.036.Node features, labels, and splits remain unchanged while graph topology is intervened on.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

3.3 ARE ENCODER AND GNN ADAPTATIONS INDEPENDENT?

  • Matched Encoder–GNN pairs outperform mismatched combinations; Independent Joint LoRA has a pairing gap of 30.6, versus 22.8 after module-wise aggregation.FedCORE exhibits a substantially smaller pairing gap, supporting the need to preserve Encoder–GNN coordination during federation.Source: §3.3 · p. 4, Figure 3 · p. 4

4 METHODOLOGY

FedCORE represents Encoder and GNN adaptations through a shared state, optimizes that state using task supervision and perception–reasoning synergy, and evolves it across clients. Its design keeps heterogeneous module updates rooted in a common parameterization during local training and federation.

4.1 OVERVIEW

  • FedCORE comprises latent state modeling, synergistic state learning, and federated state evolution to represent, optimize, and evolve coordinated Encoder–GNN adaptations.The framework overview is shown in Figure 4.Source: §4.1 · pp. 4–5, Figure 4 · p. 4

4.2 LATENT STATE MODELING

  • A client-specific r × r state Ck is the common adaptation coordinate for Encoder, fusion, and GNN matrices rather than separate module variables.Fixed pretrained coordinates are constructed using rank-r truncated SVD and remain frozen; fixed linear decoders map the shared state to module updates, which retain native parameter forms.Source: §4.2 · p. 5

4.3 SYNERGISTIC STATE LEARNING

  • FedCORE trains the shared state from downstream task supervision and perception–reasoning synergy, using joint and isolated-branch predictions to encourage the joint adaptation to outperform the stronger isolated branch without degrading either reference.Perception and reasoning signals jointly update a single client state.Source: §4.3 · pp. 5–6

4.4 FEDERATED STATE EVOLUTION

  • FedCORE evolves shared states using client coordination scores and state drift, favoring coordinated states while suppressing excessive deviation from the previous global state.Clients upload their final state and a scalar coordination score; module parameters and gradients remain local, and the resulting global state is broadcast for the next round.Source: §4.4 · pp. 6–7

5 EXPERIMENTS

Experiments compare FedCORE with federated and multimodal graph baselines across downstream tasks, then examine its design, robustness, and resource costs. FedCORE leads the overall comparison and substantially reduces communication relative to Independent Joint LoRA.

5.1 EXPERIMENTAL SETUP

  • The evaluation spans node classification, link prediction, retrieval, graph-to-text generation, and graph-to-image generation, with shared partitions, pretrained backbones, and protocols.Baselines include federated, federated graph foundation, and multimodal federated graph methods, plus Frozen Encoder + GNN and Independent Joint LoRA controls.Source: §5.1 · p. 7

5.2 OVERALL PERFORMANCE (Q1)

  • 2.27% average gain over the strongest non-FedCORE baseline, reaching 6.16% on Bili Cartoon; FedCORE also exceeds Independent Joint LoRA by 5.98% on average.The comparison spans multimodal graph benchmarks and downstream tasks.Source: Table 1 · p. 7, §5.2 · pp. 7–8

5.3 ABLATION STUDY (Q2)

  • Ablations replace the shared latent state, fixed SVD bases, geometry-balanced update, or shared-core aggregation to assess each design component.Figure 5 relates pairwise shared-core similarity to decoded Encoder and GNN update similarity on Toys and Bili Music; positive correlations indicate similar cores tend to yield similar adaptations.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 ROBUSTNESS ANALYSIS (Q3)

  • With client count varied from 3 to 10, larger client counts fragment local graphs; a broad high-performing region on Bili Music link prediction indicates limited sensitivity to the tested hyperparameters.The sensitivity analysis varies latent-state rank r and core scaling ratio α/r; modality corruption and missingness are evaluated on Grocery node classification.Source: §5.4 · p. 8, Figure 7 · p. 9, Figure 6 · p. 9

5.5 EFFICIENCY AND SCALABILITY ANALYSIS (Q4)

  • The efficiency analysis assesses trainable parameters, communication, FLOPs, GPU memory, and training time, and considers scalability as more model components are adapted.The reported comparison quantifies per-round communication; additional convergence and adaptation-scale analyses are in the appendix.Source: §5.5 · p. 9, Table 3 · p. 9
  • FedCORE communicates 5.12 × 10^2 scalars per round versus 4.64 × 10^5 for Independent Joint LoRA, a 99.89% reduction.The comparison counts one upload and one download of federated adaptation parameters per participating client per round.Source: Table 3 · p. 9, §5.5 · p. 9

6 CONCLUSION

The conclusion presents FedCORE as a shared-latent-state approach to coordinating Encoder and graph reasoning adaptations. The accompanying statements describe the paper’s AI use, reproducibility materials, and federated problem setting.

6 CONCLUSION

  • FedCORE represents perception and graph reasoning through a shared latent state and preserves their coordination during local optimization and federated aggregation.The paper concludes that experiments across diverse multimodal graph tasks show consistent gains over frozen-Encoder and independent adaptation strategies.Source: §6 · pp. 9–10

AI USE STATEMENT

  • The manuscript reports limited AI-tool use for language refinement, literature retrieval and discovery, research ideation, and methodological and experimental discussions.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The anonymized reference implementation provides experiment configurations and reproducibility checks, but excludes datasets and pretrained checkpoints.It documents required data formats, external model dependencies, and instructions for reproducing reported settings.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 PROBLEM FORMULATION

  • The formulation keeps each client’s raw multimodal graph data local while defining downstream prediction from local features, graph structure, labels, and task head.The paper studies jointly representing and federatively evolving perception–reasoning adaptations while keeping client privacy.Source: §A.1 · p. 12

A.2 STRUCTURAL RANDOMIZATION ANALYSIS

The controlled studies examine whether graph capacity can replace Encoder adaptation, whether graph relations affect Encoder updates, and whether separately learned Encoder–GNN modules remain compatible. Their findings motivate representing relation-conditioned adaptations through a shared state.

A.2 STRUCTURAL RANDOMIZATION ANALYSIS

  • FedCORE consistently achieves higher accuracy as graph integrity varies from fully randomized to original structure, while all methods improve as more structure is preserved.The study varies structural integrity while keeping node contents and supervision unchanged.Source: §A.2 · p. 12

A.3 CONTROLLED PROTOCOLS FOR EMPIRICAL STUDY

  • The diagnostic chain changes one investigated factor while holding node contents, labels, splits, pretrained backbones, and optimization budgets identical unless specified otherwise.It tests graph-side capacity substitution, relational effects on preferred Encoder updates, and independent composability of Encoder and GNN adaptations.Source: §A.3 · pp. 12–13

A.3.1 DIAGNOSING THE LIMIT OF GRAPH-SIDE ADAPTATION

  • Increasing GNN capacity improves performance, particularly on QB, but frozen-Encoder configurations remain constrained relative to Encoder-adapted models.The study compares increasing-capacity GCN, GraphSAGE, and GAT models with Encoder adaptation and full fine-tuning on Toys and QB.Source: §A.3.1 · p. 13

A.3.2 CONTROLLED RELATIONAL INTERVENTION

  • Relational interventions change graph structure while node identities, contents, labels, and data splits remain fixed, with edge rewiring approximately preserving node-degree profiles.The update-distance comparison uses low-semantic-shift controls, not semantic-preserving placebo interventions.Source: §A.3.2 · pp. 13–14
  • The strongest relational intervention yields an update distance of 0.088 versus 0.036 under low-semantic-shift controls, a 2.4× difference; corresponding dose–response slopes are 0.70 and 0.24.Because neighborhood label homophily is not explicitly controlled, this does not establish a pure topology-only causal effect.Source: §A.3.2 · p. 14

A.3.3 CROSS-CLIENT ENCODER–GNN COMPATIBILITY

  • Pairing gaps were 30.6 for Independent Joint LoRA, 22.8 after module-wise federated aggregation, and 5.9 for FedCORE.The diagnostics motivate jointly representing relation-conditioned Encoder and GNN adaptations through a common state that can survive federated aggregation.Source: §A.3.3 · p. 15

A.4 THEORETICAL PROPERTIES OF SHARED-CORE ADAPTATION

The theoretical analysis links graph-dependent Encoder learning signals to FedCORE’s shared compatibility state, then characterizes how that state behaves under aggregation and a single local update. It does not establish convergence of the full multi-step federated procedure.

A.4 THEORETICAL PROPERTIES OF SHARED-CORE ADAPTATION

  • The propositions characterize shared linear parameterization and a single preconditioned local step, but provide no convergence guarantee for multi-step federated optimization with client drift or state-quality weighting.The analysis also does not cover partial client participation or interactions between local preconditioner statistics and server-side state evolution.Source: §A.4 · pp. 15, 19
  • Changing graph relations can alter the gradient applied to the multimodal Encoder, because graph propagation depends on those relations.This holds even when multimodal inputs and labels remain unchanged, except in degenerate cases where relevant Jacobians or downstream gradients coincide.Source: §A.4 · pp. 15–16
  • FedCORE’s module updates occupy a common compatibility space of dimension r^2, and each update can be uniquely projected back to the shared core.The shared state therefore links otherwise heterogeneous Encoder and GNN parameter updates.Source: §A.4 · p. 16
  • Linear federated averaging preserves the shared compatibility space with zero compatibility defect; independent Encoder and GNN updates need not share a projected state.With the same linear decoder and aggregation weights, averaging decoded updates is equivalent to averaging cores before decoding.Source: §A.4 · p. 17
  • Encoder- and GNN-side learning signals are pulled back into the same core coordinates and jointly determine core evolution.The Encoder-side signal already reflects local graph relations through back-propagation.Source: §A.4 · p. 18
  • Under smoothness and PL conditions with fixed positive-definite preconditioners, a non-stationary local update strictly decreases the objective when the stated step-size condition holds.This sufficient-descent result concerns one local update, not the complete federated procedure.Source: §A.4 · pp. 18–19

A.5.1 DATASETS AND TASKS

The evaluation spans multimodal graph tasks from classification and retrieval to text and image generation, using decentralized client graphs under the MM-OpenFGL protocol.

A.5.1 DATASETS AND TASKS

  • Across the benchmark, each client owns a local multimodal graph and raw data remain decentralized under the MM-OpenFGL protocol.The evaluation covers node classification, link prediction, multimodal retrieval, graph-to-text generation, and graph-to-image generation.Source: §A.5.1 · pp. 19–20
  • Toys and Grocery are multimodal Amazon product graphs evaluated for node classification using accuracy.Toys has 20,695 nodes and 18 categories; Grocery has 17,074 nodes and 20 categories.Source: §A.5.1 · p. 19
  • Bili Music and DY use NineRec interaction graphs for link prediction with AUC, while QB and Bili Cartoon use them for multimodal retrieval with Recall@5.The datasets contain multimodal user–item interaction graphs with heterogeneous semantic information.Source: §A.5.1 · p. 19
  • Flickr30k is used for graph-to-text generation evaluated with ROUGE-L, and SemArt for graph-to-image generation evaluated with CLIP-Score.Both graphs are partitioned into five clients; Flickr30k uses a Flan-T5-based decoder and SemArt uses a Stable Diffusion v1.5-based generator.Source: §A.5.1 · pp. 19–20

A.5.2 EVALUATION METRICS

The appendix specifies task-specific evaluation metrics, a range of federated and adaptation baselines, and matched implementation settings for comparison.

A.5.2 EVALUATION METRICS

  • Node classification uses Accuracy, link prediction AUC, retrieval Recall@5, graph-to-text ROUGE-L, and graph-to-image CLIP-Score.These metrics respectively measure correct classifications, positive-over-negative edge ranking, top-five retrieval, text overlap, and image–target CLIP embedding similarity.Source: §A.5.2 · p. 20

A.5.3 BASELINES

  • FedAvg, federated GFM adaptation methods, and Federated PLANET provide federated comparison baselines.The GFM baselines include FedGFM+, STAGE, and FedGAMMA; Federated PLANET is the multimodal graph baseline.Source: §A.5.3 · p. 20
  • Frozen Encoder tests graph-only adaptation, while Independent Joint LoRA + FedAvg tests independently learned Encoder and GNN adapters under matched-backbone conditions.The LoRA control adapts visual, textual, fusion, and GNN transformations; its client-specific classifier is not included in the federated payload.Source: §A.5.3 · pp. 20–21

A.5.4 IMPLEMENTATION DETAILS

  • All methods use identical data splits, pretrained backbones, downstream settings, and client partitions, with five clients unless otherwise specified.FedCORE adapts upper Encoder, fusion, and primary GNN layers while lower backbone layers remain frozen; its state dimension and optimization hyperparameters are validation-selected.Source: §A.5.4 · p. 21

B ADDITIONAL EFFICIENCY AND SCALABILITY ANALYSIS

Additional analyses examine communication-round convergence on Toys and the performance–communication trade-off as adaptation scale grows on Grocery.

Convergence Efficiency

Scalability to Adaptation Scale

  • As adaptation scale increases, Independent Joint LoRA communicates more module-specific parameters, while FedCORE continues transmitting a compact r × r core.Figure 9(b) examines the associated performance–communication trade-off on Grocery node classification.Source: Figure 9 · p. 21, B ADDITIONAL EFFICIENCY AND… · p. 21

C ADDITIONAL ROBUSTNESS ANALYSIS

On Toys node classification, FedCORE degrades more slowly than baselines as client feature drift or topology perturbation increases. The analysis varies these perturbations while keeping the node set, labels, data splits, and training settings fixed.

C ADDITIONAL ROBUSTNESS ANALYSIS

  • The Toys node-classification evaluation keeps the node set, labels, data splits, and training settings fixed across perturbations.The two panels vary client feature drift and cross-community edge injection, respectively.Source: C ADDITIONAL ROBUSTNESS ANALYSIS · pp. 21–22, Figure 10 · p. 22
  • FedCORE degrades more slowly than baselines as either client feature drift or topology perturbation increases.Figure 10 varies feature drift α and cross-community edge-injection ratio ρ.Source: C ADDITIONAL ROBUSTNESS ANALYSIS · p. 22, Figure 10 · p. 22
  • Feature drift adds client-wide Gaussian offsets at α values from 0 to 0.60, while topology perturbation injects cross-community edges at ρ values from 0 to 0.25.The topology perturbation is applied before the final five-client partition.Source: C ADDITIONAL ROBUSTNESS ANALYSIS · p. 22
Loading 2610.00277v1…