PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-OpenFGL: A Comprehensive Benchmark for Multimodal Federated Graph Learning

Xunkai Li, Yuming Ai, Yinlin Zhu, Haodong Lu, Yi Zhang, Guohao Fu, Bowen Fan, Qiangqiang Dai, Rong-Hua Li, Guoren Wang

arXiv:2601.22416v1cs.LG

TL;DR

MMAG learning is difficult to federate because modalities and graph structures are distributed, while existing FGL mainly targets single-modality settings. MM-OpenFGL formalizes MMFGL through a broad benchmark and experiments, finding that multimodal models outperform single-modality baselines and that robustness and convergence vary substantially across methods and conditions.

  • Problem

    Existing FGL largely focuses on single-modality graphs and lacks a unified formulation and benchmark for multimodal federated graph learning.

  • Method

    MM-OpenFGL integrates multimodal datasets, federated simulation strategies, downstream tasks, algorithms, and a modular open-source API for MMFGL evaluation.

  • Results

    Multimodal GNNs consistently outperform single-modality GNNs, while method robustness and convergence differ across modality-missing conditions and federated designs.

  • Takeaways & Limitations

    MM-OpenFGL provides a unified evaluation standard for studying cross-modal alignment, structural resilience, and efficiency in privacy-preserving MMFGL.

Abstract

from arXiv · show

Multimodal-attributed graphs (MMAGs) provide a unified framework for modeling complex relational data by integrating heterogeneous modalities with graph structures. While centralized learning has shown promising performance, MMAGs in real-world applications are often distributed across isolated platforms and cannot be shared due to privacy concerns or commercial constraints. Federated graph learning (FGL) offers a natural solution for collaborative training under such settings; however, existing studies largely focus on single-modality graphs and do not adequately address the challenges unique to multimodal federated graph learning (MMFGL). To bridge this gap, we present MM-OpenFGL, the first comprehensive benchmark that systematically formalizes the MMFGL paradigm and enables rigorous evaluation. MM-OpenFGL comprises 19 multimodal datasets spanning 7 application domains, 8 simulation strategies capturing modality and topology variations, 6 downstream tasks, and 57 state-of-the-art methods implemented through a modular API. Extensive experiments investigate MMFGL from the perspectives of necessity, effectiveness, robustness, and efficiency, offering valuable insights for future research on MMFGL.

1. Introduction

MMAG research has largely assumed centralized access, but real-world multimodal graphs are distributed across data owners and cannot be freely shared. MM-OpenFGL addresses resulting formalization, benchmarking, and empirical-evidence gaps with a comprehensive MMFGL benchmark and open resources.

  • Centralized MMAG methods assume one learning system can access all modalities and graph structures.
  • Privacy constraints and commercial competition distribute multimodal graphs across isolated platforms, limiting centralized methods’ use of decentralized knowledge.
  • Existing FGL methods primarily target single-modality graphs, while naive adaptations can fail to reconcile cross-modal semantic conflicts and cross-client structural mismatches.
  • MMFGL lacks unified problem formalization, multimodal benchmarking support, and systematic empirical insights for principled algorithm evaluation.
  • MM-OpenFGL formalizes MMFGL with 19 datasets across 7 domains, 8 simulation strategies, 57 algorithms, and evaluations spanning necessity, effectiveness, robustness, and efficiency.
  • The benchmark’s open-source library, datasets, tutorials, and literature repository are intended to lower entry barriers for MMFGL research.

2. Problem Statement

MM-OpenFGL evaluates federated multimodal graph learning through end-to-end task-specific collaboration and two-stage foundation-model representation learning. The pipelines differ in whether models are trained from scratch for local tasks or pretrained globally before client adaptation.

  • End-to-End Pipeline: The end-to-end pipeline trains a task-specific model from scratch through iterative federated communication.
  • End-to-End Pipeline: Each end-to-end round distributes the global model, performs private client updates, uploads parameters or gradients, and aggregates them server-side.
  • Two-Stage Pipeline: The two-stage pipeline first federated-pretrains a global graph encoder using self-supervised structural or multimodal objectives.
  • Two-Stage Pipeline: After pretraining, clients download the foundation backbone and fine-tune it for client-specific tasks and graph distributions.

3. Benchmark Design

MM-OpenFGL combines heterogeneous multimodal datasets, encoders, simulation scenarios, task levels, algorithm categories, and evaluation dimensions into a unified benchmark. Its design stresses modality, topology, label, robustness, and deployment factors.

  • Datasets and Encoders: The benchmark curates raw multimodal data aligned with graph structures across e-commerce, recommendation, social-media, and medical-imaging domains.
  • Datasets and Encoders: Pre-trained multimodal and single-modality encoders support experiments trading feature quality against computational efficiency.
  • Simulation Strategies: A tri-dimensional 2 × 2 × 2 design produces eight scenarios varying modality, topology, and label heterogeneity.
  • Simulation Strategies: Modality-NonIID uses a Dirichlet distribution to simulate missing modalities and encourage representations robust to disjoint or partial modality views.
  • Simulation Strategies: Topology-unavailable settings remove original edges and reconstruct them, whereas topology-available settings preserve inherent graph structure.
  • Tasks and Algorithms: Tasks cover node classification, link prediction, modality matching, modal retrieval, modal alignment, and modality generation.
  • Tasks and Algorithms: The algorithm taxonomy includes multimodal GNNs, standard and heterogeneous federated learning, and graph foundation models.
  • Experiments and Evaluations: Evaluations span data analysis, effectiveness, robustness, and efficiency using task-specific metrics and deployment-oriented scalability analysis.

4. Experiments and Analysis

The experiments show that multimodal semantics and graph topology are jointly necessary, while naive federated adaptations often underperform and specialized methods are more robust to heterogeneous conditions. Robustness remains sensitive to missing modalities, feature encoders, label perturbations, topology loss, and efficiency trade-offs.

  • Necessity Analysis: Multimodal GNNs consistently outperform single-modality GNNs, while GCN outperforms MLP, showing that both multimodal semantics and graph topology matter.The paper concludes that neither graph structure nor multimodal semantics alone suffices for optimal performance.
  • Necessity Analysis: Naive federated adaptations can underperform isolated training because they do not align conflicting multimodal signals or resolve structural mismatches across clients.The benchmark compares isolated MM-GNNs, multimodal FL adapted to graphs, and FGL adapted to multimodal data.
  • Performance Comparison: MH-pFLID is consistently the strongest method across nearly all heterogeneous settings, while competing methods fluctuate substantially across datasets and performance declines under label-NonIID conditions.The ranking is largely stable from label-IID to label-NonIID, but overall performance deteriorates in the latter setting.
  • Performance Comparison: Graph foundation models outperform traditional MM-GNNs and federated baselines across diverse tasks and uniquely support modality generation.They also adapt to a wider range of downstream tasks than the compared alternatives.
  • Robustness Analysis: Accuracy declines as modality missing increases, with DY losing over 20% near complete missingness while RedditS leaders maintain over 90% accuracy.FedMVP and MH-pFLID are the most consistently robust, whereas FedTGP is competitive on DY but weaker on RedditS.
  • Robustness Analysis: All methods vary substantially across feature encoders, while label noise above 0.5 causes sharp collapse; topology perturbations are generally milder, but MH-pFLID is sensitive to structural loss.Fed-MGAT shows the smallest performance variance across feature encoders, identifying feature robustness as a research direction.
  • Efficiency Analysis: Dense adjacency and server-side similarity computation, rather than training-round count, create major computational bottlenecks, while communication costs differ substantially across methods.MH-pFLID has significantly higher communication overhead, whereas FedMVP, FML, FedIIH, and FedSSP are more communication-efficient.
  • Efficiency Analysis: Prototype-based semantic summaries reduce communication overhead while maintaining high accuracy, and FedMVP, FedSSP, and FedLap converge in 4 rounds with top accuracy.The paper recommends sparse operations, decentralized aggregation, and efficient aggregation or prompt/token designs.

5. Conclusion and Future Directions

MM-OpenFGL formalizes multimodal federated graph learning through a unified benchmark spanning heterogeneous datasets, algorithms, and federated scenarios. Its experiments provide ten insights while emphasizing cross-modal alignment and structural resilience.

  • MM-OpenFGL integrates 19 heterogeneous datasets, 57 state-of-the-art algorithms, and eight federated scenarios for unified MMFGL evaluation.
  • The benchmark evaluates MMFGL across necessity, effectiveness, robustness, and efficiency.
  • Extensive experiments distill ten insights, highlighting limitations of forced adaptation strategies and the roles of cross-modal alignment and structural resilience.
  • A modular open-source library lowers entry barriers and supports fair comparison in privacy-preserving multimodal graph intelligence.

A.1. Datasets

The benchmark uses three complementary dataset families covering recommendation, product understanding, and multimodal graph learning. Across these settings, graphs represent diverse entities with textual, visual, numeric, and task-related attributes.

  • The datasets span sequence recommendation, multimodal product understanding, and multimodal graph learning across video, commerce, social-media, and vision-language domains.
  • Nodes may represent videos, products, posts, image regions, or medical findings, depending on the dataset.
  • Node features combine textual, visual, or numeric attributes, with task labels included when available.
  • For most GNN-based models, multimodal node components are concatenated into the input node representation.

Sequence Recommendation Datasets (NineRec Series)

The NineRec sequence-recommendation datasets cover varied video domains and encode multimodal content alongside interaction or related graph structure. Their differing themes support cross-domain, cross-platform, and transfer-oriented evaluation.

  • Bili Cartoon, Bili Dance, Bili Food, Bili Movie, Bili Music, TN, DY, KU, and QB cover animation, dance, food, film, music, news, short video, education, and Q&A content.
  • The datasets combine textual metadata with visual features, while interaction structures or dataset-specific relations define graph connectivity.
  • The collection supports cross-category, cross-domain, cross-platform, and cross-style recommendation or transfer studies.
  • Dataset sizes range from approximately 6.5k items in QB to 9.5k items in DY among the described subsets.

Multimodal Product Graph Datasets

The multimodal product graph datasets cover fashion, electronics, movies, groceries, sports, toys, and large-scale recommendation settings. Product nodes combine descriptive or categorical text with image features, while edges encode complementary, co-purchase, semantic, or compatibility relations.

  • Product datasets represent items using textual descriptions, specifications, metadata, or category information together with image features.
  • Graph edges capture co-purchase, complementary, thematic, series-membership, accessory-compatibility, or semantic relations.
  • The datasets span approximately 42k Toys products, 45k Movies products, 48k Grocery items, 50k Cloth items, 52k Ele-fashion items, and nearly 55k Sports items.
  • PixelRec50K contains around 50k users and nearly one million user-video interactions, supporting fine-grained visual-signal evaluation.
  • GLAMI-1M contains one million products across 13 languages and 191 categories with multilingual descriptions and high-resolution images.

Social Media, Vision-Language, and Medical Graphs

MM-OpenFGL includes multimodal graphs from social media, vision-language, and medical imaging domains, representing different relationships between textual, visual, and clinical information.

  • RedditS. represents Reddit posts as multimodal nodes containing text, visual media, and auxiliary statistics, with edges capturing posting or co-occurrence patterns.
  • Flickr30k. forms a region-phrase graph whose image-region and textual-phrase nodes are connected by semantic-alignment edges for retrieval and grounding.
  • The Multimodal Medical Imaging Dataset pairs more than 20k radiological images with clinical text, linking image regions and clinical terms for diagnosis and report generation.

A.2. Baseline Description

The benchmark combines graph backbones, multimodal fusion models, federated optimization methods, and graph foundation models to cover structural, modality, and client heterogeneity.

  • MM-OpenFGL organizes baselines into GNN backbones, standard FL, heterogeneous FL, and graph foundation models using pretrain-then-finetune pipelines.
  • Traditional GNN references include GCN, GAT, GraphSAGE, GIN, ChebNet, and RevGAT, covering convolution, attention, inductive aggregation, expressiveness, spectral filtering, and memory efficiency.
  • Multimodal backbones include MMGCN, MGAT, GSMN, MGNet, MHGAT, and UniGraph2, representing modality-aware propagation, alignment, multiplex modeling, heterogeneous structure handling, and unified graph spaces.
  • Standard federated baselines include FedAvg, FedProx, and SCAFFOLD, while MOON, FedDC, and FedExP address representation consistency, local drift, and personalization.
  • FedExP learns adaptive aggregation weights to balance global knowledge sharing with client-specific personalization under heterogeneous data distributions.

A.3. Comparison with OpenFGL

MM-OpenFGL extends OpenFGL’s single-modality federated graph-learning framework to multimodal federated graph learning while retaining its modular evaluation infrastructure.

  • MM-OpenFGL inherits OpenFGL’s modular API and standardized four-stage communication protocol to provide a consistent collaborative-training environment.
  • The extension broadens evaluation to complex multimodal scenarios and introduces additional MMFGL-specific capabilities beyond the original single-modality benchmark.

A.4. Evaluation Protocols

The evaluation standardizes tasks, encoders, optimization settings, metrics, and complexity analyses across multimodal federated graph-learning methods.

  • Experimental Protocols: The protocol evaluates supervised node classification, supervised link prediction, and unsupervised node clustering using a frozen CLIP-ViT-Large-Patch14 encoder with 768-dimensional features.
  • Experimental Protocols: Modality retrieval uses contrastive learning with temperature τ = 0.07, 500 training epochs, learning rate 1 × 10^-3, batch size 256, and early stopping.
  • Experimental Protocols: G2Text uses multimodal-neighbor sampling, GNN structural encoding, and Facebook OPT-125M with prefix tuning or LoRA adaptation at rank r = 64.
  • Experimental Protocols: G2Image uses personalized PageRank neighbor sampling and conditions Stable Diffusion v1.5 at 256-pixel resolution through graph classifier-free guidance.
  • Complexity Analysis: Complexity analysis defines n, m, c, K, h, f, F, q, and E before deriving theoretical time and space costs for evaluated methods.
  • Complexity Analysis: Standard GNN training costs O(K · m · f) for sparse neighborhood aggregation, alongside dense feature transformations.
  • Complexity Analysis: Structural methods can incur O(n^2) time or space overheads for dense global structures, while FedMAC stores class-wise prototypes with space complexity O(Nf^2 + hc).
Loading 2601.22416v1…