PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach

Xunkai Li, Chenxi Wan, Yinlin Zhu, Wang Luo, Hongchao Qin, Rong-Hua Li, Guoren Wang

arXiv:2610.02881v1cs.LG

TL;DR

Incomplete node attributes limit the scale and diversity of MAG pretraining, and existing MGFMs underuse topology for multimodal binding. GraphBind uses topology to organize modality semantics in a shared space and adapts it to discriminative and generative tasks; it reports leading results on both, with up to a 28.1% relative improvement in BLEU-4 over the strongest baseline.

  • Problem

    Incomplete MAG node attributes limit the scale and diversity of usable pretraining corpora, while existing MGFMs underexplore topology's role in multimodal binding and unified representations.

    Source: §1 · p. 1

  • Method

    GraphBind uses topology to organize node semantics and neighborhood context into a unified representation, then adapts that space to discriminative and generative tasks through lightweight fine-tuning.

    Source: §1 · p. 2

  • Results

    GraphBind reports leading performance on discriminative and generative tasks, with relative improvement up to 28.1% in BLEU-4 over the strongest baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GraphBind treats topology as a structural reference for integrating structure and semantics in a shared space across discriminative and generative tasks.

    Source: §6 · p. 9

    The reliable-neighbor robustness result assumes reliable neighbors score above less reliable ones and bounds perturbations to less reliable neighbors.

    Source: §4.4.1 · p. 6

Abstract

from arXiv · show

Multimodal graph foundation models (MGFMs) seek to learn generalizable representations from large-scale graphs with heterogeneous node modalities. However, real-world Multimodal-Attributed Graphs (MAGs) often contain incomplete node attributes, limiting the scale and diversity of available pretraining corpora. Besides, existing MGFMs primarily incorporate graph topology as structural context, overlooking its role in guiding multimodal binding and shaping a unified representation space. To address these challenges, we propose GraphBind, a topology-driven approach that uses graph topology to bind rich modality information into a unified shared space. GraphBind is motivated by the stability of graph topology, which provides structural references and complementary semantic information for multimodal binding. Concretely, GraphBind uses topology to organize self semantics and reliable neighborhood semantics into a global shared space that integrates structure and semantics, and adapts this space to discriminative and generative tasks through lightweight interfaces. Extensive experiments against 11 representative baselines demonstrate that GraphBind achieves leading performance on both discriminative and generative tasks, with relative improvements of up to 28.1% over the strongest baseline.

1 INTRODUCTION

MGFMs face incomplete MAG attributes that limit pretraining corpus scale and diversity, while topology is underused as a guide for multimodal binding. GraphBind uses topology to bind modalities into a shared representation space and reports leading performance on discriminative and generative tasks.

1 INTRODUCTION

  • Real-world MAGs often have incomplete node attributes, and restricting pretraining to fully observed nodes limits the scale and diversity of usable graph corpora.Source: §1 · p. 1
  • Existing MGFMs mainly use topology as structural context, leaving its role in guiding multimodal binding and shaping a unified representation space underexplored.Source: §1 · p. 1
  • GraphBind combines incomplete-corpus pretraining, topology-driven multimodal binding, and lightweight adaptation of a unified representation to discriminative and generative tasks.Its adaptation preserves the shared semantic interface learned during pretraining.Source: §1 · p. 2
  • Against eleven representative baselines, GraphBind reports leading performance on discriminative and generative tasks, with relative improvements up to 5.4% in MRR and 28.1% in BLEU-4, respectively.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries define multimodal attributed graphs and the transfer goals of graph foundation models, then position multimodal graph learning as the integration of topology and heterogeneous node attributes.

2.1 PROBLEM FORMULATION

  • A MAG has nodes, edges, an adjacency matrix, and potentially multiple attributes per node, such as text and images.Source: §2.1 · p. 2
  • The foundation-model setup pretrains across MAGs to map nodes into a unified representation, then adapts that representation to node classification, link prediction, Graph-to-Text, and Graph-to-Image tasks.Source: §2.1 · p. 2

2.2 GRAPH FOUNDATION MODELS

  • Earlier graph foundation models support transfer across graph tasks but mainly pretrain on graph structure and a single attribute space, whereas MAGs require a shared space connecting heterogeneous attributes with topology.Source: §2.2 · p. 2

2.3 MULTIMODAL GRAPH LEARNING

  • Multimodal graph learning combines topology with attributes such as text and images, using fusion methods for individual tasks and, more recently, joint topology–modality modeling for transfer.Source: §2.3 · p. 2
  • Figure 1 compares text and vision recovery settings under single-modality absence on Grocery.Source: Figure 1 · p. 3

3 EMPIRICAL INVESTIGATION

The empirical investigation tests whether node-local modalities suffice for multimodal binding and whether graph topology can support recovery when a node has no available modalities. Its findings motivate topology as a structural reference for integrating semantics across incomplete-modality graphs.

3 EMPIRICAL INVESTIGATION

  • On RedditS with both center-node modalities absent, graph-assisted methods recover text and vision more similarly to reference features than a feature-mean prototype.The authors interpret this as evidence that neighborhoods preserve shared semantics transferable across target modalities.Source: Figure 2 · p. 3, §3 · p. 3
  • The study compares modality recovery by cosine similarity, testing single-modality versus graph-assisted recovery on Grocery and graph-assisted recovery versus a feature-mean prototype on RedditS.Source: §3 · p. 3
  • On Grocery, single-modality recovery has lower similarity than graph-assisted recovery across four methods under both absence settings.The remaining modality provides useful but limited information, especially when inferring text from vision.Source: §3 · p. 3
  • Together, the findings suggest topology can organize information across connected nodes into shared semantic representations, supporting MGFM pretraining on graphs with incomplete node modalities.Source: §3 · p. 3

4 METHODOLOGY

GraphBind simulates realistic modality absence, then uses topology to combine self and reliable-neighborhood semantics in a calibrated shared space. Lightweight task interfaces adapt that space for discriminative and generative tasks, while theoretical analyses describe reliability filtering and geometry preservation.

4.1 MODALITY ABSENCE SIMULATION

  • GraphBind controls per-node modality availability so incomplete graph corpora can participate in pretraining, simulating independent, neighborhood-correlated, and property-dependent absence.The encoder receives only observed, unmasked modalities; original features remain reconstruction targets.Source: §4.1 · p. 4

4.2 TOPOLOGY-GUIDED PRETRAINING

  • Available modalities initialize each node’s self-semantic state; topology scores neighbors and reliability-weights their messages before a learned gate combines self and neighborhood semantics.The reliability score incorporates self and neighbor semantics, topology encodings, their interactions, and local edge strength.Source: §4.2 · p. 5
  • Calibration projects completed states into a shared space and uses topology-derived feature-wise parameters, while reconstruction and consistency constraints retain semantics and reliable-edge geometry.The reconstruction constraint retains completed semantics after projection; consistency keeps local geometry stable after calibration.Source: §4.2 · p. 5

4.3 UNIFIED DOWNSTREAM ADAPTATION

  • Lightweight interfaces adapt the pretrained space: prompts support discriminative task geometry, while G2M-Adapter maps target-node and neighbor representations into generator tokens.The prompt path allows task specialization while keeping the pretrained shared space stable; generator tokens retain target and neighborhood information.Source: §4.3 · pp. 5–6

4.4.1 HOW DO RELIABLE NEIGHBORS IMPROVE ROBUSTNESS?

  • Reliability scoring prevents less reliable neighbors from dominating completion, reducing sensitivity to irrelevant or noisy neighborhood information.The theorem’s stated condition separates reliable from less reliable neighbors by their scores.Source: §4.4.1 · p. 6

4.4.2 HOW IS SHARED GEOMETRY PRESERVED?

  • Small reconstruction loss gives a node-level recovery guarantee, while consistency loss bounds weighted-average changes in reliable-edge lengths after calibration.The passage attributes these two guarantees to the reconstruction and consistency losses, respectively.Source: §4.4.2 · p. 6

5 EXPERIMENTS

GraphBind is evaluated on multimodal graph tasks, with experiments testing benchmark performance, architectural variants, modality-absence robustness, and computational costs. The results cover discriminative and generative tasks across diverse datasets.

5.1 EXPERIMENTAL SETUP

  • GraphBind is evaluated on eight MAG datasets across node classification, link prediction, graph-to-text, and graph-to-image tasks against MAG models, graph foundation models, and multimodal graph foundation models.The datasets cover e-commerce, social media, video, image-text, and art domains.Source: §5.1 · p. 7

5.2 MAIN RESULTS (ANSWER FOR Q1)

  • GraphBind ranks best on Grocery and Bili Dance and achieves the highest MRR on DY; across four few-shot settings, its Movies-5Way 5-shot accuracy exceeds the strongest baseline by 2.59 percentage points.It ranks second on Toys and on DY Hits@3.Source: §5.2 · p. 7
  • GraphBind achieves 70.01 CLIP-S and 48.91 DINO-S on G2Image, and 16.43 BLEU-4 and 43.45 ROUGE-L on G2Text, exceeding the strongest baseline by 3.60 and 4.50 points, respectively.The generative-task results are reported in Table 2.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 ABLATION STUDY (ANSWER FOR Q2)

  • The ablation variants replace reliable-neighborhood completion and topology-guided shared-latent calibration together, testing simple propagation, topology encoding without neighborhood propagation, and node-only semantics.Simple propagation uses GCN, GAT, or GraphSAGE over the original adjacency.Source: §5.3 · p. 8
  • GraphBind achieves the best result on all four ablation tasks, with larger gains over the strongest variants on the two generative tasks.Table 3 reports the ablation results for Toys, DY, SemArt, and Flickr30k.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 ROBUSTNESS ANALYSIS (ANSWER FOR Q3)

  • Across Grocery and RedditS, GraphBind has the smoothest macro-F1 decline and remains leading as modality absence rises from 0.0 to 0.8 under independent and neighborhood masking.All methods decline as modality absence increases; neighborhood masking represents locally concentrated failures.Source: §5.4 · p. 8, Figure 4 · p. 9

5.5 EFFICIENCY ANALYSIS (ANSWER FOR Q4)

  • GraphBind pretrains faster than both MGFMs and infers faster than PLANET, while finetuning and inference cost more than for most lightweight MAG baselines.It updates 5.32M trainable parameters, compared with 20.72M for UniGraph2 and 221.94M for PLANET; Table 4 compares computational efficiency.Source: §5.5 · p. 9, Table 4 · p. 9

6 CONCLUSION

GraphBind uses graph topology to bind heterogeneous modality semantics into a unified representation space. The paper reports strong performance, generalization, and robustness across discriminative and generative tasks.

6 CONCLUSION

  • GraphBind uses graph topology to bind heterogeneous modality semantics into a unified representation space, integrating structure and semantics for discriminative and generative tasks.The conclusion describes topology-guided pretraining paired with lightweight fine-tuning.Source: §6 · p. 9

A.1 THEOREM PROOFS

The proof excerpts trace how neighborhood perturbations pass through aggregation and the completion gate, and explain how two latent constraints govern semantic and local geometric properties.

A.1 THEOREM PROOFS

  • For the perturbation comparison, reliable-neighbor message differences cancel, and the completion gate’s center-node contribution is identical for both inputs.The proof also notes that element-wise multiplication by 1 − g_i is nonexpansive under the Euclidean norm.Source: §A.1 · p. 13
  • The proof bounds the less-reliable-neighbor mass, which decreases as the reliability-score separation between reliable and less reliable neighbors increases.The control follows from the fixed signs of the mass function’s partial derivatives.Source: §A.1 · p. 13
  • The two latent constraints jointly control node-level semantic decoding error and metric distortion over reliable local relations.The proof identifies this as the complementary role of the constraints in Eq. 10.Source: §A.1 · p. 14

A.2 METHOD DETAILS

GraphBind simulates missing modalities, uses topology to complete node semantics, and binds completed semantics in a shared latent space. Contrastive, reconstruction, and decorrelation objectives train that space, while task interfaces adapt it for discriminative and generative use.

Modality Absence Simulation

  • Simulated modality absence tests recovery from visible attributes and topology, while a separate reconstruction mask hides encoder-visible features as additional self-supervised targets.At inference, simulated absence indicators are removed and observed modality availability is used.Source: §A.2 · pp. 14–15

Reliable Neighborhood Semantic Completion

  • Topology features and diffusion weights score incoming neighbors, whose reliability-weighted semantics are aggregated to complete each node representation.The neighbor set is bounded and reliability-filtered; if none passes the threshold, the top-scoring neighbor is retained with its continuous reliability weight.Source: §A.2 · p. 15

Contrastive Alignment

  • Contrastive alignment uses each node’s available modality semantics and selected structurally related candidates as positives, with hard negatives emphasizing semantic distinctions.Positive candidates are selected using diffusion weight and cross-modal similarity; hard negatives have high similarity but low diffusion weight.Source: §A.2 · pp. 15–16

Masked Modality Reconstruction

  • Masked modality reconstruction uses scaled cosine error to preserve hidden modality information recovered through the semantic completion path.Cosine error focuses on feature direction rather than magnitude.Source: §A.2 · p. 16

Latent Decorrelation

  • Off-diagonal correlation penalties complement alignment and reconstruction by encouraging distinct latent coordinates to carry complementary features.The total objective jointly trains semantic alignment, modality recovery, and shared-space regularization.Source: §A.2 · pp. 16–17

Downstream Task Interfaces

  • A topology prefix prompter adapts shared representations for discriminative tasks, while the G2M-Adapter forms graph tokens for text and image generation.For text generation, tokens are prepended to language embeddings; for image generation, graph tokens are supplied to the diffusion U-Net through its cross-attention interface.Source: §A.2 · p. 17

A.3 PSEUDOCODE OF GRAPHBIND

This section points to Algorithm 1 for GraphBind’s overall procedure.

A.3 PSEUDOCODE OF GRAPHBIND

  • Algorithm 1 presents the overall procedure for GraphBind.The passage provides a pointer to the algorithm rather than its steps.Source: §A.3 · p. 17

A.4 DATASET DETAILS

The appendix describes datasets for product, social-post, video, image-caption, and artwork graphs, and outlines GraphBind’s pretraining and downstream adaptation procedure.

GraphBind Procedure

  • GraphBind’s procedure initializes modality encoders and topology features, then simulates absence and computes self semantics and neighborhood completion during pretraining.The procedure next calibrates a shared representation and optimizes the pretraining objective using original modality targets.Source: §A.4 · p. 18
  • Downstream adaptation applies a prompt and prediction head for node classification or link prediction, and supplies graph tokens to generative task interfaces.The described generative paths provide language prefixes or diffusion context tokens.Source: §A.4 · p. 18

Dataset Details

  • Grocery and Toys are product-category classification graphs, Movies is a few-shot product-genre graph, and DY and Bili Dance are short-video link-prediction graphs.The product graphs use purchase relations; the video graphs use coviewing or co-interaction edges.Source: §A.4 · p. 18
  • RedditS supports few-shot community classification, Flickr30k supports graph-to-text generation, and SemArt supports graph-to-image generation.RedditS nodes are posts; Flickr30k links image regions to caption phrases through grounding relations; SemArt nodes are paintings connected by shared metadata.Source: §A.4 · pp. 18–19
  • Ele-fashion and Sports are product graphs reserved for cross-domain pretraining, with cross-category co-purchase or complementary and co-purchase edges, respectively.Ele-fashion combines electronics and fashion products, while Sports contains sports equipment and accessories.Source: §A.4 · p. 18

A.5 COMPARED BASELINES

The appendix describes the compared baseline families and their configurations, then specifies pretraining and downstream evaluation protocols for discriminative and generative tasks.

A.5 COMPARED BASELINES

  • Mario is excluded because its evaluation centers on LLM-based graph reasoning, whereas this study focuses on shared graph representations for discriminative and generative tasks.The appendix states that eleven representative baselines are compared.Source: §A.5 · p. 19, §A.6 · p. 21
  • The MAG baselines span modality-specific graph propagation and fusion (MMGCN, MGAT), local and higher-order interest modeling (LGMRec), task-aware topology co-evolution (TMTE), and role-routed multimodal neighbors (RoleMAG).Source: §A.5 · p. 19
  • The GFM baselines cover task-tree reconstruction for shared graph pretraining (GIT), vector-quantized computation-tree vocabularies (GFT), graph tokenization with a scalable Transformer (OpenGraph), and Riemannian-space representation learning (RiemannGFM).Source: §A.5 · pp. 19–20
  • The MGFM baselines are UniGraph2, which aligns text and image features with graph structure, and PLANET, which separately models structure and modality semantics before integrating them.The appendix also specifies their encoder, expert, and codebook configurations.Source: §A.5 · p. 20, §A.6 · p. 21

A.6 EXPERIMENT SETUP

  • Pretraining uses six datasets with equal update shares, 768-dimensional CLIP text and image features, and simulated modality absence; four other datasets are reserved for adaptation and evaluation.Ele-fashion and Sports use sampled subgraphs, and visible modalities are additionally masked at 15% for reconstruction targets.Source: §A.6 · p. 20
  • Discriminative evaluation covers node classification, few-shot classification, and link prediction, with specified splits or episodic protocols and accuracy, macro-F1, MRR, and Hits@3 reporting.Link prediction uses one training negative and 1,000 evaluation negatives per positive edge.Source: §A.6 · pp. 20–21
  • G2Text generates captions from sampled multimodal neighbors on Flickr30k and reports BLEU-4 and ROUGE-L, with CIDEr supplied as a supplementary metric.All methods use OPT-125M with LoRA for this task.Source: §A.6 · pp. 20–22
  • G2Image uses graph-token-guided diffusion on SemArt and reports CLIP-S and DINO-S for generated images.The setup uses Stable Diffusion v1.5 and evaluates generated images with CLIP ViT-L/14 and DINOv2 features.Source: §A.6 · pp. 21–22

Related papers

Loading 2610.02881v1…