PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Unified Multimodal Graph Foundation Model: A Bridge-Router-Adapter Based Approach

Sirui Zhang, Yubing Zhou, Xunkai Li, Zekai Chen, Shumeng Li, Wang Luo, Yinlin Zhu, Yujin Gao, Rong-Hua Li

arXiv:2609.06668v1cs.LG

TL;DR

Existing multimodal graph methods may entangle neighborhood scopes or route modalities within a fixed scope, although relevance varies across tasks and instances. BRAIN constructs and routes scope–composition candidates, then adapts the result for downstream tasks; it reports leading comparisons across datasets and task families.

  • Problem

    Existing methods may collapse scope-specific graph context into unified representations or route modalities within a fixed graph scope, despite variation in relevance across tasks and instances.

    Source: Introduction · p. 1

  • Method

    BRAIN constructs candidates across neighborhood scopes and modality compositions, routes them by task and instance, and specializes the routed representation with a residual Adapter.

    Source: Introduction · p. 1

  • Results

    Across nine datasets and four task families, BRAIN ranks first on nine of ten discriminative comparisons and averages 14.72% relative improvement across four graph-to-text and two graph-to-image metrics.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Reported results show strong performance across datasets and task families, with clear gains on relational matching and generation.

    Source: Conclusion · p. 7

    The formulation is scoped to multimodal attributed graphs with textual and visual node attributes.

    Source: Preliminaries · p. 2

Abstract

from arXiv · show

Multimodal graphs couple node attributes in different modalities, such as text and images, with relational structure, enabling topological structure and cross-modality attributes to be modeled jointly. Multimodal graph foundation models seek unified representations from such data that transfer across different graph domains and downstream tasks. However, existing methods exhibit two fundamental limitations. (1) Cross-Scope Context Entanglement. They merge scope-specific graph contexts into a unified representation, obscuring their distinctions during multimodal construction. (2) Scope-Ignorant Modality Routing. They route modalities within a fixed graph scope, overlooking how modality relevance varies across neighborhood ranges. To address these challenges, we propose BRAIN, a unified model that focuses on graph context that combines neighborhood scope with modality composition. BRAIN comprises a scope-conditioned Bridge that combines structural information spanning local-to-global neighborhood scopes with different modality compositions; a hierarchical Router that estimates the relevance between the scope and the task, and selects compositions separately within each scope, allowing modality utility to vary with graph range; and a lightweight residual Adapter that further specializes the routed embedding for downstream prediction. BRAIN is trained through multi-graph pretraining followed by task-specific adaptation. Experiments across nine datasets and four task families demonstrate its broad effectiveness, improving node-classification and link-prediction performance by up to 4.73% relative to the strongest baseline, while achieving an average relative improvement of 14.72% across four graph-to-text and two graph-to-image metrics.

Introduction

BRAIN treats graph context as the joint choice of neighborhood scope and modality composition, addressing prior methods’ scope entanglement and fixed-scope modality routing. Its Bridge, Router, and Adapter construct, select, and specialize context candidates; the reported results span nine datasets and four task families.

Introduction

  • Prior methods can collapse scope-specific contexts into a unified representation or route modalities within a fixed graph scope.These limitations leave neighborhood range and modality composition insufficiently distinguished or jointly selected.Source: Introduction · p. 1
  • BRAIN models context as neighborhood scope paired with modality composition, with their relevance dependent on task and instance.It constructs candidates across local-to-global scopes and modality compositions, routes them, then specializes the routed representation; multi-graph pretraining is followed by task-specific adaptation.Source: Introduction · p. 1
  • The scope-conditioned Bridge derives text-dominant, image-dominant, and multimodal candidates for each neighborhood scope while preserving scope identity.This separates scope aggregation from composition specialization without repeatedly reconstructing structural context.Source: Introduction · pp. 1–2
  • The hierarchical Router estimates scope relevance before selecting modality compositions within each scope, and a residual Adapter specializes the routed representation.The Router makes modality preference conditional on graph range.Source: Introduction · p. 2
  • BRAIN ranks first on nine of ten discriminative comparisons and achieves 14.72% average relative improvement across four graph-to-text and two graph-to-image metrics.The results cover nine datasets and four task families.Source: Introduction · p. 2

Preliminaries

The preliminaries frame BRAIN around multimodal attributed graphs and task-conditioned choices among neighborhood scopes and modality compositions. They also position its setting against prior multimodal graph learning and graph foundation models, which commonly use a predefined graph view or structural range.

Preliminaries

Methodology

BRAIN constructs alternatives that preserve neighborhood scope and modality composition, then lets each downstream task route among them and adapt the result. Its two-stage training separates reusable representation learning from task-specific routing and adaptation.

Scope-Conditioned Multimodal Bridge

  • The Bridge builds a shared representation per scope, then forms text-, image-, and multimodal candidates; semantic attention selects relevant context, while structural propagation preserves within-range connectivity.Contextual aggregation is performed once per scope rather than repeated for each composition.Source: Overview · p. 3, Scope-Conditioned Multimodal Bridge · p. 4

Overview

  • BRAIN uses two stages: Stage 1 jointly samples anchor nodes across graph datasets to pretrain shared node- and scope-representations; Stage 2 optimizes composition specialization, routing, adaptation, and task heads.Source: Overview · p. 3

Hierarchical Context Router and Adapter

  • The Router first scores scope relevance, then scores compositions within each scope; their product weights candidate aggregation before a task-conditioned residual Adapter specializes the routed representation.This scope-first factorization allows modality preference to vary across graph ranges.Source: Hierarchical Context Router and Adapter · p. 4

Experiment

Across nine datasets and four task families, experiments evaluate BRAIN’s discriminative and generative performance, Stage 1 data efficiency, module and routing behavior, and computational cost. BRAIN leads reported comparisons across both discriminative and generative tasks.

Overall Effectiveness

  • BRAIN ranks first on nine of ten discriminative dataset–metric comparisons and second on the remaining one, leading every reported LP dataset and metric.Its strongest NC advantage is on Movies; the paper reports smaller margins on some NC datasets.Source: Overall Effectiveness · p. 5
  • BRAIN leads all G2Text and G2Image baselines, with gains of 19.9% in CIDEr and 29.3% in DINO Similarity.Figure 3 reports BLEU-4, ROUGE-L, CIDEr, and BERTScore for G2Text, and CLIPScore and DINOv2 similarity for G2Image.Source: Overall Effectiveness · p. 5, Figure 3 · p. 6

Data Efficiency of Stage-1 Pretraining

  • On DY, pretrained initialization improves MRR at every supervision level, including gains of 0.2123 and 0.2308 at 10% and 20% supervision; Bili_Food’s largest gain is 0.0781 MRR at 20%.Stage 1 benefits are most consistent on LP, but depend on downstream task and supervision amount.Source: Data Efficiency of Stage-1 Pretraining · p. 6, Figure 4 · p. 6
  • On Movies, pretrained initialization is slightly weaker at the lowest supervision levels, but at 80% supervision Accuracy rises from 51.84% to 53.96%.The reported crossover indicates that the pretraining benefit is not uniform across supervision levels.Source: Data Efficiency of Stage-1 Pretraining · p. 6

Ablation and Routing Analysis

  • Removing graph context causes the largest and most consistent degradation; collapsing composition candidates, retaining one scope, routing controls, and removing the Adapter also reduce performance.The findings support complementary contributions from scope-preserving candidate construction, context-dependent selection, and post-routing specialization.Source: Table 2 · p. 6, Ablation and Routing Analysis · p. 7
  • Routing preferences vary by dataset: Grocery and RedditS favor local context, Movies balances scopes while preferring multimodal candidates, DY is image-dominant, and Bili_Food favors broader text-dominant context.Sports relies mainly on local text and image evidence; the reported patterns show no single range or composition is optimal across datasets.Source: Ablation and Routing Analysis · p. 7, Figure 5 · p. 7

Efficiency and Complexity

  • BRAIN has 7.33M trainable parameters in the profiled NC configuration, with the Router accounting for 8.3%; its three scopes and three compositions yield a fixed candidate bank of K = 9.The main additional cost is constructing and scoring candidates, while the Adapter is evaluated once after routing; top-k or hard routing is identified as a future acceleration target.Source: Efficiency and Complexity · p. 7
  • BRAIN achieves the highest accuracy with 8.12 minutes of Stage 2 training and uses 2.37 GiB peak GPU memory in Stage 2, the second-lowest footprint among compared methods.Stage 2 memory is 0.17 GiB above MMGCN and 32.9%–89.2% below the remaining baselines; Stage 1 peaks at 18.76 GiB.Source: Efficiency and Complexity · p. 7, Figure 6 · p. 7

Conclusion

BRAIN builds graph-scope and modality-composition candidates, routes them conditionally, and specializes the routed representation for downstream prediction. Across nine datasets and four task families, it reports gains in relational matching and generation, with task-dependent low-resource benefits.

Conclusion

  • Across nine datasets and four task families, BRAIN reports strong performance with clear gains on relational matching and generation.The conclusion also reports task-dependent data-efficiency benefits in low-resource experiments.Source: Conclusion · p. 7
  • BRAIN combines attention-based semantic aggregation and GCN-based structural propagation to construct candidates spanning graph scopes and modality compositions.A hierarchical conditional Router mixes the candidates, and a residual Adapter specializes the routed representation for downstream prediction.Source: Conclusion · p. 7
  • Two-stage training first learns reusable multimodal and structural representations across graphs, then optimizes candidate construction, routing, adaptation, and task heads with supervision.Source: Conclusion · p. 7
  • Ablations and routing analyses validate graph context, structured candidate construction, conditional selection, and domain-specific context preferences across heterogeneous datasets and downstream tasks.Source: Conclusion · p. 7
Loading 2609.06668v1…