PaperlayerView exact revision on arXiv

Source-linked AI summary

Mario: Multimodal Graph Reasoning with Large Language Models

Yuanfu Sun, Kang Li, Pengkang Guo, Jiajin Liu, Qiaoyu Tan

arXiv:2603.05181v2cs.CV

TL;DR

Existing multimodal reasoning methods often ignore graph relations and face weak cross-modal consistency and heterogeneous modality preferences. Mario addresses these challenges with graph-conditioned alignment and modality-adaptive instruction tuning, outperforming baselines across supervised and zero-shot multimodal graph tasks.

  • Problem

    Treating multimodal data as isolated image–text pairs ignores relational information in multimodal graphs and leaves cross-modal consistency and modality preference unresolved.

  • Method

    Mario combines a graph-conditioned vision–language model for topology-aware image–text alignment with modality-adaptive graph instruction tuning and a learnable routing mechanism.

  • Results

    Mario consistently outperforms strong baselines across multimodal graph benchmarks in supervised and zero-shot node classification and link prediction.

  • Takeaways & Limitations

    Structure-aware alignment and node-specific modality routing support more reliable multimodal graph reasoning with LLMs.

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have opened new avenues for multimodal reasoning. Yet, most existing methods still rely on pretrained vision-language models (VLMs) to encode image-text pairs in isolation, ignoring the relational structure that real-world multimodal data naturally form. This motivates reasoning on multimodal graphs (MMGs), where each node has textual and visual attributes and edges provide structural cues. Enabling LLM-based reasoning on such heterogeneous multimodal signals while preserving graph topology introduces two key challenges: resolving weak cross-modal consistency and handling heterogeneous modality preference. To address this, we propose Mario, a unified framework that simultaneously resolves the two above challenges and enables effective LLM-based reasoning over MMGs. Mario consists of two innovative stages. Firstly, a graph-conditioned VLM design that jointly refines textual and visual features through fine-grained cross-modal contrastive learning guided by graph topology. Secondly, a modality-adaptive graph instruction tuning mechanism that organizes aligned multimodal features into graph-aware instruction views and employs a learnable router to surface, for each node and its neighborhood, the most informative modality configuration to the LLM. Extensive experiments across diverse MMG benchmarks demonstrate that Mario consistently outperforms state-of-the-art graph models in both supervised and zero-shot scenarios for node classification and link prediction. The code will be made available at https://github.com/sunyuanfu/Mario.

1. Introduction

Multimodal graph reasoning addresses the limits of treating image–text pairs independently by using graph structure to improve alignment and adapt modality use to each node. Mario combines structure-aware cross-modal alignment with modality-adaptive prompting for LLM reasoning.

  • Independent image–text processing leaves relational multimodal information unused, motivating multimodal graphs with textual and visual node attributes and structural edges.
  • Weak cross-modal consistency arises because images and texts may be noisy, incomplete, or semantically underspecified when considered in isolation.
  • Mario reports a 68% average gain over frozen CLIP and an additional 6% improvement over node-wise fine-tuning in cross-modal consistency.
  • Heterogeneous modality preference means that different nodes can benefit from different modality configurations, making a shared instruction template unsuitable.
  • Mario uses a graph-conditioned VLM for topology-aware image–text alignment and modality-adaptive instruction tuning to select informative modality views for each node.

2. Related Work

Related work has explored LLM-based graph reasoning and multimodal representation alignment, but effective multimodal graph modeling remains an open challenge. Mario is presented against this gap as a framework combining graph-conditioned alignment with adaptive instruction tuning.

  • LLM graph-reasoning studies enhance graph representations or provide graph-related text and tokens for prompting, instruction tuning, or in-context learning.
  • Mario’s framework overview combines graph-conditioned vision–language alignment with modality-adaptive graph instruction tuning for multimodal graph reasoning.
  • Multimodal graph benchmarks and multimodal knowledge-graph research have expanded the area, while effective multimodal graph models remain an open challenge.

3. Preliminary

The paper represents multimodal graphs as structured data whose nodes carry multiple modalities and whose edges are encoded in an adjacency matrix. It also frames instruction tuning as adapting LLMs through structured prompts.

  • A multimodal graph is denoted G = (V, E, A, X), with nodes associated with multiple modalities, including textual descriptions and images.
  • The adjacency matrix A ∈R^N×N records graph connectivity, with A_ij = 1 when (v_i, v_j) ∈E and 0 otherwise.
  • Each multimodal node feature is a structured pair containing textual and visual features.
  • Instruction tuning adapts LLM parameters through structured prompts that combine task-specific instructions with optional learnable soft tokens.

4. Methodology

Mario uses graph-conditioned multimodal encoding to align text and images with topology, then adaptively routes graph-aware modality views to the LLM. Its two-stage pipeline combines structure-aware contrastive learning with modality-adaptive instruction tuning.

  • Graph-conditioned vision–language model: Mario’s graph-conditioned vision–language model uses topology-aware multimodal mixing to refine modality-specific node representations across transformer layers.Each modality’s [CLS] summaries attend over graph nodes with graph-aware position bias, then are reinjected into the token stream for iterative refinement.
  • Graph-conditioned vision–language model: The mixer’s graph position bias encodes structural roles through head-specific learnable scalars indexed by shortest-path-distance buckets.The resulting structure-aware representations preserve modality-specific nuances while incorporating topology-aware signals.
  • Cross-modal contrastive learning: Mario trains the structure-aware modality embeddings with a symmetric, temperature-scaled InfoNCE objective using same-node text–image pairs as positives and cross-node pairs as negatives.Because the embeddings already incorporate neighborhood signals, the contrastive objective encourages representations that are both modality-aligned and structure-aware.
  • Modality-adaptive graph instruction tuning: Stage 2 constructs text, image, and multimodal instruction views by combining aligned node features with selected 1-hop and 2-hop neighbor evidence.Neighbor tokens and labels are used to form modality-specific prompt templates, with inference exemplars restricted to training nodes.
  • Modality-adaptive graph instruction tuning: MAPR learns routing probabilities over prompt templates using a performance-weighted LLM loss and KL regularization toward a loss-derived posterior.The teacher–student coupling shifts probability toward lower-loss templates and attenuates gradients from mismatched modalities.
  • Training and runtime: The multimodal mixer costs O(|V_s|^2d) per layer on sampled nodes, while Stage 2 executes three forward–backward passes per training sample.In practice, one to two mixer layers are sufficient, and the router reportedly converges in roughly half the epochs of a single-template baseline despite extra per-step computation.

5. Experiments

Mario is evaluated on multimodal graph benchmarks for node classification and link prediction, including single-focus, mix-training, zero-shot, ablation, and efficiency studies. Across these settings, its graph-conditioned alignment and modality-adaptive prompting generally improve performance, generalization, and convergence.

  • Experiment setup: Experiments cover node classification and link prediction across diverse multimodal graph datasets, with single-focus, mix-training, and zero-shot transfer settings.The study also evaluates the graph-conditioned model and modality-adaptive instruction tuning through ablations and efficiency analyses.
  • Overall performance: Mario achieves the highest accuracy across all datasets and both tasks in the single-focus setting.On CDs, node-classification performance rises from 56.45% for the best baseline to 63.43%, while link-prediction accuracy improves by an average of 4.73% across four datasets.
  • Overall performance: Mario outperforms augmented GraphLLMs on node classification, with average gaps of 5.48%, 11.00%, and 135.9% over LLaGA, GraphPrompter, and GraphGPT, respectively.The reported comparison favors directly interpreting graph-conditioned aligned multimodal features over image-to-text augmentation.
  • Generalization and transferability: Mario maintains strong performance under joint training, achieving average relative improvements of 2.88% in node classification and 2.57% in link prediction over the best baseline.The mix-training evaluation tests training on an equal four-way dataset mixture and evaluation on each domain individually.
  • Generalization and transferability: Mario achieves zero-shot gains of 1.64× on Toys → Movies node classification, 1.48× on Toys+Movies → CDs node classification, and 1.25× on Toys → Movies link prediction.The reported explanation attributes transfer performance to graph-invariant multimodal semantics and adaptive prompt selection across unseen graph topologies.
  • Ablation study: The graph-conditioned vision–language model surpasses GNNs and MLPs across datasets, reaching a +5.15% relative average gain on Movies, with runtime only 1.5× higher.The ablation attributes the advantage to fine-grained alignment of structured image-text features rather than global or structure-agnostic alignment.
  • Efficiency and visualization: Modality-adaptive graph instruction tuning converges 2.3× faster on Movies and 1.3× faster on Reddit than a single-template variant, while reaching lower final losses.Although each epoch takes approximately 1.5–2× longer, accelerated convergence yields comparable overall training time.
  • Efficiency and visualization: Router modality preferences show a non-uniform, locally clustered pattern, with some graph regions dominated by a single modality configuration.The visualization suggests that modality choices vary across nodes and can exhibit regional homophily.

6. Conclusion

The paper identifies cross-modal inconsistency and heterogeneous modality preference as underexplored challenges in multimodal graph reasoning. Mario addresses them with structure-aware alignment followed by node-specific modality routing, and experiments report stronger multimodal graph reasoning than strong baselines.

  • Conclusion: Mario targets cross-modal inconsistency and heterogeneous modality preference in multimodal graph reasoning.These are presented as the paper’s two underexplored challenges.
  • Conclusion: Mario combines graph-conditioned vision–language alignment with modality-adaptive graph instruction tuning through a lightweight node-specific router.The two stages align image-text features structurally and select modality configurations according to node preferences.
  • Conclusion: Experiments across multiple multimodal graph benchmarks show that Mario consistently outperforms strong baselines and supports more reliable multimodal graph reasoning.The conclusion states this outcome without limiting it to a single task or evaluation regime.

7. Appendix

The appendix supplies dataset statistics, implementation details, qualitative feature visualizations, and additional baseline comparisons. These materials clarify the experimental setup and support the reported differences between Mario and alternative multimodal graph models.

  • Dataset statistics: The datasets contain product or post nodes, relationship edges, category labels, and textual and visual attributes.The appendix identifies relationships such as co-purchase and co-comment interactions.
  • Implementation details: Image-to-caption conversion uses Qwen-VL-Chat to provide auxiliary textual information for GraphLLM baselines that cannot process image features directly.This procedure is used in text+vision baseline experiments.
  • Implementation details: The experiments use vLLM to serve LLaMA and LLaVA, while Stage 1 typically uses one or two GraphTransformer layers and samples approximately 10 nodes.Stage 2 typically selects 10–15 neighbors for contextual prompting.
  • Qualitative visualization: t-SNE projects randomly sampled aligned text and image representations into two dimensions for Movies and Reddit, enabling qualitative comparison of feature organization across three models.Movies occupy the top row and Reddit the bottom row of the six-panel visualization.
  • Additional baselines: MMGCN and MGAT provide no clear advantage over standard GNNs in the additional Movies and Arts comparisons.MGAT performs worse than all three GNN baselines on Movies, while gains on Arts are modest and remain below strong multimodal models and Mario.

7.5. Additional GNNs Zero-Shot Results

Mario is evaluated for zero-shot transfer across unseen multimodal graphs, retaining the trained GNN feature extractor while replacing only the classifier. It outperforms conventional GNNs and remains effective across different LLM backbones, with lightweight LoRA tuning providing further gains.

  • Additional GNNs Zero-Shot Results: Zero-shot transfer retains the trained GNN architecture and parameters while replacing only the classifier for the target dataset.This evaluates transfer without retraining the core graph feature extractor.
  • Additional GNNs Zero-Shot Results: Mario significantly outperforms all baselines across both transfer settings, demonstrating strong zero-shot adaptation capabilities.The evaluation covers node classification and link prediction accuracy.
  • Additional GNNs Zero-Shot Results: 41.00 NC accuracy on Toys →Movies exceeds GCN’s 3.29 by more than ten times.In Toys+Movies →CDs, Mario also reaches 54.32 NC accuracy and substantially outperforms all baselines.
  • LLM and Parameter Ablations: LoRA tuning improves node classification by about 1.7–3.0 points and link prediction by roughly 2–4 points across the evaluated datasets.Trainable parameters increase only from 18.9M (0.2346%) to 22.3M (0.2768%).
  • LLM and Parameter Ablations: Mario remains effective across LLaMA2, Vicuna-v1.5, FLAN-T5-XXL, and LLaMA3 backbones, indicating backbone-agnostic robustness.Mario-8B achieves 92.13% on Arts and 82.58% on Toys, while FLAN-T5-XXL achieves 92.08% and 81.63%, respectively.

7.8. Sensitivity Analysis

Sensitivity analyses show that projection layers and a controlled amount of neighbor context improve Mario’s performance. Excessive neighbor expansion yields marginal or unstable gains, while experimental variance remains small.

  • Projection Layer: Introducing a projection layer consistently improves performance over the no-projection baseline.The analysis examines the number of projection layers in Arts and Movies.
  • Neighbor Context: Adding a limited number of neighbors per hop significantly improves performance compared with using no neighbors.In Toys, neighbor context improves accuracy by over 10%.
  • Neighbor Context: Further increasing the number of neighbors produces marginal or unstable gains, making moderate structural context preferable.The result supports controlled integration of neighborhood information into prompts.
  • Variance: Variance is typically around ±0.07 or ±0.14 across three random runs.Partial variance results are reported using accuracy as the metric.

7.10. Quantitative Analysis of Modality Preference

The modality-preference analysis quantifies when individual or combined instruction templates succeed and describes how Mario selects structurally diverse neighbors. The model remains effective across graphs with widely varying heterophily ratios, while experiments use compute-matched comparisons.

  • Template Agreement: 70.96% of nodes are correctly classified by all three modality-template types, leaving 29.04% not jointly correct across all templates.The percentages are normalized among nodes correctly classified by at least one template.
  • Neighbor Selection: Mario uses multi-hop neighbor selection to retrieve distant relevant nodes and reduce reliance on locally uniform neighborhoods.The strategy exposes richer semantic context when forming prompts.
  • Structural Diversity: Benchmark heterophily ratios range from 0.04 on Reddit to 0.69 on CDs and 0.53 on Movies.Heterophily is defined as the fraction of edges linking nodes with different labels.
  • Structural Diversity: Mario maintains strong performance across widely varying heterophily regimes, supporting the reliability of feature-based similar-neighbor selection.This finding is reported across near-homophilic and strongly heterophilic datasets.
  • Evaluation Setup: Compute-matched comparisons report training cost and resulting performance for Mario and baselines under capped Stage-1 runs.The prompt templates cover node classification and link prediction with modality-specific node and neighbor features.

7.14. Case Study

Qualitative cases show Mario adapting its preferred modality configuration to the task and node pair, using graph context to down-weight misleading modality signals. Across Movies, Toys, and CDs examples, it is competitive with or sometimes superior to closed-source multimodal models.

  • Node Classification: In a Movies classification case, the image emphasizes after-sales information while the text describes a lecture series, so Mario down-weights the visually salient but task-irrelevant image.The decision is conditioned on the anchor’s multimodal features and local subgraph.
  • Model Comparison: Mario’s behavior is consistently competitive with, and sometimes superior to, strong closed-source L(V)LMs across the illustrated cases.The comparison includes ChatGPT-5.1-Thinking, Gemini-3-Pro, and Qwen3-Max.
  • Model Comparison: In the Figures 9–10 case, Mario alone assigns the ground-truth category while all other compared systems fail.This provides the clearest qualitative contrast reported in the case study.
  • Prompt Templates: The prompt templates retain the relevant modality-specific features for text-only, image-only, or text+image inputs.The templates are used for both node classification and link prediction.
  • Modality Preferences: Mario’s router identifies Text+Image, Text-only, or Image-only modality preferences across node-classification and link-prediction cases.The examples span Movies, Toys, and CDs datasets.
Loading 2603.05181v2…