PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenMAG: A Comprehensive Benchmark for Multimodal-Attributed Graph

Chenxi Wan, Xunkai Li, Yilong Zuo, Haokun Deng, Sihan Li, Bowen Fan, Hongchao Qin, Ronghua Li, Guoren Wang

arXiv:2602.05576v1cs.LG

TL;DR

Existing MAG benchmarks have limited coverage of domains, encoders, models, and tasks, motivating a more rigorous standard for fair evaluation. OpenMAG unifies diverse datasets, encoders, models, tasks, and evaluation perspectives, deriving 14 insights that emphasize integrated structure-modality learning, flexible alignment, generalist models, robustness, and scalability.

  • Problem

    Existing MAG benchmarks have limited domain coverage, encoder flexibility, model diversity, and downstream-task scope, restricting standardized evaluation.

  • Method

    OpenMAG integrates 19 datasets across 6 domains, 16 encoders, 24 models, and 8 downstream tasks within five systematic evaluation perspectives.

  • Results

    The benchmark derives 14 fundamental insights spanning structure-modality integration, modality encoding, generalist models, robustness, and efficiency.

  • Takeaways & Limitations

    Future MAG research should pursue unified structure-modality architectures, flexible alignment, generalist models, robust representations, and scalable efficient designs.

Abstract

from arXiv · show

Multimodal-Attributed Graph (MAG) learning has achieved remarkable success in modeling complex real-world systems by integrating graph topology with rich attributes from multiple modalities. With the rapid proliferation of novel MAG models capable of handling intricate cross-modal semantics and structural dependencies, establishing a rigorous and unified evaluation standard has become imperative. Although existing benchmarks have facilitated initial progress, they exhibit critical limitations in domain coverage, encoder flexibility, model diversity, and task scope, presenting significant challenges to fair evaluation. To bridge this gap, we present OpenMAG, a comprehensive benchmark that integrates 19 datasets across 6 domains and incorporates 16 encoders to support both static and trainable feature encoding. OpenMAG further implements a standardized library of 24 state-of-the-art models and supports 8 downstream tasks, enabling fair comparisons within a unified framework. Through systematic assessment of necessity, data quality, effectiveness, robustness, and efficiency, we derive 14 fundamental insights into MAG learning to guide future advancements. Our code is available at https://github.com/YUKI-N810/OpenMAG.

1. INTRODUCTION

MAG learning combines graph topology with multimodal attributes but existing benchmarks provide limited coverage of domains, encoders, models, and tasks. OpenMAG addresses these gaps with a unified benchmark and five-dimensional evaluation framework.

  • Existing MAG benchmarks are limited in domain coverage, encoder flexibility, model diversity, and downstream-task scope.
  • OpenMAG integrates 19 datasets across 6 domains, 16 modality encoders, 24 MAG models, and 8 downstream tasks for standardized evaluation.
  • The benchmark evaluates MAG learning through necessity, data quality, effectiveness, robustness, and efficiency.
  • OpenMAG summarizes systematic experiments as 14 insights and releases an open-source library for evaluating custom methods and datasets.

2. Preliminaries

MAGs represent nodes with multimodal attributes and graph topology, using encoding, fusion, interaction, and task adaptation stages. OpenMAG evaluates both graph-based and modality-based learning objectives.

  • A multimodal-attributed graph is G = (V, E, X_M), where nodes, edges, and raw attributes from text and visual modalities are represented explicitly.
  • MAG learning encodes modality-specific attributes, fuses them into unified node representations, propagates information across neighbors, and projects outputs into task-specific spaces.
  • OpenMAG covers graph tasks including node classification, link prediction, and node clustering, alongside modality matching, retrieval, alignment, and graph-to-text and graph-to-image generation.

3. BENCHMARK DESIGN

OpenMAG standardizes a benchmark pipeline spanning diverse datasets, flexible encoders, representative MAG models, and five evaluation protocols. Its design targets fair assessment across real-world domains and learning objectives.

  • Dataset Overview For OpenMAG: The benchmark spans e-commerce, social media, art, video recommendation, book recommendation, and image-network datasets with domain-specific graph construction.
  • Encoder Module: OpenMAG supports frozen feature extraction and end-to-end fine-tuning, including representative text, vision, and vision-language encoders.
  • MAG Models: Its model library organizes MAG methods into graph-enhanced, multimodal-enhanced, and MLLM-enhanced paradigms.
  • Evaluation Protocol: The evaluation protocols measure necessity, data quality, effectiveness, robustness, and efficiency across discriminative, generative, noisy-data, and resource-cost settings.
  • Evaluation Protocol: Efficiency assessment combines theoretical time complexity, convergence epochs, and peak GPU memory to identify performance-resource trade-offs.

4. EXPERIMENTS AND ANALYSIS

OpenMAG evaluates multimodal-attributed graph learning across modality integration, graph structure, feature quality, model effectiveness, robustness, and efficiency. The experiments show complementary multimodal information, appropriately controlled topology, unified encoders, fine-tuning, and multimodal-enhanced models are important, while excessive structural aggregation and some architectures impose trade-offs.

  • 4.1. Validation of Structure-Modality Necessity: Multimodal approaches consistently outperform unimodal baselines, with textual features preferred in e-commerce and visual features in social scenarios.The results attribute these preferences to product descriptions and user interaction, respectively.
  • 4.1. Validation of Structure-Modality Necessity: Moderate graph-structure aggregation improves multimodal generation consistency, whereas excessive neighbors introduce semantic noise and reduce generation fidelity.Table 2 compares structure-agnostic generation with non-zero-neighbor Graph-QFormer settings on SemArt and Toys.
  • 4.2. Assessment of Multimodal Feature Quality: Fine-tuning strategies outperform fully frozen encoders, mitigating domain shift but increasing computational overhead.Four encoder-training settings are compared on node classification using an MLP baseline.
  • 4.2. Assessment of Multimodal Feature Quality: Unified encoders jointly pretrained on image-text pairs provide better cross-modal alignment than independent unimodal encoders for modality retrieval.Figure 4 evaluates encoding strategies on Flickr30k, where independent encoders often suffer modality misalignment.
  • 4.3. Effectiveness Analysis: Graph-enhanced models dominate most graph-learning scenarios, while multimodal-enhanced models excel in node classification and generative tasks and simple MLPs can excel at modality retrieval.The experiments compare traditional GNNs, graph-enhanced, multimodal-enhanced, MLLM-enhanced models, and an MLP baseline.
  • 4.5. Complexity and Practical Efficiency Analysis: Multimodal-enhanced models offer the most favorable accuracy-efficiency trade-off, whereas graph-enhanced and MLLM-based models often face quadratic complexity.The efficiency conclusions identify latency and memory overheads as practical concerns for graph-enhanced models.

5. CONCLUSION AND FUTURE DIRECTIONS

OpenMAG’s evaluation identifies complementary requirements for MAG learning: integrated multimodal-structural modeling, flexible aligned encoding, broad generalization, robustness to realistic noise, and scalable efficiency. These findings motivate unified, adaptable architectures that balance representation quality with deployment feasibility.

  • Unification of Multimodal and Graph Structures: MAG models require deep integration of multimodal semantics and graph structure because neither unimodal semantics nor pure topology suffices alone.The benchmark identifies rigid fusion mechanisms as a central challenge and calls for architectures that blend semantic and structural guidance.
  • Flexible and Aligned Modality Encoding: Flexible fine-tuning and precise cross-modal alignment are indispensable for high-quality node embeddings, but alignment can be computationally prohibitive.The proposed direction is to develop lightweight alignment techniques and more flexible multimodal encoding processes.
  • Unified Generalist MAG Models: Current MAG models often specialize in isolated tasks and fail to generalize, motivating unified generalist models for diverse structural semantics and downstream scenarios.The conclusion frames model fragmentation as a limitation of requiring distinct architectures for different learning objectives.
  • Robustness in Realistic Scenarios: Models relying strictly on local message passing are fragile under realistic semantic corruption and structural unreliability.Future models should explore structure-agnostic pathways and self-supervised denoising to improve stability under dual noisy conditions.
  • Scalable and Efficient Architectures: Quadratic complexity constrains MAG deployment, making scalable architectures and model compression important for balancing reasoning capability with feasibility.The conclusion emphasizes maintaining efficiency on large-scale graphs while transferring heavy capabilities to lightweight models.

A. Dataset Details

OpenMAG documents datasets spanning multiple domains, with graph construction and multimodal attributes tailored to each application. It also includes visual, textual, and graph-learning components that support standardized MAG evaluation.

  • Dataset organization: OpenMAG organizes datasets from e-commerce, social media, video, and image-related domains with application-specific graph structures and multimodal attributes.Examples include co-viewing, co-purchasing, semantic grounding, and compatibility relationships.
  • Video datasets: Video datasets represent content items as nodes and use co-consumption, co-viewing, or co-interaction behavior to define graph edges.Textual features come from titles, descriptions, tags, lyrics, metadata, or comments, while visual features come from covers, keyframes, thumbnails, or video frames.
  • E-commerce datasets: Amazon product datasets connect items through purchasing or visual-compatibility relationships and use product descriptions alongside product images for node classification.Grocery, Toys, and Cloth labels correspond to fine-grained product categories, toy types, or fashion styles.
  • Encoders and models: The benchmark includes trainable graph-structured visual encoding and trainable sentence-embedding backbones alongside conventional graph and multimodal models.ViG represents image patches as graph nodes, while bert-base-nli-mean-tokens can be fine-tuned for sentence embeddings.

D. Downstream Tasks

OpenMAG evaluates MAG models across graph-based and modality-based tasks. The task suite covers topology-aware representation learning, cross-modal alignment, and multimodal generation.

  • Task taxonomy: OpenMAG’s downstream suite divides tasks into graph-based tasks for topology and node representations and modality-based tasks for cross-modal interaction and generation.The benchmark associates each task with formally defined evaluation metrics.

D.1. Graph-Based Tasks

Graph-based tasks evaluate whether models learn useful node representations from multimodal attributes and graph structure. They cover supervised classification, edge inference, and unsupervised semantic grouping.

  • Node Classification: Node Classification learns node embeddings and predicts class probabilities through a projection head and Softmax layer.Training minimizes a supervised classification objective.
  • Link Prediction: Link Prediction ranks node pairs by similarity to distinguish true graph edges from negative samples.The task requires aligning structural proximity with semantic similarity across modalities.
  • Node Clustering: Node Clustering partitions nodes into semantic groups without labels using disentangled graph views and dual-frequency signal fusion.The protocol combines homophilic and heterophilic views with high-pass and low-pass filtered signals.

D.2. Modality-Based Tasks

Modality-based tasks test semantic relationships between modalities and generation conditioned on multimodal graph context. They range from matching and retrieval to fine-grained alignment and graph-conditioned text or image synthesis.

  • Modality Matching: Modality Matching determines whether paired inputs from different modalities are semantically correlated in a unified embedding space.Models learn to distinguish positive ground-truth pairs from mismatches.
  • Modality Retrieval: Modality Retrieval ranks candidate instances in one modality for a query from another, including image-to-text and text-to-image retrieval.Similarity scores are computed in a shared latent space.
  • Modality Alignment: Modality Alignment measures fine-grained geometric consistency between paired visual and textual feature distributions.The objective aligns detailed semantic elements such as visual regions and textual phrases.
  • Graph-to-Text: G2Text generates natural-language descriptions from graph-structured multimodal inputs containing many-to-many neighbor relationships.The workflow processes multimodal neighbor information through neighbor encoding.
  • Graph-to-Image: G2Image synthesizes images conditioned on multimodal graphs that combine textual descriptions with graph associations.The InstructG2I workflow uses semantic PPR-based neighbor sampling to select informative neighbors.

E. Evaluation Strategies

OpenMAG evaluates MAG systems with task-appropriate metrics spanning classification, ranking, clustering, cross-modal alignment, and generation. These measures capture both predictive correctness and semantic or structural fidelity.

  • Accuracy and F1-Score evaluate node classification, with F1 balancing precision and recall under long-tailed class distributions.Accuracy is also used for Modality Matching, while F1 targets the trade-off between false positives and false negatives.
  • MRR and Hits@K assess ranking quality in Link Prediction and Modality Retrieval by measuring first-correct-result position and top-K success.MRR rewards placing the ground truth near the top, whereas Hits@K measures recall at a fixed cutoff.
  • NMI and ARI evaluate Node Clustering by comparing predicted assignments with ground-truth classes, with ARI adjusting for chance grouping.NMI measures normalized mutual dependence, while ARI provides a stricter structural-consistency check.
  • CLIP-Score measures cross-modal semantic consistency for Modality Alignment and Graph-to-Image Generation without human annotations.It uses cosine similarity between pretrained image and text encoder representations, clipped at zero and scaled by 100.
  • BLEU-4, ROUGE-L, and CIDEr evaluate Graph-to-Text generation through n-gram precision, longest-common-subsequence recall, and TF-IDF-weighted consensus.CIDEr emphasizes distinctive, semantically important terms and is described as correlating better with human judgment.
  • DINOv2-Score evaluates Graph-to-Image visual fidelity, object layout, and structural consistency using pretrained feature embeddings.Unlike CLIP-Score’s semantic focus, it compares generated and reference images through cosine similarity in DINOv2 feature space.

F. Experiment Environment

The experiments use a high-memory GPU workstation and a standardized software environment. Hyperparameter organization separates task-level settings shared across models from model-specific settings.

  • Experiments run on Intel Xeon processors, NVIDIA RTX 6000 Ada GPUs with 96GB VRAM, and 256GB RAM using CUDA 12.9.The software stack uses Python 3.10.18 and PyTorch 2.8.
  • The benchmark distinguishes task-level hyperparameters shared within downstream tasks from model-level hyperparameters tailored to individual architectures.This separation is intended to support fair and reproducible comparisons.

G.1. Task-level Hyperparameters

Task-level configurations standardize optimization and training conditions across graph, alignment, and generative workloads, while model-specific settings accommodate architectural differences. The benchmark also fixes common encoder and dimensionality choices for fair comparison.

  • Graph-Based Tasks: Graph-based tasks use a 5×10−3 learning rate, batch size 512, 1×10−5 weight decay, and 25-neighbor aggregation.Node Classification trains for 100 epochs, whereas Node Clustering trains for 500 epochs.
  • Modality-Based Tasks: Alignment tasks use contrastive learning with temperature τ=0.07, a 1×10−3 learning rate, batch size 256, and 500 training epochs.Early stopping uses patience between 10 and 25 epochs.
  • Modality-Based Tasks: Generative G2Text experiments use OPT-125M or LLaMA-3.2-1B-Instruct with frozen CLIP and XLM-R encoders, 20 epochs, and per-device batch size 4.Visual features are mapped to 256 tokens for LLaMA and adaptively pooled to 4 tokens for OPT.
  • Shared Settings: The benchmark uses clip-vit-large-patch14 as the default encoder and fixes node embeddings at 768 dimensions across tasks.Adam or AdamW optimization and repeated initialization checks support standardized evaluation.
  • Model-Level Settings: Model-level hyperparameters are tuned for architectural designs, including differing depths and hidden dimensions for conventional GNN variants.Advanced MAG models have method-specific search spaces summarized in Table 8.

H. Theoretical Complexity Analysis Details

OpenMAG analyzes efficiency by relating computational cost to graph size, modality count, feature dimensions, layers, token length, and parameter scale. Graph-enhanced models are generally more scalable, whereas MLLM-based methods incur substantial sequence and memory costs.

  • Graph-enhanced models typically scale linearly with edge count O(|E|) through sparse message passing, while explicit structure learning or dense clustering can reach O(|V|^2).GraphMAE2 and DGF follow the linear pattern, whereas DMGC incurs quadratic complexity.
  • Multimodal-enhanced models incur costs tied to modality count M and may add O(d^2) attention computation through complex fusion mechanisms.Cross-modal interaction modeling can raise complexity beyond standard fusion operations.
  • MLLM-enhanced models are dominated by Transformer self-attention O(S^2) and large parameter counts Θ, producing high memory and compute demands.Long-context generative methods face greater training and inference overhead than traditional GNNs, often requiring high-end hardware.
  • Efficient alignment and adapter-tuning methods can keep complexity related to graph size linear, but long-context generation remains resource-intensive.The analysis contrasts mitigation strategies such as MLaGA and UniGraph2 with the overheads of GraphGPT-O.
Loading 2602.05576v1…