Source-linked AI summary
UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs
Yufei He, Yuan Sui, Xiaoxin He, Yue Liu, Yifei Sun, Bryan Hooi
TL;DR
Existing foundation models overlook graph structure in multimodal graphs, while graph foundation models mainly target text-attributed graphs. UniGraph2 combines modality-specific encoders, a GNN, cross-domain multi-graph pre-training, and MoE alignment to learn unified representations. Experiments report that it outperforms state-of-the-art models across representation learning, transfer learning, and multimodal generation, with significant computational costs during pre-training.
Problem
Existing foundation models overlook graph structure in multimodal data, while graph foundation models primarily focus on text-attributed graphs rather than multimodal graphs.
Method
UniGraph2 uses modality-specific encoders, a GNN, generalized masked prediction, cross-domain multi-graph pre-training, and MoE alignment to learn unified multimodal graph representations.
Results
UniGraph2 outperforms state-of-the-art models across representation learning, transfer learning, and multimodal generative tasks on diverse graph domains and modalities.
Takeaways & Limitations
UniGraph2 provides a scalable and flexible solution for learning unified representations on multimodal graphs across diverse domains and modalities.
Takeaways & Limitations
UniGraph2 incurs significant computational costs primarily during the pre-training phase.
Abstract
from arXiv · showhide
Existing foundation models, such as CLIP, aim to learn a unified embedding space for multimodal data, enabling a wide range of downstream web-based applications like search, recommendation, and content classification. However, these models often overlook the inherent graph structures in multimodal datasets, where entities and their relationships are crucial. Multimodal graphs (MMGs) represent such graphs where each node is associated with features from different modalities, while the edges capture the relationships between these entities. On the other hand, existing graph foundation models primarily focus on text-attributed graphs (TAGs) and are not designed to handle the complexities of MMGs. To address these limitations, we propose UniGraph2, a novel cross-domain graph foundation model that enables general representation learning on MMGs, providing a unified embedding space. UniGraph2 employs modality-specific encoders alongside a graph neural network (GNN) to learn a unified low-dimensional embedding space that captures both the multimodal information and the underlying graph structure. We propose a new cross-domain multi-graph pre-training algorithm at scale to ensure effective transfer learning across diverse graph domains and modalities. Additionally, we adopt a Mixture of Experts (MoE) component to align features from different domains and modalities, ensuring coherent and robust embeddings that unify the information across modalities. Extensive experiments on a variety of multimodal graph tasks demonstrate that UniGraph2 significantly outperforms state-of-the-art models in tasks such as representation learning, transfer learning, and multimodal generative tasks, offering a scalable and flexible solution for learning on MMGs.
1 Introduction
UniGraph2 addresses the mismatch between multimodal foundation models that overlook graph structure and graph models that lack cross-graph, cross-task transfer. It extends masked prediction to multimodal graphs and introduces cross-domain multi-graph pre-training for unified, transferable representations.
- Motivation: Multimodal foundation models such as CLIP learn shared spaces for paired modalities but overlook many-to-many graph relationships and multimodal graph structure.This limitation affects domains such as social and e-commerce networks, where entities and interactions are important.
- Motivation: Existing multimodal graph methods train separately for individual graphs and tasks, preventing cross-graph and cross-task transfer without retraining or fine-tuning.
- UniGraph2: UniGraph2 generalizes UniGraph’s masked prediction framework to multimodal graphs by predicting missing text, image, or other node attributes from graph structure and available multimodal information.The resulting representations capture both node multimodal features and graph relationships.
- UniGraph2: UniGraph2 introduces cross-domain multi-graph pre-training to learn compact, transferable knowledge across graph datasets with varying modality and domain distributions.Its Mixture of Experts module dynamically selects experts to align node features across domains and modalities.
- Results: Extensive experiments show that UniGraph2 outperforms state-of-the-art models on representation learning, transfer learning, and multimodal generative tasks, especially with multiple graph domains.
2 Related Work
Prior multimodal graph models largely target specialized graphs or tasks, while graph foundation models seek transfer across graphs and tasks. Existing approaches do not yet provide a general multimodal graph representation model spanning domains, modalities, and tasks.
- Multimodal Learning: Earlier multimodal learning approaches unify modalities such as vision, language, and audio, including image-text models trained with contrastive learning or masked language modeling.
- Multimodal Graph Learning: Most multimodal graph models focus on knowledge, molecular, or brain graphs and use domain-specific designs for individual tasks and graphs.
- Multimodal Graph Learning: These specialized multimodal graph models generally cannot transfer across different graphs, modalities, or tasks.MMGL uses foundation models on multimodal graphs but focuses solely on generative tasks.
- Graph Foundation Models: Graph foundation model research has pursued transfer across graphs and tasks, including domain-specific models and methods using large language models for graph learning.
3 Preliminaries
The paper defines multimodal graphs as graphs whose nodes may have different subsets of modalities and formulates general representation learning as self-supervised pre-training that generates embeddings for unseen graphs without further training. UniGraph provides the text-attributed special case through masked prediction combining language-model and graph representations.
- Multimodal Graphs: A multimodal graph is G = (V, E, M, Ω), where nodes and edges form the graph and M maps each node to a subset of possible modalities.Nodes may contain multiple modality features, but need not contain every modality.
- Text-Attributed Graphs: Text-attributed graphs are a special case in which the available modality is limited to text.
- General Representation Learning: General representation learning on multimodal graphs seeks a self-supervised model that produces meaningful representations for new graphs without additional training or fine-tuning.
- General Representation Learning: For an unseen graph, the pre-trained function generates node embeddings that facilitate downstream tasks on that graph.
- UniGraph: UniGraph integrates a language model and GNN in one encoder and uses masked prediction to learn representations for text-attributed graphs.Its process masks textual features, propagates embeddings across the graph, combines masked textual and graph embeddings, and reconstructs masked tokens.
- UniGraph: The masked language modeling process obtains token probabilities, minimizes a masked language modeling loss, and uses the pre-trained encoder to generate embeddings for unseen TAGs.
- UniGraph: This inference procedure captures both structural and textual graph attributes.
4 The UniGraph2 Framework
UniGraph2 combines modality-specific encoding, MoE alignment, graph neural message passing, and domain-aware reconstruction to learn unified embeddings for multimodal graphs. Its pre-training jointly reconstructs masked node features and graph structure, while PPR sampling supports web-scale deployment.
- UniGraph2 processes multimodal graph features with modality-specific encoders, an MoE alignment module, and decoders within one unified framework.The framework targets nodes with diverse modalities and relationships represented by graph edges.
- Multimodal masking strategies: Modality-specific encoders transform raw data such as text and images into node feature vectors before masking.Features from all modalities associated with a node are averaged into its input representation.
- Multimodal masking strategies: Approximately 75% of nodes are randomly masked with a learnable token, requiring reconstruction from graph context and unmasked nodes.The masked features are then passed to the MoE module for cross-domain and cross-modality alignment.
- MoE alignment: The MoE dynamically routes each node to specialized experts using domain- and modality-dependent gating, producing aligned node embeddings.Expert outputs are combined with softmax-derived weights before graph propagation.
- GNN encoding: A GNN refines aligned embeddings by aggregating neighbor information, incorporating graph structure into the final node representations.Personalized PageRank sampling generates structurally significant local subgraphs for web-scale graphs.
- Multiple decoders: UniGraph2 uses domain-specific decoders for node-feature reconstruction and a shared shortest-path-distance decoder for structural reconstruction.Cosine similarity measures feature reconstruction, while shortest-path regression captures global connectivity; the losses are combined with coefficient λ.
- Inference: During inference, modality encoders, MoE alignment, and the GNN generate embeddings for classification, transfer learning, and generative tasks.The same streamlined pipeline is applied to unseen multimodal graphs.
5 Experiments
Experiments evaluate UniGraph2 across self-supervised representation learning, few-shot transfer, and multimodal generation. Across these settings, the model generally transfers well across graphs and modalities, outperforming or improving over the compared baselines.
- Experimental Setup: UniGraph2 is evaluated on self-supervised representation learning, few-shot transfer, and multimodal generative tasks across multiple graph datasets.The self-supervised evaluation uses linear probing on frozen pretrained embeddings, while the experiments include 14 datasets overall.
- Self-Supervised Representation Learning: UniGraph2 improves under multi-graph pre-training, whereas BGRL and GraphMAE2 experience performance drops relative to single-graph pre-training.This result addresses negative transfer in the in-distribution setting.
- Self-Supervised Representation Learning: UniGraph2 outperforms multimodal-only models such as CLIP and ImageBind and TAG-only UniGraph when graph structure and multimodal data are evaluated together.The comparison motivates foundation models designed specifically for multimodal graphs.
- Generalization Capability: UniGraph2 transfers knowledge to unseen graphs, including out-of-domain datasets, and achieves gains over NoPretrain while approaching models trained directly on target datasets.The reported benefit includes avoiding exhaustive task-specific training during downstream use.
- Few-Shot Transfer: UniGraph2 consistently outperforms the few-shot-transfer baselines, including Prodigy, OFA, and GCOPE, without requiring complex prompt graph designs.The paper characterizes its few-shot strategy as simple, efficient, and effective.
- Multimodal Generative Tasks: UniGraph2 embeddings improve multimodal generative results over MMGL’s default CLIP embeddings and remain robust across different neighbor encoding strategies.The embeddings do not rely on a specific neighbor encoding strategy.
- Model Analysis: Ablations indicate that the MoE, feature loss, and SPD loss each contribute positively to overall performance.The ablation compares variants without MoE, without feature loss, and without SPD loss.
- Model Analysis: Different modality-specific encoders achieve comparable performance, suggesting that UniGraph2 aligns features whether or not they were pre-aligned.The comparison includes CLIP, ImageBind, and T5+ViT encoders.
6 Conclusion
UniGraph2 integrates multimodal information and graph structures into a unified embedding space for multimodal graphs. Experiments report strong performance and generalization across graph domains and modalities.
- UniGraph2 combines multimodal information and graph structures in a unified embedding space using modality-specific encoders, a GNN, and MoE.
- UniGraph2 outperforms state-of-the-art models on classification, transfer learning, and multimodal generation tasks.
- Experiments demonstrate UniGraph2's generalization across diverse graph domains and modalities.
A Datasets
The evaluation uses multimodal, text-attributed, citation, product, knowledge, and Wikipedia graph datasets. These datasets cover node classification, link prediction, and multimodal content understanding settings.
- Text-attributed and link graphs: Citation and Internet-link datasets represent publications or pages as nodes connected by citations or reference links.Cora and PubMed use citation networks; ogbn-Arxiv and ogbn-Papers100M are directed citation graphs; Wiki-CS is an Internet link network.
- Prediction tasks and knowledge graphs: Product co-purchasing datasets support multiclass category prediction, while knowledge graphs contain entities, relation triples, and textual mentions.ogbn-Products uses 47 top-level product categories; FB15K237 and WN18RR are knowledge graphs with node text features used in the framework.
- Multimodal datasets: The multimodal datasets include product, book, fashion, and Wikipedia content graphs with text-image or many-to-many text-image relationships.Amazon-Sports, Amazon-Cloth, Goodreads-LP, Goodreads-NC, Ele-Fashion, and WikiWeb2M provide product, book, fashion, or Wikipedia multimodal data.
B Implementation Notes
The implementation evaluates frozen embeddings with linear probing and few-shot transfer across multiple dataset splits and repeated random seeds. Training uses standard deep-learning infrastructure and specified pre-training hyperparameters.
- Infrastructure: Experiments use Linux, 945G RAM, eight NVIDIA A100 GPUs, Python 3.11, PyTorch 2.0.1, DGL 1.1.2, Transformers 4.32.1, and CUDA 11.8.The detailed pre-training hyperparameters are listed in Table 9.
- Baselines: Baselines use public source code, with sampling adaptations for methods that cannot scale to large graphs.
- Dataset protocols: Dataset splits follow established or official protocols, with repeated random initialization or seeds used to report average accuracy.The protocols cover Cora, PubMed, Arxiv, Products, Wiki-CS, FB15K237, WN18RR, Amazon, Goodreads, Ele-Fashion, and WikiWeb2M.
- Evaluation: Linear probing trains a classifier on frozen embeddings by minimizing a loss over training embedding-label pairs and evaluates performance on validation or test data.
- Evaluation: Few-shot transfer follows UniGraph's in-context learning approach, with baseline-specific methods and ten repetitions using random seeds.
C Mixture of Experts (MoE) in Graph Learning
MoE distributes learning across specialized experts and uses a gating model to select or combine expert outputs. UniGraph2 applies MoE to align multimodal features across graph domains and modalities.
- MoE definition: Mixture of Experts distributes learning across specialized expert models, while a gating model selects which experts to activate for each input.
- Graph-learning applications: Graph-learning MoE systems can improve inference efficiency or address distribution shifts by assigning experts to distinct challenges.
- UniGraph2 application: UniGraph2 uses MoE to align text and image features from different graph domains into coherent cross-modal embeddings.