PaperlayerView exact revision on arXiv

Source-linked AI summary

Chimaera: A Mixture-of-Graph-Experts Architecture for Cross-Task and Cross-Dataset Graph Learning

Jonathan Frank, David Richerby, Ansgar Scherp

arXiv:2609.08709v1cs.LG

TL;DR

Graph models face feature- and label-space dependencies that constrain transfer across tasks and datasets. Chimaera combines graph foundation models with a mixture of experts, and its reported results show strong transfer, with linear GNNs tending to outperform PRODIGY in cross-task or pre-training scenarios.

  • Problem

    GNNs trained on a specific dataset and task remain tied to their feature and label spaces, limiting their use across datasets and tasks.

    Source: §I · p. 1

  • Method

    Chimaera combines graph foundation models, including prompting and linear GNNs, with a mixture-of-experts architecture and supports node-, link-, and graph-level classification.

    Source: §I · pp. 1–2

  • Results

    In more than half of cross-task or pre-training scenarios, Chimaera performs close to models trained directly on the target dataset and task, and linear GNNs tend to outperform PRODIGY in those settings.

    Source: §VI · p. 7

  • Takeaways & Limitations

    Linear GNNs in Chimaera transfer across tasks and datasets with only a minimal performance decrease relative to models trained specifically for those tasks.

    Source: §VII · p. 8

    The authors train models for one task type, and PRODIGY performs worse on link-level tasks when its link projection is untrained; they also do not train all components simultaneously without pre-training.

    Source: §VI · p. 8

Abstract

from arXiv · show

Designing foundation models for graphs is challenging due to the irregular structure of graphs and the different sizes and characteristics of embeddings. Chimaera integrates mixture-of-experts with graph foundation models (GFM). It integrates different GFM architectures, such as graph prompts and linear GNN models. Large language models are used to generate embeddings, and experts can be trained and combined following different strategies, GFMs, embeddings, etc. Furthermore, Chimaera extends existing linear GNNs to support link-level and graph-level tasks in addition to node-level tasks. Empirical analyses are performed on same-task and cross-task experiments with node, link, and graph classification tasks using six benchmark text-attributed graph datasets. The experiments demonstrate the effectiveness of Chimaera and its capabilities for transfer across tasks and datasets. Further insights include the need to use both large and small language models to generate embeddings for the experts, a strong cross-task transferability of simple but effective linear GNNs, and using few samples only to provide strong results.

I. INTRODUCTION

Graph foundation models must handle irregular graphs, heterogeneous feature embeddings, and transfer across tasks and datasets. Chimaera combines existing GFM designs through a mixture-of-experts architecture and is evaluated for transfer across tasks and datasets.

I. INTRODUCTION

  • Graph learning poses challenges because graphs are irregular, lack a standardized vocabulary, and use features with different kinds and sizes of embeddings.The paper contrasts sparse TF-IDF features with dense neural embeddings such as BERT.Source: §I · p. 1
  • GNNs trained for a specific dataset and task remain tied to its feature and label spaces, while LLM embeddings and prompting or linear GNNs address those dependencies in text-attributed graphs.A mixture of experts can combine GNNs according to their strengths, but does not by itself remove feature-space dependence.Source: §I · p. 1
  • Chimaera combines existing GFMs, including prompting and linear GNNs, with a mixture-of-experts architecture.It extends linear GNNs beyond node classification to link- and graph-level tasks.Source: §I · pp. 1–2
  • Chimaera combines different expert GNNs and embeddings through a gating function, then uses a GFM classifier for node-, link-, or graph-level classification.The encoders may include the original dataset encoder and neural encoders such as BERT.Source: §I · pp. 1–2, Figure 1 · p. 1
  • The experiments compare encoder, expert, gating, and classifier choices, as well as co-training, pre-training, same-task, cross-task, and few-shot settings.The authors report 64 settings on six benchmarks and four tasks.Source: §I · p. 2
  • In more than half of cross-task or pre-training scenarios, Chimaera performs close to models trained directly on the target dataset and task.The paper reports experiments across six benchmark datasets and four tasks.Source: §I · p. 2

II. RELATED WORK

Related work spans GNN-, LLM-, and hybrid graph foundation models, alongside graph mixture-of-experts methods. Chimaera differs by targeting transfer across multiple graph tasks and datasets.

II. RELATED WORK

  • Graph foundation models include GNN-based approaches such as PRODIGY, GraphAny, and TrainlessGNN, LLM-based approaches such as GLM and GPT4Graph, and GNN–LLM hybrids such as OFA and GOFA.The cited hybrid methods use textual features or prompts.Source: §II · p. 2
  • Existing graph MoE work largely targets performance on specific graph tasks, whereas Chimaera aims to transfer knowledge across tasks and datasets.Prior examples include GMoE and DA-MoE, which adapt graph networks to training graphs or complex patterns.Source: §II · p. 2
  • MoE combines experts through a mixture function; prior designs include sparse top-k routing, balanced expert workloads, and graph-structure-based gating.The section cites Fedus et al. on single-expert routing and balanced workloads, and describes GMoE and DA-MoE.Source: §II · p. 2

III. CHIMAERA MOE-ARCHITECTURE FOR GFMS

Chimaera combines encoder and expert embeddings through a gating function and supplies the result to a graph foundation model classifier. Its design adapts linear GNNs and graph prompting to multiple graph-task levels.

III. CHIMAERA MOE-ARCHITECTURE FOR GFMS

  • Encoders produce input embeddings for GNN experts, whose same-dimensional outputs and a skip embedding from an encoder feed the gating function.The experts may differ in input dimension because encoder output sizes vary.Source: §III · p. 2
  • Chimaera’s component choices—encoders, experts, gating functions, and classifiers—form an optimization space not typically found in other GNN or GFM architectures.The considered classifiers are linear GNNs and prompt-based PRODIGY.Source: §III · p. 3
  • The gate forms the shared embedding by averaging or concatenating expert outputs, or by weighting experts according to input-conditioned mixture scores.GMoE uses noisy top-k gating, while DA-MoE uses a structure-based gating network.Source: §III · pp. 2–3
  • Linear GNNs are adapted through task graphs whose entities represent nodes, edges, or graphs, with task-specific embeddings used for classification.Link embeddings concatenate endpoint representations, while graph embeddings use mean pooling.Source: §III · pp. 2–3
  • For prompting, PRODIGY represents examples, queries, and classes in a task graph and classifies query nodes by cosine similarity to class nodes.Chimaera supplies data-node embeddings from the gate; graph tasks use mean pooling, and link tasks combine endpoint embeddings with max pooling over node embeddings.Source: §III · pp. 3–4
  • Training proceeds in stages: experts are selected using validation results and frozen, then the gate and chosen GFM classifier are trained and frozen in turn.The staged approach reduces overall training time and complexity versus simultaneous training, and allows experts to be trained in parallel.Source: §III · p. 4
  • With gates and experts run simultaneously, the authors describe gating overhead as negligible; linear-GNN inference is O(|V|), while prompt classification uses a task graph treated as much smaller than the original graph.The task-graph complexity approximation is O(ℓ0(|E|d + nd^2)).Source: §III · p. 4

IV. EXPERIMENTS

The experiments evaluate Chimaera with text-attributed graph datasets, varied language-model embeddings and GNN experts, and multiple classifier and transfer settings. The protocol includes few-shot testing and analyses of gating behavior and hyperparameters.

IV. EXPERIMENTS

  • Six TAGLAS datasets cover node classification, link prediction, edge reasoning, and graph classification.The datasets include Cora, PubMed, WikiCS, WN18RR, BBBP, and BACE.Source: §IV · p. 4
  • The experiments use BERT, Sentence-BERT, E5, Llama2 7B, and Llama2 13B embeddings with GIN, GAT, or GCN experts.The reported embedding dimensions range from 768 for BERT and Sentence-BERT to 5,120 for Llama2 13B.Source: §IV · p. 4
  • Main experiments select expert combinations by validation performance, freeze selected experts, train a gate, and provide the resulting embeddings to LinearGNN, TrainlessGNN, or PRODIGY classifiers.Only prompt-based PRODIGY is trained on the training data in this setup.Source: §IV · p. 4
  • Additional experiments train full Chimaera with unfrozen linear-GNN parameters or train PRODIGY across node-, link-, and graph-level tasks.Multi-task PRODIGY combines losses for the tasks during each training step.Source: §IV · p. 5
  • Testing varies between 3, 10, and 20 shots per class, uses five runs with different shots and random seeds, and evaluates same-task or cross-task co-training and pre-training settings.WikiCS is limited to 10 shots; accuracy is used for multiclass tasks and AUC for binary tasks.Source: §IV · p. 5
  • The gating analysis tracks DA-MoE expert scores with k = 3 across tested samples, classes, training data, and three expert-selection strategies.The analysis focuses on DA-MoE because its gate uses graph structure.Source: §IV · p. 5
  • Grid search varies embedding size, layer count, dropout, and weight decay, and tunes TrainlessGNN edge weight and gating-function settings.Experts and gates use early stopping with a patience of five epochs.Source: §IV · p. 5

V. RESULTS

Chimaera performs comparably to trained baselines in several settings, while linear GNNs show cross-task transfer; results also reveal configuration-dependent gating and task-specific expert–encoder choices.

V. RESULTS

  • PRODIGY’s shot count has minimal effect, whereas linear GNNs tend to improve with more shots; stepwise training with frozen components outperforms fully trained Chimaera.On unseen datasets, linear GNNs also outperform PRODIGY, although multi-task PRODIGY is mostly better than its single-task variant.Source: §V · p. 5
  • Chimaera matches trained experts and gating functions in several settings, while linear GNNs transfer across tasks, especially to link prediction and graph classification.The reported comparable settings include node classification on Cora and PubMed, link prediction on PubMed, and graph classification on BBBP and BACE.Source: Table II · p. 6, §V · p. 6
  • No gating strategy wins all settings, but gating is the best GFM input in more than 93% of cases and improves accuracy or AUC except on graph tasks; k=3 beats k=2.The reported comparison covers MoE combinations against the best GNNs.Source: §V · p. 6, Table III · p. 7
  • Best expert–encoder pairs vary by task: Llama2 13B or E5 for nodes, Llama2 13B with GCN for reasoning, E5 with GAT for links, and GCN combinations for graphs.For graph classification, the reported GCN pairings use a sentence encoder or Llama2 7B.Source: §V · pp. 6–7, Table IV · p. 7
  • DA-MoE gating scores vary by dataset and strategy; BACE-trained gating uses experts comparably across test sets, while most classes show distinct score distributions.Cora node- and link-trained gating functions exhibit similar behavior, and score distributions can distinguish classes.Source: §V · p. 7

VI. DISCUSSION

Chimaera transfers across tasks and datasets, with linear GNN classifiers performing particularly well in cross-task and pre-training settings; its results also depend on expert combination, gating, and encoder choices.

VI. DISCUSSION

  • Linear GNN classifiers tend to outperform PRODIGY in pre-training and cross-task settings, while Chimaera’s step-by-step training with frozen components benefits transfer.Training Chimaera as a whole reduces the transferability of experts and gating functions.Source: §VI · p. 7
  • Combining experts with gating usually beats using one ungated expert; more experts and the multi-layer strategy generally improve performance, except that expert combinations do not help graph-level tasks.DA-MoE is recommended because it uses the graph for gating and can be tailored to specific needs.Source: §VI · p. 7
  • Larger embeddings improve results on tasks with more classes, making the encoder choice an important optimization consideration for GNN performance.The authors report this trend across all examined tasks.Source: §VI · pp. 7–8
  • Fig. 3 reports average scores for each expert on test sets under DA-MoE with k = 3 and a multi-layer gating strategy.The test dataset is identified in each sub-caption.Source: Figure 3 · p. 8
  • Fig. 4 shows, across test datasets, how many class pairs have similar distributions of scores from the multi-layer DA-MoE gating function with k = 3.The test dataset is identified in each sub-caption.Source: Figure 4 · p. 8
  • The study trains models for one task type, caps the number of experts at three, and leaves the optimal expert count open for future work.Without link-task training, PRODIGY's edge projection is untrained and it performs worse on link tasks.Source: §VI · p. 8

VII. CONCLUSION AND FUTURE WORK

Chimaera integrates gating functions and graph foundation models using multiple LLM embeddings. The authors identify further strategies, encoders, and training procedures to explore for improving transfer.

VII. CONCLUSION AND FUTURE WORK

  • Chimaera integrates gating functions and graph foundation models using multiple LLM embeddings, and the authors report that gating-based GNN combinations enhance performance.The conclusion also states that Chimaera enables linear GNNs across classification tasks and supports transfer across tasks and datasets with only a minimal decrease relative to task-specific models.Source: §VII · p. 8
  • Future work may explore additional expert-combination strategies, gating optimization, and a broader range of encoders, including non-LLM encoders such as Node2Vec.Source: §VII · p. 8
  • The authors also suggest exploring different training procedures, including GNN pre-training and transductive GNNs, to potentially improve transfer performance.Source: §VII · p. 9

VIII. ETHICAL CONSIDERATIONS

The authors describe the work as foundational research without a specific application focus and state that no specific societal consequences require immediate emphasis.

VIII. ETHICAL CONSIDERATIONS

  • The authors characterize the work as foundational research rather than research tied to a specific application.Source: §VIII · p. 9
  • They state that, despite broader implications, they see no specific societal consequences requiring immediate emphasis.Source: §VIII · p. 9
  • The acknowledgments report support from Baden-Württemberg through bwHPC and co-funding from DFG's CodeInspector Project.Source: §VIII · p. 9

APPENDIX

The appendix compares Chimaera with other graph foundation models and evaluates full-data and few-shot transfer across datasets, reporting competitive comparisons and several cross-dataset wins.

APPENDIX

  • The comparison uses Sentence-BERT node embeddings and ten random-seed runs; HIV is evaluated by test AUC, while other datasets use accuracy.The setup includes citation and web-link networks for node classification, knowledge graphs for reasoning, and HIV for graph-level tasks; training parameters and dataset characteristics are in Table V.Source: APPENDIX · p. 11, Table V · p. 12
  • Chimaera and its components show competitive performance against other GFMs in same-task co-training comparisons, while the performance drop with an untrained linear GNN classifier is typically under five percentage points.Figure 5 illustrates performance across datasets; Table VI compares models and components in the same-task and co-training setting.Source: APPENDIX · p. 11, Figure 5 · p. 11, Table VI · p. 12
  • In few-shot tests, FB15K237-trained Chimaera outperforms other baselines and Chimaera variants on 3-way arXiv, beats OFA on Cora, and matches or exceeds OFA on WN18RR.On Cora, GFT adapted with additional fine-tuning instances outperforms the FB15K237-trained Chimaera; on HIV, results are similar whether or not Chimaera was trained on that dataset.Source: APPENDIX · p. 11, Table VII · p. 12, Table VIII · p. 12
  • Across comparisons, Chimaera is generally similar to or better than OFA, transfers from FB15K237 to other datasets, and can outperform GFT on arXiv with fewer classes.The authors report high standard deviations and suggest that selecting high-quality examples may improve adaptation.Source: APPENDIX · p. 11
  • The few-shot tables cover arXiv, Cora, FB15K237, WN18RR, and HIV; their captions identify fine-tuning instances per class for GFT and mark the best result and best Chimaera.Source: Table VII · p. 12, Table VIII · p. 12, Table IX · p. 13

Related papers

Loading 2609.08709v1…