PaperlayerView exact revision on arXiv

Source-linked AI summary

Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks

Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng, Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, Jie Tang

arXiv:2112.14936v1cs.LGcs.SI

TL;DR

HGNN progress is difficult to assess because individual works use different data processing and evaluation setups. The paper reproduces 12 models, builds a standardized 11-dataset benchmark, and finds that properly evaluated homogeneous GNNs are strong while Simple-HGN outperforms prior HGNNs on HGB.

  • Problem

    Different HGNN works use unique data processing and experimental settings, limiting comprehensive understanding of their comparative progress.

  • Method

    The paper reproduces 12 HGNNs using their released resources, constructs HGB with standardized splits, feature processing, and evaluation, and introduces Simple-HGN.

  • Results

    Properly configured homogeneous GNNs, especially GAT, can match or outperform existing HGNNs, while Simple-HGN consistently outperforms previous HGNNs across three tasks and 11 datasets.

  • Takeaways & Limitations

    Fair, standardized comparisons are needed to understand HGNN progress and support reproducible development.

  • Takeaways & Limitations

    The reproduced evidence identifies concrete problems in prior work, including tuning on test sets and data leakage in reported experiments.

Abstract

from arXiv · show

Heterogeneous graph neural networks (HGNNs) have been blossoming in recent years, but the unique data processing and evaluation setups used by each work obstruct a full understanding of their advancements. In this work, we present a systematical reproduction of 12 recent HGNNs by using their official codes, datasets, settings, and hyperparameters, revealing surprising findings about the progress of HGNNs. We find that the simple homogeneous GNNs, e.g., GCN and GAT, are largely underestimated due to improper settings. GAT with proper inputs can generally match or outperform all existing HGNNs across various scenarios. To facilitate robust and reproducible HGNN research, we construct the Heterogeneous Graph Benchmark (HGB), consisting of 11 diverse datasets with three tasks. HGB standardizes the process of heterogeneous graph data splits, feature processing, and performance evaluation. Finally, we introduce a simple but very strong baseline Simple-HGN--which significantly outperforms all previous models on HGB--to accelerate the advancement of HGNNs in the future.

1 INTRODUCTION

The paper revisits heterogeneous GNN progress through reproduction, finding that inconsistent setups obscure fair comparisons and that properly configured homogeneous GNNs are stronger than previously reported. It introduces HGB for standardized evaluation and Simple-HGN as a strong baseline.

  • 12 HGNNs are reproduced using their original codes, datasets, settings, and hyperparameters to reassess reported progress.
  • Properly configured GCN and GAT baselines reveal that homogeneous GNN performance was largely underestimated, with vanilla GAT outperforming existing HGNNs in most cases.
  • The investigation finds that some reported results are affected by inappropriate settings or data leakage, while meta-paths are unnecessary in most heterogeneous datasets.
  • HGB contains 11 heterogeneous datasets spanning varied heterogeneity, three tasks, and multiple domains to support robust, reproducible benchmarking.
  • Simple-HGN extends GAT with learnable type embeddings, residual connections, and L2-normalized output embeddings, and consistently outperforms previous HGNNs across three tasks and 11 datasets.
  • The work’s contributions are to identify issues blocking progress, benchmark HGNNs with HGB, and refine them with Simple-HGN.

2 PRELIMINARIES

The preliminaries define heterogeneous graphs, explain standard GCN and GAT message passing, and introduce meta-paths as typed connectivity patterns used to represent heterogeneous semantics.

  • A heterogeneous graph contains multiple node and edge types; when each has only one type, it becomes an ordinary homogeneous graph.
  • After L graph transformations, each node obtains a representation in R^d_L that can support node classification, graph classification after pooling, and link prediction.
  • A GCN layer transforms node representations by applying an activation function to normalized adjacency aggregation followed by a trainable weight matrix.
  • GAT replaces average neighbor aggregation with attention-weighted aggregation, using learnable parameters and optionally multiple attention heads.
  • A meta-path is a typed path pattern whose composite relations are treated as carrying distinct semantics, such as co-authorship or potential user-item interest.
  • Given a meta-path, a meta-path neighbor graph connects two nodes when the original graph contains at least one path following that pattern.

3 ISSUES WITH EXISTING HETEROGENEOUS GNNS

Reproduction exposes unfair baseline comparisons, computational inefficiency, experimental flaws, and limited dataset openness in existing HGNN studies. HGB and Simple-HGN address these issues through standardized evaluation and a strong GAT-based model.

  • 3.1.1 HAN: HAN compares against GAT using incomplete meta-path graphs, while GAT on the original graph consistently outperforms HAN.The corrected baseline ignores node types but retains target-node features.
  • 3.1.2 GTN: GTN’s average scores are not significantly different from GAT despite consuming >400× its training time and 120× its memory.On DBLP, GTN requires 120 GB memory and 12 hours, compared with 1 GB and 10 seconds for GCN and GAT.
  • 3.1.3 RSHN: RSHN tunes hyperparameters and selects reported checkpoints on the test set, while properly tuned GAT reaches 100% accuracy on AIFB and BGS under that setting.The reproduced GAT results exceed RSHN’s reported 91.67% and 66.32% values on those datasets.
  • 3.1.5 MAGNN: MAGNN underperforms GAT with correct inputs and leaks link-prediction information through sequentially ordered positive and negative minibatches with batch normalization.Shuffling the Last.fm test set reduces MAGNN’s AUC from 98.91 to 71.49.
  • 3.4 Summary: Existing HGNN studies commonly lack fair comparisons and include issues such as data leakage, test-set tuning, and excessive resource use without effectiveness improvements.The paper argues that inconsistent individual data and experimental setups hinder progress.
  • Benchmark and refinement: HGB standardizes heterogeneous-graph evaluation across three tasks and diverse datasets, while Simple-HGN combines GAT with type embeddings, residual connections, and L2 normalization.Simple-HGN consistently outperforms previous HGNNs across three tasks and 11 datasets.

4 HETEROGENEOUS GRAPH BENCHMARK

HGB addresses inconsistent datasets and pipelines by standardizing data splits, feature processing, and evaluation across 11 heterogeneous datasets and three task types. Its benchmark fixes a reproducible pipeline while selecting preprocessing and decoders using validation performance.

  • Benchmark motivation: Different dataset settings, splits, and retained edges make results from existing HGNN papers difficult to compare fairly.Examples include different ACM citation-link choices and dataset splits across studies.
  • Benchmark motivation: Pipeline components outside the HGNN encoder, including feature selection and downstream decoders, can substantially affect task performance.The benchmark therefore controls feature preprocessing and decoder choices alongside the encoder.
  • HGB design: HGB standardizes the pipeline as feature preprocessing → HGNN encoder → downstream decoder, selecting preprocessing and decoders by validation performance.This provides a consistent interface for data loading, feature processing, and evaluation.
  • HGB design: HGB contains 11 medium-scale heterogeneous datasets spanning node classification, link prediction, and knowledge-aware recommendation.The datasets vary in node and edge-type heterogeneity and include predefined meta-paths from previous works.
  • Evaluation setup: Node classification uses a transductive split with 24% training, 6% validation, and 70% test labels per dataset.All edges remain available during training.
  • Evaluation setup: Link prediction uses 81% training, 9% validation, and 10% test edges, with 2-hop negative test pairs sampled at a 1:1 ratio to positives.Uniform negatives were rejected because they made predictions nearly perfect for most models.
  • Evaluation setup: The benchmark evaluates node classification with Macro-F1 and Micro-F1, link prediction with ROC-AUC and MRR, and recommendation with recall@20 and ndcg@20.All methods are run five times with different random seeds, reporting mean scores and standard deviations.

5 A SIMPLE HETEROGENEOUS GNN

Simple-HGN extends GAT with edge-type information, residual connections, multi-head attention, and output normalization. The design adapts components for link prediction and uses a unified pipeline with feature preprocessing and downstream evaluation.

  • Model overview: Simple-HGN uses GAT as its backbone with learnable edge-type embeddings, residual connections, and L2 normalization.These techniques are presented as a simple redesign for heterogeneous graph modeling.
  • Heterogeneous attention: Edge-type embeddings are incorporated into attention scores so the mechanism uses both node representations and edge-type information.A learnable matrix transforms the embedding associated with each edge type.
  • Residual connection: Node residual connections aggregate representations across layers using a pre-activation design, with an additional linear transformation when dimensions change.The approach is related to layer-wise representation aggregation in JKNet.
  • Residual connection: Edge residual connections add residual information to raw attention scores before subsequent attention processing.This design is motivated by prior work showing that residual connections can help attention scores.
  • Multi-head attention: Simple-HGN uses K independent attention mechanisms and averages their outputs in the final layer when concatenation is unsuitable.Averaging avoids requiring the output dimension to divide exactly by the number of heads.
  • Task adaptation: For link prediction, Simple-HGN removes edge residuals and concatenates the final layer representations before decoding.This is a task-specific architectural adaptation.
  • L2 normalization: L2 normalization makes dot-product similarity equivalent to cosine similarity and improves both retrieval-based and classification tasks.The authors also report that tuning a scaling parameter can improve performance, although its useful value varies across datasets.

6 EXPERIMENTS

The experiments benchmark prior HGNNs, homogeneous GNNs, and Simple-HGN under a common HGB pipeline. Across three tasks, GAT often matches the best HGNNs, while Simple-HGN consistently outperforms advanced HGNNs on the reported datasets.

  • The benchmark evaluates HGNNs, GCN, GAT, and Simple-HGN on HGB, reporting averages and standard variances across five runs.
  • Under fair comparison, simple homogeneous GAT matches the best HGNNs in most cases across node classification, link prediction, and knowledge-aware recommendation.
  • Simple-HGN consistently outperforms advanced HGNNs for node classification on four datasets, link prediction on three datasets, and recommendation on three datasets.
  • Official implementations are used, with only data-loading interfaces and downstream decoders modified when needed for HGB compatibility.
  • Hyperparameters are tuned using validation-set performance, with optimization details recorded in the appendix.
  • Time and memory are measured for one training epoch on DBLP, while convergence epochs may differ across models.HetSANN is omitted because the authors could not obtain a reasonable Micro-F1 score.
  • 6.3 Ablation Studies: Residual connections and L2 normalization consistently improve performance, whereas type embeddings provide only slight gains on node classification.

7 DISCUSSION AND CONCLUSION

The discussion identifies unresolved questions about type information and meta-paths while emphasizing that heterogeneous graph research still has open problems. It also highlights computational comparisons and evaluation concerns relevant to interpreting benchmark results.

  • The paper identifies open questions remaining for heterogeneous GNNs and broader heterogeneous graph representation learning.
  • Type information: Type embeddings bring only minor improvements, possibly because heterogeneous node features already imply type distinctions.The paper also hypothesizes that current graph attention mechanisms may be too weak to fuse type and feature information.
  • Meta-paths: Results suggest that meta-path-based GNNs do not outperform homogeneous GAT, leaving open whether better ways exist to use meta-paths.
  • Efficiency: Figure 2 compares HGNN time and relative memory consumption on DBLP using circle area to represent memory.
  • Evaluation: Random negative samples produce substantially higher link-prediction scores than the benchmark's test negatives, because they are easier for many methods to distinguish.

C HYPER-PARAMETERS

The benchmark specifies shared hyperparameter-search ranges, dropout defaults, batch-size options, and validation-based early stopping. It also defines notation and feature-input modes used across experiments.

  • Learning rates are searched over {1, 5}×{10^-6, 10^-5, 10^-4, 10^-3, 10^-2}, while weight decay uses {0, 1, 2, 5}×{10^-6, 10^-5, 10^-4, 10^-3}.
  • Dropout is set to 0.1 for recommendation and 0.5 for node classification and link prediction by default.
  • Batch sizes are searched over {1024, 2048, 4096, 8192} unless an author's code imposes special requirements.
  • Training uses early stopping based on validation-set evaluation to support full training.
  • The notation defines d, d_e, d_a, L, n_h, and s as embedding, attention, layer, head, and LeakyReLU parameters.
  • Feature modes distinguish all given features, target-node-only features, and one-hot features for all nodes.

C.1 Simple-HGN

Simple-HGN uses fixed architectural and feature settings that vary by dataset group. The settings include embedding dimensions, attention heads, layer counts, activation slopes, and feature-input modes.

  • For all datasets, Simple-HGN sets d = d_e = 64, n_h = 8, and β = 0.05.
  • DBLP, ACM, and Freebase use L = 3 and s = 0.05, while IMDB uses L = 6 and s = 0.1.
  • Feature mode is feat = 0 for IMDB, feat = 1 for ACM, and feat = 2 for DBLP and Freebase.
  • A separate configuration sets d = 64, d_e = 32, n_h = 2, β = 0, and s = 0.01 for all datasets.The passage begins describing Amazon and PubMed settings but does not provide further dataset-specific values.

C.1.2 Link prediction.

The link-prediction experiments use dataset-specific decoder, depth, and feature settings, with several models requiring distinct sampling or training configurations. GTN additionally depends on graph structure, while MAGNN could not be adapted beyond LastFM.

  • Experimental settings: Link-prediction models use varied dataset-specific depths, dimensions, feature types, decoders, and learning-rate settings.Examples include GTN with d = 64 and two channels, and HAN with d = 8, d_a = 128, n_h = 8, and L = 2.
  • Model behavior: GTN collapses when trained on the whole graph, indicating sensitivity to graph structure.The benchmark instead aggregates keyword-node information into neighboring attributes and uses the left subgraph for node classification.
  • Experimental settings: Sampling-based experiments use embedding dimension 128, batch size 200, walk length 30, and window size 5.These settings are reported with different input feature types across configurations.
  • Model coverage: MAGNN was evaluated on LastFM with batch size 8 and 100 neighbor samples, but its code could not be adapted to the other datasets because of hard-coding.This limits the cross-dataset coverage of the MAGNN experiments.

C.8.1 Node Classification.

The node-classification benchmark combines standardized model configurations with selected meta-paths and sampling choices. Deeper graph networks can deteriorate performance, and the reported meta-paths draw on prior work plus dataset-specific selection and additions.

  • Depth and aggregation: Adding more graph layers deteriorates performance, so the experiments use d^(0) = 64 and d^(1) = 48.The passage notes that this behavior is also found in two cited prior studies.
  • Depth and aggregation: The node-classification experiments use sum aggregation because it had the best overall performance in the cited report.The same configuration passage reports negative samples of 5 and aggregation neighbor samples of 10, alongside walk length 30 and window size 5.
  • Depth and aggregation: Attention-based configurations preserve the Bi-Interaction aggregator according to the official code.One such setting uses d^(0) = 64, d^(1) = 32, and d^(2) = 16 for all datasets.
  • Meta-paths: The benchmark uses dataset-specific meta-paths drawn from prior works, with added ACM citation/reference paths and seven manually selected Freebase paths.Freebase candidates are first identified by frequency for lengths 2 to 4, then selected using validation performance.
Loading 2112.14936v1…