PaperlayerView exact revision on arXiv

Source-linked AI summary

HINormer: Representation Learning On Heterogeneous Information Networks with Graph Transformer

Qiheng Mao, Zemin Liu, Chenghao Liu, Jianling Sun

arXiv:2302.11329v2cs.LG

TL;DR

Message-passing GNN limitations and the under-exploration of Graph Transformers on HINs motivate the study. HINormer combines local structure and heterogeneous relation encoders with Transformer attention, and experiments on four HIN benchmarks report performance above state-of-the-art baselines.

  • Problem

    Graph Transformers remain under-explored for large, heterogeneity-complex HINs, where quadratic global attention also challenges scalability.

    Source: §1 · p. 2

  • Method

    HINormer uses Transformer attention with a local structure encoder and a heterogeneous relation encoder to learn HIN node representations end to end without pre-designed meta-paths.

    Source: §1 · p. 2

  • Results

    HINormer outperforms state-of-the-art baselines on four HIN benchmark datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments support the effectiveness of HINormer for capturing structural and heterogeneous information in HIN node representations.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent studies have highlighted the limitations of message-passing based graph neural networks (GNNs), e.g., limited model expressiveness, over-smoothing, over-squashing, etc. To alleviate these issues, Graph Transformers (GTs) have been proposed which work in the paradigm that allows message passing to a larger coverage even across the whole graph. Hinging on the global range attention mechanism, GTs have shown a superpower for representation learning on homogeneous graphs. However, the investigation of GTs on heterogeneous information networks (HINs) is still under-exploited. In particular, on account of the existence of heterogeneity, HINs show distinct data characteristics and thus require different treatment. To bridge this gap, in this paper we investigate the representation learning on HINs with Graph Transformer, and propose a novel model named HINormer, which capitalizes on a larger-range aggregation mechanism for node representation learning. In particular, assisted by two major modules, i.e., a local structure encoder and a heterogeneous relation encoder, HINormer can capture both the structural and heterogeneous information of nodes on HINs for comprehensive node representations. We conduct extensive experiments on four HIN benchmark datasets, which demonstrate that our proposed model can outperform the state-of-the-art.

1 INTRODUCTION

HINormer applies Graph Transformers to HIN node representation learning, combining local structure and heterogeneous relation encoders. The introduction motivates this design by the limitations of message passing and the under-exploration and scalability challenges of Graph Transformers on HINs.

1 INTRODUCTION

  • Message-passing GNNs face expressiveness limits, over-smoothing, and over-squashing that restricts propagation of signals from distant nodes.The introduction notes that separate GNN-based remedies have not removed the broader message-passing limitation.Source: §1 · p. 1
  • Global attention has quadratic complexity in graph size, while large, heterogeneity-complex HINs remain under-explored for Transformer-based node learning.The paper identifies large perceptual fields as a risk to preserving local-view structure and asks how to capture both local structure and heterogeneous semantics.Source: §1 · p. 2
  • HINormer combines a local structure encoder and a heterogeneous relation encoder with Graph Transformer attention for HIN node representation learning.The encoders capture structure and heterogeneity together, without pre-designed semantic schemas such as meta-paths.Source: §1 · p. 2
  • A lightweight local structure encoder recursively aggregates contextual information for a target node, using its enhanced features as Transformer input.The paper describes D-hop context sampling and GATv2 attention as design choices for node-level learning and reduced parameter count.Source: §1 · p. 2
  • The heterogeneous relation encoder learns type-aware relational encodings of semantic proximity and uses them as a bias in global self-attention.It focuses on relative proximity without node features, complementing the feature-assisted local structure encoder.Source: §1 · p. 2

2 RELATED WORK

Related work spans shallow heterogeneous network embeddings, heterogeneous GNNs, and Graph Transformers. The section distinguishes their approaches to heterogeneity and structural context, while noting the scale challenge of global attention.

2 RELATED WORK

  • Heterogeneous network embedding methods include random-walk and proximity-based approaches, but are described as shallow and unable to preserve node attributes.Meta-paths or type-aware closeness constraints are used to exploit heterogeneity.Source: §2 · p. 2
  • HGNNs model heterogeneity through meta-path-based or meta-path-free approaches, with HGT retaining one-hop message passing and its short-range view.Meta-path methods rely on expert selection, while meta-path-free methods use type-aware modules on the original network.Source: §2 · pp. 2–3
  • Graph Transformers integrate structural information through GNN combinations, positional or structural encodings, or other graph features, but whole-graph attention scales quadratically with node count.The section says this makes existing approaches difficult to apply to large-scale real-world networks such as HINs.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define heterogeneous networks, describe neighborhood aggregation in GNNs, and outline the standard Transformer components used as background for the proposed model.

3.1 Heterogeneous Information Network

  • A HIN is represented by nodes, edges, features, node types, and edge types; a graph with one node type and one edge type is homogeneous.Node and edge type mappings assign each node and edge its respective type.Source: §3.1 · p. 3

3.2 Graph Neural Networks

  • GNNs and HGNNs form target-node representations by recursively aggregating and transforming messages from neighboring nodes.The neighborhood may be type-specific for HGNNs, and the local structure encoder is described as flexible across neighborhood-aggregation GNNs.Source: §3.2 · p. 3

3.3 Transformer Architecture

  • A standard Transformer layer uses multi-head self-attention and a feed-forward network, with stacked layers producing sequence representations.The preliminaries describe query, key, and value projections followed by scaled dot-product attention, layer normalization, and residual connections.Source: §3.3 · p. 3

4 THE PROPOSED MODEL: HINORMER

HINormer combines local structural features and heterogeneity-based semantic proximities as inputs to a node-level Graph Transformer, which uses sampled contexts to produce node representations for supervised classification.

4.1 Node-level Heterogeneous Graph Transformer Architecture

  • HINormer samples fixed-length, D-hop node contexts for global attention, replacing dot-product attention with GATv2 and removing feed-forward networks to reduce parameters.The sampled context emphasizes local structure and limits noise from overly long-range nodes.Source: §4.1 · pp. 4–5

4.2 Local Structure Encoder

  • The local structure encoder projects different node types into a shared feature space and aggregates neighborhood information to provide locally enhanced features to the Transformer.It can use GNN or HGNN architectures, including non-parametric implementations.Source: §4.2 · p. 5

4.3 Heterogeneous Relation Encoder

  • A type-initialized relation encoder aggregates local heterogeneity with type-aware weights, then supplies contextual heterogeneity as positional encoding to adjust attention toward semantic proximities.The encoder uses neighborhood aggregation, with GCN employed in practice.Source: §4.3 · pp. 5–6

4.4 Training objective

  • HINormer trains node representations for node classification with cross-entropy for multi-class tasks and binary cross-entropy for multi-label tasks.The authors leave other downstream tasks for future work.Source: §4.4 · p. 6

5 EXPERIMENTS

The experiments evaluate node classification on four HIN benchmarks and analyze HINormer’s performance, component contributions, attention mechanisms, efficiency, and parameter sensitivity.

5.1 Experimental Setups

  • Experiments cover citation, movie-rating, and knowledge-graph HINs, using four benchmarks and averaged Micro-F1 and Macro-F1 over five runs.DBLP and IMDB follow standard HGB splits; nodes are randomly split 24:6:70 for training, validation, and test.Source: §5.1 · p. 6

5.2 Performance Evaluation

  • HINormer outperforms all baselines; standard Transformer is competitive with HGNNs in most cases, while HINormer exceeds Transformer and GAT by a large margin.Table 2 reports node-classification results with GCN as the local structure encoder.Source: §5.2 · pp. 6–7, Table 2 · p. 7
  • With GIN, GraphSAGE, or parameter-free Adj encoders, HINormer outperforms Transformer, showing flexibility across local-structure backbones.The authors report promising performance even for HINormer-Adj, which has no learnable parameters.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Model Analysis

  • Across four datasets, GAT, Transformer, and GATv2 variants achieve approaching performance; GATv2 is more stable and slightly outperforms the others with fewer parameters.The default HINormer attention mechanism is GATv2.Source: Table 4 · p. 8, §5.3 · p. 8
  • Removing the local structure encoder sharply degrades performance; removing the heterogeneous relation encoder generally causes a smaller decrease, while the full model performs best.Without either encoder, HINormer still outperforms vanilla Transformer, which uses self-attention rather than the simplified GATv2 mechanism.Source: §5.3 · pp. 7–8, Figure 2 · p. 8
  • On DBLP, HINormer is as efficient as SimpleHGN and much faster than HGT in training time per epoch and inference time.The authors note that its relatively low time cost and fewer parameters enhance its availability across settings.Source: §5.3 · p. 8, Figure 3 · p. 8
  • Smaller Transformer depth and context size can favor performance or efficiency; larger hidden dimensions usually help, while heterogeneous similarity weights of 0.1–0.5 benefit performance.The context size’s changes do not significantly affect performance; moderate values of β are beneficial.Source: §5.3 · p. 8, Figure 4 · p. 8

6 CONCLUSION

The paper investigates Graph Transformers for node representation learning on heterogeneous information networks and proposes HINormer, whose two encoders capture structural and heterogeneous information. Experiments on four benchmark datasets demonstrate its effectiveness.

  • The paper investigates Graph Transformers for node representation learning on heterogeneous information networks.Source: §6 · p. 8
  • Experiments on four benchmark datasets demonstrate the effectiveness of HINormer.Source: §6 · p. 9
  • HINormer uses a local structure encoder and a heterogeneous relation encoder to capture structural and heterogeneous information for node representations.Source: §6 · p. 9

A Further Details of Datasets

The evaluation uses DBLP, AMiner, IMDB, and Freebase, spanning academic citation, movie, and knowledge-graph data. The appendix describes their scope and node types.

B Further Details of Baselines

The baseline suite covers homogeneous GNNs, meta-path-based HGNNs, and meta-path-free HGNNs. It includes conventional neighborhood models, semantic-path models, and heterogeneous attention approaches.

C Hyperparameters Settings

Baseline settings specify meta-paths for Freebase and AMiner, alongside model-specific hyperparameters. HINormer’s optimizer, dimensions, sequence lengths, and encoder depths are also reported.

  • Meta-path baselines use M-A-M, M-D-M, and M-W-M on Freebase, and P-A-P and P-R-P on AMiner.Source: C Hyperparameters Settings · p. 11
  • For both datasets, GCN uses d=64; GAT uses d=64, nh=8, L=3, s=0.05; and RGCN uses d=32, L=5.Source: C Hyperparameters Settings · p. 11
  • Other baseline settings include HetGNN d=128 with batch size 200, HAN d=8 with L=3, and HGT d=64 with nh=8 and L=3.Source: C Hyperparameters Settings · p. 11
  • HINormer uses Adam, learning rate 1e-4, hidden dimension 256, two heads, and sequence lengths from 10 to 200; encoder depths vary by dataset.Dropout is 0 for Freebase and 0.5 for other datasets; local and relation encoder depths are 5 for DBLP, 4 for IMDB, and 3 for others.Source: C Hyperparameters Settings · p. 12
Loading 2302.11329v2…