PaperlayerView exact revision on arXiv

Source-linked AI summary

Heterogeneous Graph Attention Network

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Peng Cui, P. Yu, Yanfang Ye

arXiv:1903.07293v2cs.SI

TL;DR

Heterogeneous graphs combine diverse node and edge types, making it difficult for graph neural networks to preserve their complex structure and semantics. HAN uses node-level and semantic-level attention to weight neighbors and meta-paths, outperforming state-of-the-art models while offering potentially interpretable graph analysis.

  • Problem

    Heterogeneous graphs contain diverse node types, edges, and feature spaces, creating a challenge for preserving complex structure and rich semantics in graph neural networks.

  • Method

    HAN projects node types into a shared space and hierarchically applies node-level and semantic-level attention to aggregate meta-path neighbors and weight meta-paths.

  • Results

    Experiments on three real-world heterogeneous graphs show HAN outperforms state-of-the-art models and demonstrates potentially good interpretability for graph analysis.

  • Takeaways & Limitations

    HAN captures complex heterogeneous-graph structures and rich semantics while its learned attention weights support potentially interpretable graph analysis.

Abstract

from arXiv · show

Graph neural network, as a powerful graph representation technique based on deep learning, has shown superior performance and attracted considerable research interest. However, it has not been fully considered in graph neural network for heterogeneous graph which contains different types of nodes and links. The heterogeneity and rich semantic information bring great challenges for designing a graph neural network for heterogeneous graph. Recently, one of the most exciting advancements in deep learning is the attention mechanism, whose great potential has been well demonstrated in various areas. In this paper, we first propose a novel heterogeneous graph neural network based on the hierarchical attention, including node-level and semantic-level attentions. Specifically, the node-level attention aims to learn the importance between a node and its metapath based neighbors, while the semantic-level attention is able to learn the importance of different meta-paths. With the learned importance from both node-level and semantic-level attention, the importance of node and meta-path can be fully considered. Then the proposed model can generate node embedding by aggregating features from meta-path based neighbors in a hierarchical manner. Extensive experimental results on three real-world heterogeneous graphs not only show the superior performance of our proposed model over the state-of-the-arts, but also demonstrate its potentially good interpretability for graph analysis.

1 INTRODUCTION

The paper introduces HAN, a heterogeneous graph neural network that uses hierarchical node-level and semantic-level attention to model diverse node, neighbor, and meta-path importance. It addresses heterogeneous graph challenges while offering efficient large-scale application and potentially interpretable analysis.

  • Motivation: Heterogeneous graphs combine multiple node and edge types with diverse feature spaces, making it difficult to preserve structural and feature information simultaneously.The paper identifies graph heterogeneity as a central challenge in designing graph neural networks for heterogeneous graphs.
  • Design requirements: Node-level attention learns the importance of meta-path-based neighbors, while semantic-level attention weights meta-paths according to task relevance.These mechanisms address neighbor selection and semantic-information fusion in heterogeneous graphs.
  • Method: HAN projects different node types into a shared space and hierarchically aggregates information using node-level and semantic-level attention.The model learns attention values for both nodes and meta-paths from node features and heterogeneous graph structure.
  • Contributions: HAN considers node and meta-path importance simultaneously and has linear complexity with respect to the number of meta-path-based node pairs.This efficiency is intended to support large-scale heterogeneous graphs.
  • Evaluation: Experiments report superior performance over state-of-the-art models and potentially good interpretability for heterogeneous graph analysis.The paper evaluates HAN through extensive experiments and analyzes its hierarchical attention mechanism.

2 RELATED WORK

Related work spans graph neural networks, attention-based graph learning, and network embedding. Existing methods address homogeneous graphs or require user-defined meta-path weighting, leaving heterogeneous graph attention and automatic semantic importance insufficiently addressed.

  • Graph Neural Networks: Graph neural networks extend deep neural networks to arbitrary graph-structured data, with graph convolutions broadly divided into spectral and non-spectral approaches.Propagation models can use gated recurrent units to spread information across nodes.
  • Attention Mechanisms: Attention mechanisms have been applied to graph tasks, and Graph Attention Network learns node–neighbor importance for node classification.Prior graph-based applications include recommendation, while the passage notes limitations for heterogeneous graphs.
  • Network Embedding: Network embedding learns low-dimensional representations that preserve network structure and properties for downstream tasks, but surveyed methods target homogeneous graphs.Examples include random-walk, deep-neural-network, matrix-factorization, and LINE-based methods.
  • Heterogeneous Graph Embedding: Heterogeneous graph embedding emphasizes meta-path-based structural information, but ESim cannot learn meta-path importance and requires grid search for optimal weights.ESim uses user-defined meta-paths and can incorporate multiple meta-paths.

3 PRELIMINARY

This section defines heterogeneous graphs, meta-paths, and meta-path-based neighbors as structures that encode diverse semantic and structural information. It motivates a heterogeneous graph neural network because existing graph neural networks are designed for homogeneous networks and do not exploit differences among nodes and meta-paths.

  • Heterogeneous Graph: A heterogeneous graph contains multiple object or link types, represented as G = (V, E) with node-type and link-type mappings.The mappings are 𝜙: V → A and 𝜓: E → R, where |A| + |R| > 2.
  • Meta-path: A meta-path is a composite relation connecting objects through a sequence of node and link types, with different paths revealing different semantics.For example, MAM denotes a co-actor relation, whereas MDM denotes movies directed by the same director.
  • Meta-path-based Neighbors: Meta-path-based neighbors are nodes connected to a given node through a specified meta-path, including the node itself.These neighbors expose diverse structural information and rich semantics; they can be obtained by multiplying sequences of adjacency matrices.
  • Motivation: Because existing graph neural networks target homogeneous networks, the paper proposes a heterogeneous model that exploits subtle differences among nodes and meta-paths.Meta-paths and meta-path-based neighbors are identified as fundamental heterogeneous-graph structures.

4 THE PROPOSED MODEL

HAN is a semi-supervised heterogeneous graph neural network that uses hierarchical node-level and semantic-level attention to learn node and meta-path importance. It projects heterogeneous node features into a shared space, aggregates meta-path neighbors, fuses semantic-specific embeddings, and supports efficient, inductive, interpretable graph analysis.

  • Node-level attention: HAN projects each node type into a unified feature space, then uses node-level attention to weight and aggregate meaningful meta-path-based neighbors.A type-specific transformation matrix handles arbitrary node types, while masked attention incorporates graph structure.
  • Node-level attention: Node-level attention preserves asymmetric importance between connected nodes, allowing each direction of a meta-path-based relationship to contribute differently.The asymmetry arises from feature-dependent attention and differing concatenation and normalization contexts.
  • Node-level attention: HAN repeats node-level attention K times and concatenates the outputs as semantic-specific embeddings to improve training stability on scale-free heterogeneous graphs.Each meta-path produces an embedding capturing one kind of semantic information.
  • Semantic-level attention: Semantic-level attention automatically learns the importance of different meta-paths and fuses their semantic-specific embeddings into a more comprehensive node representation.The semantic attention network captures varied semantic information behind the heterogeneous graph.
  • Model properties: HAN has overall complexity linear in the number of nodes and meta-path-based node pairs, with attention computations parallelizable across nodes and meta-paths.Its shared hierarchical attention is independent of graph scale, enabling embeddings for previously unseen nodes or unseen graphs; learned attention values also support interpretability.

5 EXPERIMENTS · 5.1 Datasets · 5.2 Baselines

The experiments evaluate HAN on three heterogeneous graphs using dataset-specific node types, labels, features, and meta-paths. Comparisons include network-embedding and graph-neural-network baselines, plus variants isolating node-level and semantic-level attention.

  • 5.1 Datasets: DBLP2 contains 14328 papers, 4057 authors, 20 conferences, and 8789 terms, with four author research areas and meta-paths {APA, APCPA, APTPA}.Author features are bag-of-words keyword representations, and research areas are labeled from submitted conferences.
  • 5.1 Datasets: ACM3 contains 3025 papers, 5835 authors, and 56 subjects, with three paper classes and meta-paths {PAP, PSP}.Paper features are bag-of-words keyword representations, and labels derive from publication conferences.
  • 5.1 Datasets: IMDB contains 4780 movies, 5841 actors, and 2269 directors, with three movie genres and meta-paths {MAM, MDM}.Movie features are bag-of-words plot representations.
  • 5.2 Baselines: The study compares HAN with heterogeneous and homogeneous network-embedding and graph-neural-network methods, and tests variants for node-level and semantic-level attention.The comparisons are designed to verify HAN’s effectiveness and the contributions of its two attention levels.
  • 5.2 Baselines: DeepWalk ignores node heterogeneity on the whole graph, while ESim captures multiple-meta-path semantics using weights learned from HAN.DeepWalk is random-walk based; ESim’s meta-path weights are otherwise difficult to search.
  • 5.2 Baselines: metapath2vec and HERec use meta-path-based or type-constrained random walks with skip-gram, testing all meta-paths and reporting the best performance.metapath2vec performs meta-path-based random walks, whereas HERec filters node sequences with type constraints.
  • 5.2 Baselines: GCN and GAT are homogeneous-graph semi-supervised baselines tested across all meta-paths, while HAN𝑛𝑑 and HAN𝑠𝑒𝑚 remove node-level and semantic-level attention, respectively.HAN simultaneously employs both attention mechanisms.

5.3 Implementation Details

The proposed HAN is randomly initialized and optimized with Adam using specified learning, regularization, attention, dropout, and early-stopping settings.

  • Optimization: HAN training uses Adam optimization with a learning rate of 0.005 and regularization parameter of 0.001.Parameters are randomly initialized before optimization.
  • Attention: The model sets the semantic-level attention vector q dimension to 128 and uses 8 attention heads with attention dropout of 0.6.
  • Training control: Early stopping uses a patience of 100, stopping training when validation loss fails to decrease for 100 consecutive epochs.

5.4 Classification · 5.5 Clustering

HAN achieves the strongest node-classification and clustering performance across the evaluated heterogeneous graphs. The results indicate that modeling the importance of nodes and meta-paths is central to effective heterogeneous-graph analysis.

  • 5.4 Classification: KNN classification uses k=5, with Macro-F1 and Micro-F1 averaged over 10 repetitions to reduce variance effects.The classification results are reported in Table 3.
  • 5.4 Classification: HAN achieves the best performance in node classification, while ESim outperforms metapath2vec among traditional heterogeneous graph embedding methods.Graph neural network methods such as GCN and GAT generally perform better by combining structure and feature information.
  • 5.4 Classification: GAT and HAN improve over neighbor averaging methods such as GCN and HAN_nd by properly weighting node information.The paper links this phenomenon to semantic-level attention analysis in Section 5.7.
  • 5.4 Classification: HAN achieves the best performance on all datasets, demonstrating the importance of capturing node and meta-path importance.This conclusion is drawn from the classification results.
  • 5.5 Clustering: KMeans clustering uses the number of classes as K, the node-classification ground truth, and NMI and ARI to evaluate embedding quality.The clustering evaluation uses embeddings obtained by feeding forward through the trained algorithms; repeated initialization affects KMeans performance.
  • 5.5 Clustering: HAN performs consistently much better than all baselines in clustering, while graph neural network methods generally achieve better performance.The passage specifically reports that metapath2vec and GCN perform poorly without distinguishing node or meta-path importance.
  • 5.5 Clustering: Multiple meta-path guidance makes HAN significantly better than GCN and GAT in clustering.Removing node-level attention in HAN_nd or semantic-level attention in HAN_sem also reduces performance.
  • 5.5 Clustering: The clustering analysis concludes that HAN provides a comprehensive description of heterogeneous graphs and achieves significant improvements.This conclusion summarizes the reported clustering comparisons.

5.6 Analysis of Hierarchical Attention Mechanism

HAN’s hierarchical attention mechanism jointly models the importance of neighboring nodes and meta-paths when learning node embeddings. Analyses on ACM and DBLP show that node-level attention highlights task-relevant nodes, while semantic-level attention differentiates meta-path importance.

  • Hierarchical attention mechanism: HAN incorporates node-level and semantic-level attention to consider the importance of neighbors and meta-paths in representative node embeddings.The node-level weight is αΦᵢⱼ, and the semantic-level weight is βΦᵢ.
  • Analysis of node-level attention: In the ACM example, P831 receives the highest node-level attention, followed by P699 and P133, indicating that the node itself is most important.Neighbors provide supplementary information, while P699 and P133 receive the second- and third-largest attention values.
  • Analysis of semantic-level attention: For DBLP, HAN assigns APCPA the largest semantic-level attention weight because it best supports identifying an author’s research area.The passage relates this to correlations between authors’ research areas and the conferences where they submit papers.
  • Analysis of semantic-level attention: Treating DBLP meta-paths equally in HANsem causes performance to drop significantly, whereas semantic-level attention weights their differing usefulness.The passage specifically notes that APA is less effective for identifying authors’ research areas.
  • Analysis of semantic-level attention: For ACM, HAN assigns the greatest weight to PAP, whose slightly better clustering performance than PSP still permits HANsem to perform well by simple averaging.The analysis concludes that semantic-level attention reveals differences among meta-paths and weights them appropriately.

5.7 Visualization

The authors visualize DBLP author embeddings in two dimensions with t-SNE, coloring nodes by research area. Homogeneous-graph GCN and GAT mix research areas, while metapath2vec improves separation but retains blurry boundaries because it uses only one meta-path.

  • Visualization: The authors project proposed-model embeddings of DBLP authors into two dimensions with t-SNE and color nodes by research area.The visualization lays out the heterogeneous graph in a low-dimensional space for intuitive comparison.
  • Visualization: GCN and GAT mix authors from different research areas, indicating poor visualization performance on the heterogeneous graph.Both models were designed for homogeneous graphs.
  • Visualization: Metapath2vec separates research areas better than GCN and GAT, showing that an appropriate meta-path such as APCPA contributes to heterogeneous graph analysis.Its improvement over the homogeneous-graph neural networks is visible in the author embedding visualization.
  • Visualization: Metapath2vec still produces blurry boundaries because it considers only one meta-path.The limitation remains despite its improvement over GCN and GAT.

5.8 Parameters Experiments

Parameter experiments on ACM show that HAN is sensitive to embedding and semantic-attention dimensions, while increasing attention heads generally yields only slight gains but more stable training. The semantic-level attention vector performs best at dimension 128.

  • Final embedding dimension Z: HAN’s performance first increases and then slowly declines as the final embedding dimension grows, because larger dimensions may introduce redundant information.The model requires a suitable dimension to encode semantic information.
  • Semantic-level attention vector q: HAN achieves its best performance when the semantic-level attention vector q has dimension 128, after which performance begins to degenerate.The experiment evaluates how q’s dimension affects semantic-level attention ability.
  • Number of attention heads K: Increasing the number of attention heads generally improves HAN’s performance, but the gains are only slight.With one head, multihead attention is removed.
  • Number of attention heads K: Multihead attention makes HAN’s training process more stable as the number of attention heads changes.The parameter sensitivity results are reported using clustering NMI on the ACM dataset.

6 CONCLUSION

The paper proposes a semi-supervised heterogeneous graph neural network, HAN, based solely on attention mechanisms to model complex structures and rich semantics in heterogeneous graphs.

  • 6 CONCLUSION: HAN uses node-level attention to learn the importance of nodes in heterogeneous graph analysis.The model is semi-supervised and based solely on attention mechanisms.
  • 6 CONCLUSION: HAN uses semantic-level attention to learn the importance of meta-paths.Together, the two attention levels address heterogeneous graphs’ complex structures and rich semantics.
Loading 1903.07293v2…