PaperlayerView exact revision on arXiv

Source-linked AI summary

Higher-Order Attribute-Enhancing Heterogeneous Graph Neural Networks

Jianxin Li, Hao Peng, Yuwei Cao, Yingtong Dou, Hekai Zhang, Philip S. Yu, Lifang He

arXiv:2104.07892v1cs.LGcs.SI

TL;DR

Heterogeneous graph methods often use limited semantic structures and do not fully capture content-based relationships among higher-order neighbors. HAEGNN combines meta-paths and meta-graphs with layered self-attention. The paper reports superior performance across node classification, node clustering, and visualization, along with memory efficiency and explainability.

  • Problem

    Existing heterogeneous graph methods may omit meta-graphs or fail to capture content-based relationships among higher-order neighbors.

    Source: §1 · pp. 1–2

  • Method

    HAEGNN combines meta-paths and meta-graphs with self-attention in a layered framework that examines first-order and higher-order neighborhoods.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    HAEGNN outperforms state-of-the-art baselines in node classification, node clustering, and visualization, with reported efficiency and explainability.

    Source: §1 · pp. 2–3, §6 · p. 14

  • Takeaways & Limitations

    The framework learns the importance of semantic structures while integrating heterogeneous semantics and content-based interactions in node representations.

    Source: §1 · pp. 2–3

    The paper identifies leveraging heterogeneous edge types, sequence information, dynamic inductive learning, and additional modalities such as images as future directions.

    Source: §6 · p. 14

Abstract

from arXiv · show

Graph neural networks (GNNs) have been widely used in deep learning on graphs. They can learn effective node representations that achieve superior performances in graph analysis tasks such as node classification and node clustering. However, most methods ignore the heterogeneity in real-world graphs. Methods designed for heterogeneous graphs, on the other hand, fail to learn complex semantic representations because they only use meta-paths instead of meta-graphs. Furthermore, they cannot fully capture the content-based correlations between nodes, as they either do not use the self-attention mechanism or only use it to consider the immediate neighbors of each node, ignoring the higher-order neighbors. We propose a novel Higher-order Attribute-Enhancing (HAE) framework that enhances node embedding in a layer-by-layer manner. Under the HAE framework, we propose a Higher-order Attribute-Enhancing Graph Neural Network (HAEGNN) for heterogeneous network representation learning. HAEGNN simultaneously incorporates meta-paths and meta-graphs for rich, heterogeneous semantics, and leverages the self-attention mechanism to explore content-based nodes interactions. The unique higher-order architecture of HAEGNN allows examining the first-order as well as higher-order neighborhoods. Moreover, HAEGNN shows good explainability as it learns the importances of different meta-paths and meta-graphs. HAEGNN is also memory-efficient, for it avoids per meta-path based matrix calculation. Experimental results not only show HAEGNN superior performance against the state-of-the-art methods in node classification, node clustering, and visualization, but also demonstrate its superiorities in terms of memory efficiency and explainability.

1 INTRODUCTION

The introduction identifies gaps in heterogeneous graph representation learning: existing methods use limited semantic structures and do not fully capture content-based relationships among higher-order neighbors. HAEGNN combines meta-paths, meta-graphs, and stacked attention layers, and the paper reports strong performance, efficiency, and explainability.

1 INTRODUCTION

  • Existing heterogeneous methods either rely on meta-paths, which may miss complex semantic similarities, or use complex neural technologies.Meta-graphs can capture richer semantics than meta-paths, and their combination can better capture entity similarities.Source: §1 · pp. 1–2
  • Existing methods also incompletely capture content-based relationships, especially among non-immediate neighbors that share neighbors.The introduction argues for attention that examines first-order, second-order, and higher-order neighborhoods.Source: §1 · p. 2
  • The HAE framework combines Semantic-based Convolutional Layers and Content-based self-Attention Layers to enhance node features layer by layer.HAEGNN uses one SCL followed by CALs; stacking CALs enables attention to higher-order neighbors.Source: §1 · p. 2
  • SCL learns weighted similarities from meta-path and meta-graph instances, then applies graph convolutions to combine node features with semantics.The resulting weighted adjacency connects target-type nodes in a converted homogeneous network.Source: §1 · p. 2
  • CAL applies content-based self-attention to nodes connected by meta-path or meta-graph relations, using node representations rather than neighbor order to score attention.The introduction positions this as a way to account for content-based interactions in node embeddings.Source: §1 · p. 2
  • HAEGNN combines meta-paths and meta-graphs with self-attention over immediate and higher-order neighbors for heterogeneous network representation learning.Its evaluations cover node classification, node clustering, and visualization on three datasets, with reported gains over strong baselines and memory efficiency.Source: §1 · p. 2

2 RELATED WORK

Related work spans graph neural networks, network representation learning, semantic modeling, and self-attention. The review positions HAEGNN as combining heterogeneous semantics with layered attention to address limitations in existing approaches.

2 RELATED WORK

  • GNNs apply graph convolutions or attention to nearby nodes, but standard approaches such as GCN and GAT neglect heterogeneous semantics.GAT uses masked self-attention for content-based interactions with one-step neighbors.Source: §2 · p. 3
  • Network representation learning produces low-dimensional node embeddings for downstream tasks, with heterogeneous methods using semantic structures such as meta-paths.The review distinguishes unsupervised approaches from semi-supervised GNN methods that use available labels to tailor embeddings.Source: §2 · p. 3
  • Existing semi-supervised methods often capture either limited heterogeneous semantics or incomplete content-based relationships between nodes.GTN learns task-oriented meta-paths but lacks node-level attention; HAN uses meta-paths and first-order attention but omits meta-graphs and long-range dependencies.Source: §2 · p. 3
  • Prior semantic approaches use meta-paths and/or meta-graphs; the review notes that meta-graphs can capture richer semantics and that combining both has shown better experimental results.HAEGNN fuses the structures before learning, rather than concatenating or mapping separate embeddings afterward.Source: §2 · pp. 3–4
  • Self-attention measures content-based similarity, while the review identifies higher-order attention as a gap in heterogeneous network methods.HAEGNN stacks CALs to attend to first-order and higher-order neighborhoods, alongside convolution through SCLs.Source: §2 · p. 4

3 PRELIMINARY

The preliminary section defines the network and schema concepts used to describe heterogeneous semantics. It explains how meta-paths and meta-graphs express relations between nodes of a target type.

3 PRELIMINARY

  • A heterogeneous information network is a graph whose nodes and edges may have multiple types.The DBLP example includes authors, papers, conferences, and terms, connected by several relation types.Source: §3 · p. 4
  • A meta-schema is a directed graph over a network’s node types and edge types.For DBLP, it summarizes the dataset’s node and edge types.Source: §3 · p. 4
  • A meta-path is a path on the meta-schema that defines a composite relation between node types; a network path following it is a meta-path instance.The DBLP example relates authors who publish papers in the same conference.Source: §3 · p. 4
  • A meta-graph is a directed acyclic graph on the meta-schema with one source and one sink.The example P4 relates authors through papers sharing both a conference and a keyword, expressing richer semantics than the conference-only meta-path P1.Source: §3 · p. 4
  • The paper collectively calls meta-paths and meta-graphs semantic structures and represents them by their component node types.The structures used begin and end with the same node type, designated the target type.Source: §3 · p. 4

4 THE PROPOSED MODEL

HAEGNN combines semantic convolution and content-based self-attention, then stacks these layers to enhance node embeddings across progressively higher-order neighborhoods. Its components support heterogeneous semantics from meta-paths and meta-graphs as well as attention to neighbors’ representations.

4 THE PROPOSED MODEL

  • HAEGNN uses one SCL followed by multiple CALs; its order is the total number of SCL and CAL layers.The initial features are enhanced layer by layer, with SCL adding semantics and CAL adding content-based interactions.Source: §4 · p. 5, Figure 1 · p. 5

4.1 Semantic-Based Convolutional Layer

  • SCL combines weighted meta-path and meta-graph instance counts into semantic similarity, then uses convolution to fuse node embeddings according to those similarities.Its trainable weights assign importance to each semantic structure.Source: §4.1 · pp. 5–6

4.2 Content-Based Self-Attention Layer

  • CAL applies masked self-attention to nodes connected by at least one meta-path or meta-graph instance, weighting neighbors’ representations to produce updated embeddings.The neighborhood includes the node itself, and multi-head attention concatenates independent attention outputs.Source: §4.2 · pp. 6–7

4.3 Higher-Order Attribute-Enhancing Framework

  • HAE stacks semantic and content-based layers so each output embedding becomes the next layer’s input, extending interactions from first-order to higher-order semantic neighborhoods.The framework permits arbitrary sequences of SCLs and CALs, including SCL-only and CAL-only variants.Source: §4.3 · p. 7

5 EXPERIMENTS

The experiments assess HAEGNN on node classification, node clustering, and visualization, alongside analyses of its semantic and higher-order design. They use three heterogeneous datasets and compare against unsupervised network-embedding methods, semi-supervised GNNs, and HAE framework variants.

5 EXPERIMENTS

  • The evaluation covers node classification, node clustering, and visualization, with additional analyses of semantic enhancement, higher-order strategy, memory consumption, and parameters.Experiments use three datasets.Source: §5 · p. 8

5.1 Datasets

  • DBLP authors, IMDB movies, and HUAWEI users are the prediction targets, with labels and initial bag-of-words features drawn from research areas and keywords, genres and plots, and age groups and behavioral attributes, respectively.The datasets contain heterogeneous node and edge types.Source: §5.1 · p. 8

5.2 Baselines

  • Baselines include unsupervised random-walk and attributed-network methods, semi-supervised GNNs, and HAE variants using only SCLs, only CALs, or different layer orders.Unsupervised methods include DeepWalk, Metapath2vec, DANE, and HONE; semi-supervised baselines include SDNE, GCN, GAT, HAN, GTN, RSHN, HetGNN, HGT, and MAGNN.Source: §5.2 · p. 8

5.3 Experimental Settings

The experimental settings standardize embedding dimensions and semi-supervised training ratios, specify HAE and random-walk hyperparameters, and use a multi-GPU system.

5.3 Experimental Settings

  • All models use 64-dimensional embeddings, and semi-supervised models use training ratios from 0.2 to 0.8.HAE learning rate is 0.0003; its SCL sub-layers number two and CALs use eight attention heads.Source: §5.3 · p. 8
  • HAE uses a 0.4 dropout rate in its first CAL, halving the rate at each following CAL, and initializes semantic-structure weights with Xavier.The models are optimized with Adam.Source: §5.3 · p. 8
  • Experiments ran on a 64-core Intel Xeon CPU with 512GB RAM and eight NVIDIA Tesla P100 GPUs; random-walk methods used window size 10, walk length 40, and 80 walks per node.CPU calculations used parallel processing across 64 cores.Source: §5.3 · p. 8

5.4 Node Classification

HAEGNN outperforms state-of-the-art baselines across DBLP, IMDB, and HUAWEI node-classification evaluations. Comparisons report gains over GTN, HAN, and RSHN, while results against HGT and HetGNN are described as comparable but slightly better for HAEGNN.

5.4 Node Classification

  • Up to 7.83% Ma-F1 and 7.37% Mi-F1 on DBLP over GTN; HAE4l GNN outperforms state-of-the-art baselines across all three datasets.The reported gains over GTN reach 5.61% Ma-F1 and 5.79% Mi-F1 on IMDB, and 6.02% Ma-F1 and 6.38% Mi-F1 on HUAWEI.Source: §5.4 · pp. 8–9, Table 1 · p. 9
  • HAE4l GNN improves over HAN by at least 2.05% Ma-F1 and 2.02% Mi-F1 on DBLP, 2.51% and 1.95% on IMDB, and 1.82% and 2.40% on HUAWEI.The authors relate the comparison to HAEGNN’s use of meta-graphs and higher-order interactions, which HAN omits.Source: §5.4 · p. 9
  • Against RSHN, HAE4l GNN improves Ma-F1 and Mi-F1 by at least 1.78% and 1.76% on DBLP, 2.66% and 2.10% on IMDB, and 1.53% and 2.17% on HUAWEI.The authors attribute the advantage to multi-hop relationships, such as meta-paths and meta-graphs, rather than the single-hop relations RSHN uses for information aggregation.Source: §5.4 · p. 9
  • HAE4l GNN improves over HGT and HetGNN by at least 0.16% Ma-F1 and 0.02% Mi-F1 on DBLP, 0.86% and 0.25% on IMDB, and 0.14% and 0.10% on HUAWEI.The paper describes these models’ performances as comparable overall.Source: §5.4 · p. 9
  • Semi-supervised GNNs generally outperform unsupervised methods, and heterogeneous-semantic models score higher than homogeneous models on all datasets.The reported heterogeneous-versus-homogeneous comparison concerns Ma-F1 and Mi-F1 scores.Source: §5.4 · p. 9

5.5 Node Clustering and Visualization

HAEGNN outperforms clustering baselines across datasets, and its DBLP embeddings show clearer cluster separation. The comparisons examine clustering metrics and visual organization of authors by research area.

5.5 Node Clustering and Visualization

  • The clustering evaluation uses K-means with the number of clusters set to the ground-truth class count, reporting average NMI, ARI, FMI, and standard deviations.The study uses five-fold cross-validation on each dataset and t-SNE to visualize DBLP embeddings.Source: §5.5 · p. 9
  • HAEGNN consistently outperforms all baselines by large margins across datasets in node clustering.The reported comparison covers NMI, ARI, and FMI.Source: §5.5 · pp. 9–10, Table 2 · p. 10
  • Against GTN, HAEGNN reports higher NMI, ARI, and FMI on DBLP, IMDB, and HUAWEI.The gains are 8.67%, 10.78%, and 5.70% on DBLP; 5.04%, 4.91%, and 6.14% on IMDB; and 4.43%, 4.43%, and 4.01% on HUAWEI, respectively.Source: §5.5 · p. 10
  • In the DBLP visualization, HAEGNN forms compact clusters with clear separations, while baseline embeddings show more overlap or blurrier boundaries.The visualization colors each author by research area; heterogeneous GNNs show fewer overlaps than homogeneous GCN and GAT.Source: Figure 4 · p. 10, §5.5 · p. 11

5.6 Comparison of the HAE Variants

The comparison evaluates HAEGNN, HAESCL, and HAECAL across classification, clustering, and visualization. Results favor combining semantic and content-based interactions with higher-order layers.

5.6 Comparison of the HAE Variants

  • HAE4l GNN outperforms HAE2l CAL in all reported classification cases, while HAE4l CAL wins four of six cases.The results indicate that semantic and content-based interactions and higher-order attention each contribute to classification performance, with their combination recommended by the authors.Source: §5.6 · p. 11
  • Most HAE variants outperform the best baseline results in node clustering.Table 4 reports clustering results and standard deviations alongside copied best-baseline scores.Source: §5.6 · p. 11
  • HAE4l GNN achieves consistent, significant, and stable improvements over the best baseline results.The comparison reports node-classification performance and standard deviations in Table 3.Source: §5.6 · p. 11
  • HAE4l CAL performs better than the other two HAE models in six of nine cases and yields more compact, clearly separated visualization clusters.The authors conclude that HAEGNN, combining the approaches, is generally the better-performing HAE model.Source: §5.6 · p. 11

5.7 Evaluation of Semantic Incorporation

The semantic-incorporation evaluation tests the contribution of meta-graphs and examines learned weights for semantic structures. Results support using meta-graphs and show that the model assigns them higher weights than meta-paths.

5.7 Evaluation of Semantic Incorporation

  • Removing meta-graphs reduces Macro-F1 by 0.3%–2.2% and Micro-F1 by 0.3%–1.7% across the evaluated HAE models.All models score higher when using meta-paths and meta-graphs together.Source: §5.7 · p. 12
  • Across all three datasets, HAEGNN assigns higher weights to meta-graphs than to meta-paths, indicating greater reliance on the meta-graphs.The authors relate this to the more complex semantics conveyed by meta-graphs.Source: §5.7 · p. 12
  • Learned weights distinguish semantic structures within datasets, such as higher weight for DBLP APTPA than APCPA and for IMDB MDM than MAM.The authors conclude that the weights identify important semantic structures and are interpretable.Source: §5.7 · p. 12

5.8 Evaluation of Higher-Order Attribute-Enhancing

Higher order improves HAEGNN classification up to order four, with performance depending on dataset and declining beyond that point. Higher-order models use more memory and time, though HAEGNN can remain more efficient than baselines.

5.8 Evaluation of Higher-Order Attribute-Enhancing

  • HAEGNN Macro-F1 and Micro-F1 generally rise through order four on all three datasets, then decline beyond four as performance begins to overfit.The authors attribute the rise to incorporating more CAL layers and attending to more distant neighbors.Source: §5.8 · p. 12, Figure 6 · p. 13
  • Higher order helps HAECAL less than HAEGNN, and its effects vary by dataset, with stronger gains on DBLP than on IMDB and HUAWEI.HAECAL performance also starts to drop when order exceeds four.Source: §5.8 · p. 13
  • On DBLP, order-two HAEGNN uses around 19.6% less memory than HAN and 7.4% less than GTN, and runs 100 epochs in 20 versus GTN’s 23 seconds.The memory comparison measures maximum usage over 100 epochs; the authors report that higher-order HAEGNN models require more memory, but the increments are not significant.Source: §5.8 · p. 13, Figure 7 · p. 13
  • Higher-order HAEGNN models take more time as their deeper structures add layers, while the authors conclude that the architecture improves performance.The paper reports superior performance with less memory and/or time than state-of-the-art baselines.Source: §5.8 · pp. 13–14

6 CONCLUSION AND FUTURE WORK

HAEGNN combines meta-path and meta-graph semantics with content-based interactions across first- and higher-order neighbors, and outperforms state-of-the-art baselines in node classification, node clustering, and visualization. The authors also identify future extensions to edge heterogeneity, sequence information, inductive learning in dynamics, and additional modalities.

6 CONCLUSION AND FUTURE WORK

  • HAEGNN outperforms state-of-the-art baselines in node classification, node clustering, and visualization.Experimental results also verify the effectiveness of its semantic-learning approaches and higher-order attribute-enhancing strategy.Source: §6 · p. 14
  • HAEGNN combines meta-paths and meta-graphs with content-based interactions among first- and higher-order neighboring nodes.Source: §6 · p. 14
  • Experimental results also demonstrate the model’s efficiency and the explainability of its results.Source: §6 · p. 14
  • The authors identify future work on edge heterogeneity, Transformer-based node and edge sequences, inductive learning in dynamics, and additional modalities such as images.Source: §6 · p. 14

Related papers

Loading 2104.07892v1…