PaperlayerView exact revision on arXiv

Source-linked AI summary

Heterogeneous Graph Masked Autoencoders

Yijun Tian, Kaiwen Dong, Chunhui Zhang, Chuxu Zhang, Nitesh V. Chawla

arXiv:2208.09957v2cs.LG

TL;DR

Heterogeneous graphs raise challenges for generative SSL because methods must account for complex structure, varying node attributes, and node positions. HGMAE combines masking techniques with reconstruction and prediction strategies to address these challenges, and experiments report superiority over state-of-the-art contrastive and generative baselines across multiple tasks and datasets.

Abstract

from arXiv · show

Generative self-supervised learning (SSL), especially masked autoencoders, has become one of the most exciting learning paradigms and has shown great potential in handling graph data. However, real-world graphs are always heterogeneous, which poses three critical challenges that existing methods ignore: 1) how to capture complex graph structure? 2) how to incorporate various node attributes? and 3) how to encode different node positions? In light of this, we study the problem of generative SSL on heterogeneous graphs and propose HGMAE, a novel heterogeneous graph masked autoencoder model to address these challenges. HGMAE captures comprehensive graph information via two innovative masking techniques and three unique training strategies. In particular, we first develop metapath masking and adaptive attribute masking with dynamic mask rate to enable effective and stable learning on heterogeneous graphs. We then design several training strategies including metapath-based edge reconstruction to adopt complex structural information, target attribute restoration to incorporate various node attributes, and positional feature prediction to encode node positional information. Extensive experiments demonstrate that HGMAE outperforms both contrastive and generative state-of-the-art baselines on several tasks across multiple datasets. Codes are available at https://github.com/meettyj/HGMAE.

Introduction

Generative SSL on heterogeneous graphs must address complex structure, node-specific attributes, and node positions, while existing approaches leave these challenges unresolved. HGMAE combines masking techniques and training strategies to model these aspects.

Introduction

  • Contrastive graph SSL depends on manually chosen augmentations and often complex optimization and negative sampling, while generative SSL reconstructs input data without those requirements.GraphMAE reconstructs node features but leaves heterogeneous structure, attributes, and positions unaddressed.Source: Introduction · p. 1
  • Heterogeneous graphs pose three challenges: capturing complex structure, choosing suitable attribute masking across node types, and encoding different node positions.The paper says reconstructing masked features alone is insufficient to capture structural semantics.Source: Introduction · pp. 1–2
  • HGMAE combines metapath masking and adaptive attribute masking, with a dynamic mask rate for effective and stable learning on heterogeneous graphs.Its training strategies reconstruct metapath-based edges and target attributes, and predict positional features.Source: Introduction · p. 2
  • The paper presents generative self-supervised learning on heterogeneous graphs as a problem and identifies three challenges the proposed HGMAE model addresses.HGMAE is described as capturing complex structure, varied attributes, and different node positions.Source: Introduction · p. 2

Related Work

Prior heterogeneous graph neural networks learn node representations but are described as requiring labels. Graph SSL includes contrastive and generative approaches, with augmentation dependence and heterogeneous-data challenges motivating this work.

Heterogeneous Graph Neural Networks

  • HAN encodes node- and semantic-level structures; MAGNN uses metapath information; HGT uses a transformer; and HetGNN captures structure and content, but these methods require node labels and cannot extract supervised signals from data itself.Source: Related Work · p. 2

Self-supervised Learning on Graphs

  • Contrastive graph SSL aligns views or distributions, but its performance depends on manually selected augmentations that vary across graphs.Examples include GCC [Qiu et al. 2020], GraphCL [You et al. 2020], and HeCo.Source: Related Work · p. 2
  • GraphMAE reconstructs node features [Hou et al. 2022], but it does not address complex structure, varied attributes, or different node positions in real-world graphs.The passage frames using heterogeneous graph information for self-supervised guidance as an open question.Source: Related Work · pp. 2–3

Problem Definition

The paper defines generative self-supervised learning on heterogeneous graphs as learning node embeddings that encode graph structure, attributes, and positions, then reconstructing graph input for downstream tasks.

Problem Definition

  • A heterogeneous graph is represented as G = (V, A, TV, TE, X, Φ), with node types, edge types, attributes, and metapaths.A metapath connects different node types through distinct edge types.Source: Problem Definition · p. 3
  • The task is to learn node embeddings that encode heterogeneous graph structure, attributes, and node positions, then use a decoder to reconstruct the input.The reconstructed input may be adjacency, attributes, or node positions.Source: Problem Definition · p. 3
  • The learned embeddings can support downstream graph-mining tasks such as node classification and node clustering.Source: Problem Definition · p. 3

Heterogeneous Graph Masked Autoencoders

HGMAE combines metapath masking and adaptive attribute masking with three training strategies to learn from heterogeneous graph structure, attributes, and positional information.

Metapath-based Edge Reconstruction

  • Metapath-based edge reconstruction masks and reconstructs each metapath adjacency separately, using semantic-level attention to weight the metapath losses.The strategy targets high-order relations and structural information.Source: Metapath-based Edge Reconstruction · p. 3

Target Attribute Restoration

  • Target attribute restoration masks attributes for target-type nodes and reconstructs them, using a rising mask rate plus leaving-unchanged and replacement strategies.The mask rate increases linearly by training epoch, from MINpa to MAXpa.Source: Target Attribute Restoration · pp. 3–4

Positional Feature Prediction

  • Positional feature prediction trains the encoder to represent metapath-aware positions by predicting Mp2vec features from latent embeddings, with an MLP decoder that omits graph structure.Its loss is combined with the metapath-reconstruction and attribute-restoration losses in the final objective.Source: Positional Feature Prediction · pp. 4–5

Experiments

The experiments compare HGMAE with ten baselines on four real datasets and describe the evaluation and implementation setup.

Experiments

  • The experiments include ablations, parameter sensitivity, and embedding visualization to assess HGMAE.Source: Experiments · p. 5

Experimental Setup

  • The evaluation uses DBLP, Freebase, ACM, and AMiner, comparing HGMAE with ten homogeneous and heterogeneous baselines.Baselines include GraphSAGE, GAE, DGI, GraphMAE, Mp2vec, HERec, HetGNN, DMGI, HeCo, and HAN.Source: Experimental Setup · p. 5
  • HGMAE uses HAN as its default encoder and decoder, with learning rate and early-stopping patience tuned.The learning rate is searched from 1e-4 to 5e-3, and early-stopping patience from 5 to 20.Source: Experimental Setup · pp. 5–6
  • All methods are evaluated over ten random-seed runs, with mean and standard deviation reported.Source: Experimental Setup · p. 6

Node Classification

HGMAE performs strongly on node clustering and generally outperforms baselines on node classification, with a few exceptions.

Node Classification

  • On node classification, HGMAE outperforms baselines across datasets except in a few cases, remaining second-best on AMiner.The authors note that HeCo is slightly better on a few AMiner F1 values; HGMAE is reported as more stable across datasets and settings.Source: Node Classification · p. 6

Node Clustering

  • HGMAE achieves the best node-clustering results across datasets, improving over HeCo by +17% NMI and +25% ARI on ACM.Table 2 presents the node-clustering performance comparison.Source: Node Clustering · p. 6

Ablation Study

Ablations support the contributions of HGMAE’s training strategies, while sensitivity analyses examine masking choices and hidden dimensions.

Ablation Study

  • Removing metapath-based edge reconstruction (MER) significantly reduces node-classification performance; removing target attribute restoration (TAR) or positional feature prediction (PFP) also lowers it.HGMAE retains the best results among the variants in all cases.Source: Ablation Study · p. 6

Impact of Dynamic Mask Rate

  • Dynamic mask rate (DMR) performs best against fixed mask rates and avoids manual mask-rate selection.A fixed rate of 0.9 is optimal for Freebase without DMR but performs poorly on DBLP, motivating adaptive learning across mask rates.Source: Impact of Dynamic Mask Rate · pp. 6–7
  • Figure 4 presents performance with respect to the leave-unchanged and replace rates, but its caption does not specify dataset-specific values or best-rate pairs.Source: Figure 4 · p. 7
  • Figure 5 reports HGMAE performance with respect to hidden dimensions.Source: Figure 5 · p. 7

Impact of Leaving Unchanged and Replacing

Leaving-unchanged and replacement masking improve HGMAE’s embeddings, but the best rates differ by dataset. Hidden-dimension sensitivity likewise varies across datasets, with overly wide layers reducing performance.

Impact of Leaving Unchanged and Replacing

  • DBLP performs best with leave-unchanged and replace rates of 0.3 and 0.1, while Freebase performs best at 0.0 and 0.1, respectively.For DBLP, higher leave-unchanged and lower replace rates are usually beneficial; Freebase shows the opposite pattern.Source: Impact of Leaving Unchanged and… · p. 7

Parameter Sensitivity

  • HGMAE’s optimal hidden dimension differs across datasets, and increasing it generally improves performance until the layer becomes too wide.The authors attribute degradation at very wide dimensions such as 1024 to the model’s focus being spread away from meaningful information.Source: Parameter Sensitivity · p. 7
  • Standard hidden dimensions such as 256 or 512 are sufficient for HGMAE to capture complex information and achieve superior performance.A small hidden dimension may prevent the model from fully capturing knowledge, while further increasing the dimension can degrade performance.Source: Parameter Sensitivity · p. 7

Embedding Visualization

The t-SNE visualization compares how learned embeddings group nodes by category. HGMAE shows clear category boundaries and dense clusters for nodes in the same category.

Embedding Visualization

  • HGMAE’s t-SNE embeddings show clear category boundaries and dense same-category clusters, unlike the fragmented or poorly separated groupings described for the other models.Mp2vec mixes categories; HeCo separates categories but does not cluster same-category nodes, while GraphMAE forms separated chunks that can split a category.Source: Embedding Visualization · p. 7
  • Mp2vec mixes nodes from different categories in the visualization.Source: Embedding Visualization · p. 7
  • HeCo separates categories but does not cluster nodes that share a category.Source: Embedding Visualization · p. 7
  • GraphMAE separates categories but forms small chunks, including multiple chunks for nodes in the same category.Source: Embedding Visualization · p. 7

Conclusion

The paper formalizes generative self-supervised learning on heterogeneous graphs and proposes HGMAE, a masked autoencoder for this setting. It jointly considers graph structure, node attributes, and node positions, and experiments across multiple datasets and tasks show superiority to state-of-the-art methods.

Conclusion

  • HGMAE jointly addresses heterogeneous graph structure, node attributes, and node positions through two masking techniques and three training strategies.Experiments on multiple datasets and tasks demonstrate superiority over state-of-the-art methods.Source: Conclusion · p. 7
Loading 2208.09957v2…