PaperlayerView exact revision on arXiv

Source-linked AI summary

HDMI: High-order Deep Multiplex Infomax

Baoyu Jing, Chanyoung Park, Hanghang Tong

arXiv:2102.07810v5cs.LGcs.ITcs.SI

TL;DR

DGI leaves node-attribute dependence underused and does not fully exploit multiple relation types in networks. HDMI combines extrinsic and intrinsic signals with high-order mutual information, fuses multiplex layers through attention, and reports state-of-the-art results across evaluated downstream tasks.

  • Problem

    DGI focuses on node-summary mutual information while underusing node attributes and assuming a single relation type, despite multiplex networks containing varied relations.

  • Method

    HDMI uses high-order mutual information to combine extrinsic and intrinsic signals, optimizes them with HDI, and fuses embeddings from multiplex layers using attention.

  • Results

    HDMI achieves state-of-the-art performance on the evaluated supervised and unsupervised downstream tasks.

  • Takeaways & Limitations

    HDMI offers a self-supervised embedding framework for attributed multiplex networks that incorporates node attributes and multiple relation layers.

Abstract

from arXiv · show

Networks have been widely used to represent the relations between objects such as academic networks and social networks, and learning embedding for networks has thus garnered plenty of research attention. Self-supervised network representation learning aims at extracting node embedding without external supervision. Recently, maximizing the mutual information between the local node embedding and the global summary (e.g. Deep Graph Infomax, or DGI for short) has shown promising results on many downstream tasks such as node classification. However, there are two major limitations of DGI. Firstly, DGI merely considers the extrinsic supervision signal (i.e., the mutual information between node embedding and global summary) while ignores the intrinsic signal (i.e., the mutual dependence between node embedding and node attributes). Secondly, nodes in a real-world network are usually connected by multiple edges with different relations, while DGI does not fully explore the various relations among nodes. To address the above-mentioned problems, we propose a novel framework, called High-order Deep Multiplex Infomax (HDMI), for learning node embedding on multiplex networks in a self-supervised way. To be more specific, we first design a joint supervision signal containing both extrinsic and intrinsic mutual information by high-order mutual information, and we propose a High-order Deep Infomax (HDI) to optimize the proposed supervision signal. Then we propose an attention based fusion module to combine node embedding from different layers of the multiplex network. Finally, we evaluate the proposed HDMI on various downstream tasks such as unsupervised clustering and supervised classification. The experimental results show that HDMI achieves state-of-the-art performance on these tasks.

1 INTRODUCTION

The paper identifies two limitations of DGI: it omits node-attribute dependence and assumes a single relation type. HDMI addresses these gaps with high-order mutual information and multiplex-layer fusion.

  • Evaluation: The paper evaluates HDMI on real-world datasets using multiple metrics and downstream network-embedding tasks.The stated evaluation includes node classification, clustering, and link-related network tasks.
  • Motivation: DGI models dependence between node embeddings and global summaries but does not fully capture dependence between embeddings and node attributes.Node attributes can contain discriminative information and reveal relations absent from network connectivity.
  • Motivation: Real-world networks often contain multiple relation types, whereas DGI assumes a single relation type.Multiplex networks represent these relations as separate layers, each containing one relation type.
  • Approach: HDMI combines extrinsic and intrinsic mutual information through a high-order mutual-information supervision signal.The signal is designed for attributed networks and attributed multiplex networks.
  • Approach: High-order Deep Infomax optimizes the proposed supervision signal, while an attention-based fusion module combines embeddings from different multiplex layers.The fusion module is trained using the high-order mutual-information signal.

2 PRELIMINARIES

The preliminaries define attributed multiplex networks and high-order mutual information, then describe DGI as mutual-information maximization between node embeddings and a global summary. DGI uses corruption, shared encoding, readout, and discrimination to distinguish positive from negative embeddings.

  • Attributed Networks: An attributed network consists of an adjacency matrix and an attribute matrix describing node relations and node attributes.The matrices are denoted A and F, with N nodes and attribute dimension d_F.
  • Attributed Multiplex Networks: An attributed multiplex network contains multiple attributed layers with different node relations that share one node attribute matrix.Each layer has its own adjacency matrix, while the attribute matrix is common across layers.
  • Mutual Information: Mutual information measures mutual dependence between two random variables, while high-order mutual information generalizes this concept to three or more variables.High-order mutual information also captures synergy among multiple variables.
  • Deep Graph Infomax: DGI learns node embeddings on attributed networks by maximizing mutual information between node embeddings and a global summary vector.Its objective uses a discriminator to distinguish positive node embeddings from negative embeddings.
  • Deep Graph Infomax: DGI creates a corrupted negative network, applies the same encoder to positive and negative networks, and computes the summary from positive embeddings.A readout function such as average pooling produces the summary before discrimination.

3 METHODOLOGY

The methodology introduces HDI, which jointly models node embeddings, global summaries, and node attributes through high-order mutual information, then extends it to multiplex networks with attention-based layer fusion.

  • 3.1 High-order Deep Infomax: HDI jointly maximizes mutual information among node embedding h_n, global summary s, and node attributes f_n.Its high-order formulation combines extrinsic dependence I(h_n; s), intrinsic dependence I(h_n; f_n), and their interaction I(h_n; s, f_n).
  • 3.1.1 High-order Mutual Information Estimation.: High-order mutual information is estimated by separately estimating I(h_n; s), I(h_n; f_n), and I(h_n; s, f_n), then combining them by difference-based estimation.The estimated quantity is written as ^I(h_n; s; f_n) = ^I(h_n; s) + ^I(h_n; f_n) − ^I(h_n; s, f_n).
  • 3.1.1 High-order Mutual Information Estimation.: HDI maximizes a weighted objective containing extrinsic, intrinsic, and joint mutual-information terms, with tunable coefficients λ_E, λ_I, and λ_J.The three terms correspond to I(h_n; s), I(h_n; f_n), and I(h_n; s, f_n).
  • 3.1.1 High-order Mutual Information Estimation.: The joint signal contrasts (h_n, s, f_n) with (h_n, s, ˜f_n) to capture dependence, especially between the global summary and node attributes.Negative attributes are used because the extrinsic and intrinsic signals already distinguish their respective positive and negative pairs.
  • 3.1.2 Model Architecture.: The network encoder is a single-layer GCN, the readout function uses average pooling, and corruption randomly shuffles rows of the attribute matrix.The joint discriminator projects attributes and summaries into a common hidden space before applying a bi-linear scoring function.
  • 3.2 High-order Deep Multiplex Infomax: HDMI learns embeddings separately on multiplex layers and combines them using semantic attention trained with high-order mutual information.The fused embedding is used to form the multiplex global summary by average pooling, while the fusion objective is jointly combined with layer objectives.

4 EXPERIMENTS

The experiments are designed to evaluate whether HDI and HDMI improve learned node embeddings and whether the fusion module assigns proper attention scores.

  • Experimental Questions: The experiments examine how HDI and HDMI affect the quality of learned node embeddings.This is posed as the first experimental question.
  • Experimental Questions: The experiments also examine whether the fusion module assigns proper attention scores to multiplex-network layers.This is posed as the second experimental question.

4.1 Experimental Setup

The evaluation uses four multiplex datasets, compares HDMI with network and multiplex embedding baselines, and measures supervised and unsupervised downstream performance.

  • Datasets: The datasets comprise ACM, IMDB, DBLP, and Amazon multiplex networks with attributed nodes and multiple relation types.ACM has two relations and three classes; IMDB has two relations and three classes; DBLP has three relations and four categories; Amazon has three relations and four categories.
  • Comparison Methods: The comparison includes methods that disregard attributes, methods that use attributes, and multiplex network embedding methods.The baselines include DeepWalk, node2vec, GCN, GAT, DGI, ANRL, CMNA, MNE, mGCN, HAN, DMGI, and DMGIattn.
  • Evaluation Tasks and Metrics: HDMI is evaluated on node classification, node clustering, and similarity search.Macro-F1 and Micro-F1 measure classification, NMI measures clustering, and Sim@5 measures same-label neighbors among the top five similar nodes.
  • Evaluation Procedure: Classification and clustering train self-supervised embeddings before logistic regression or K-means, while similarity search uses cosine similarity and top-five neighbors.The evaluation reports Macro-F1, Micro-F1, NMI, and Sim@5.
  • Implementation: Node embeddings have dimension 128, models use Adam with learning rate 0.001, and early stopping uses patience 100.HDMI uses the same discriminator across layers, while coefficients are tuned by grid search.

4.2 Quantitative Evaluation

Quantitative experiments evaluate HDMI on supervised and unsupervised multiplex-network tasks. HDMI outperforms state-of-the-art methods on classification and achieves higher scores in most unsupervised cases, while its fusion and supervision ablations support the proposed design.

  • Overall Performance: HDMI outperforms state-of-the-art models on all supervised tasks and achieves higher scores in most unsupervised tasks.The evaluated tasks are node classification, node clustering, and similarity search.
  • Overall Performance: HDMI produces more discriminative embeddings than competing methods according to the reported supervised and unsupervised results.HDI is generally competitive with state-of-the-art methods.
  • Fusion Module: HDMI outperforms HDI on every reported metric and dataset except ACM Sim@5, where the performance gap is very small.This comparison evaluates the proposed fusion module against average pooling.
  • Supervision Signals: Adding intrinsic mutual dependence improves performance over the extrinsic signal alone, and E.+I. outperforms E.+R.The comparison contrasts mutual-information-based attribute modeling with attribute reconstruction error.
  • Supervision Signals: The joint E.+I.+J. signal further improves node-embedding discriminative ability for node classification.Combining embeddings from different multiplex layers also yields better results, suggesting mutual assistance across layers.

4.3 Qualitative Evaluation

Qualitative visualizations show that intrinsic and joint supervision improve class separation, while multiplex fusion further separates embeddings and assigns greater attention to stronger layers.

  • Node Embedding of an Attributed Network: Intrinsic and joint supervision significantly improve the discriminative ability of Amazon node embeddings compared with the extrinsic signal.The joint signal further improves embedding quality, whereas reconstruction error does not substantially track discriminative ability.
  • Node Embedding of a Multiplex Network: Average pooling separates ACM nodes better than the PAP layer alone, and the fusion module separates them better than average pooling.The comparison is shown for embeddings from the PSP and PAP layers.
  • Attention Scores: Layers with higher Micro-F1 scores generally receive higher attention scores in the multiplex fusion module.Figure 6 visualizes attention scores alongside Micro-F1 scores for each layer.

(a) ACM

Figure 6 compares attention scores with Micro-F1 scores for each multiplex-network layer.

  • ACM: The figure presents attention scores and Micro-F1 scores for each multiplex-network layer.Use it to compare whether layers with stronger Micro-F1 scores also receive stronger attention.

5 RELATED WORKS

Related work covers self-supervised network embedding, graph neural networks, multiplex network embedding, and mutual-information methods. These approaches use proximity, aggregation, attention, random walks, negative sampling, or mutual-information objectives.

  • Self-supervised Network Representation Learning: Self-supervised network embedding methods learn node representations from network structure without external supervision.
  • Multiplex Network Embedding: Multiplex network methods model multiple network layers using random walks, negative sampling, preservation and collaboration, or unified embeddings.
  • Graph Neural Networks: Graph neural network methods include GCN, GraphSAGE, GAT, and DGCN, using aggregation, neighbor attention, or local-global consistency.
  • Multiplex Network Methods: Attention-based methods merge node embeddings across multiplex layers, while other approaches address network alignment, dynamic knowledge graphs, or tensor graphs.
  • Mutual Information Methods: Mutual-information research includes neural estimators and infomax objectives for self-supervised representation learning, including DGI for network embeddings.

6 CONCLUSION

The paper introduces HDMI for self-supervised network embedding on multiplex networks. It combines high-order mutual-information supervision with attention-based layer fusion and reports effectiveness across four real-world datasets and supervised and unsupervised tasks.

  • HDMI learns network embeddings for multiplex networks through self-supervised learning.
  • HDI uses high-order mutual information to jointly capture extrinsic, intrinsic, and interaction signals.
  • An attention-based fusion module combines node embeddings from different multiplex network layers.
  • Evaluation on four real-world datasets and supervised and unsupervised downstream tasks demonstrates the effectiveness of HDMI, HDI, and the fusion module.
Loading 2102.07810v5…