PaperlayerView exact revision on arXiv

Source-linked AI summary

Do We Really Need Complicated Model Architectures For Temporal Networks?

Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, Mehrdad Mahdavi

arXiv:2302.11636v1cs.LGcs.AI

TL;DR

Temporal graph learning often uses RNNs or self-attention, but the paper asks whether they are always necessary. It proposes GraphMixer, a simpler MLP- and mean-pooling-based architecture, and reports strong benchmark performance with faster convergence and better generalization. The findings motivate reconsidering the importance of simpler architectures and input structures.

  • Problem

    RNNs and self-attention dominate temporal graph learning, but their complexity leaves it unclear whether these components are indispensable.

  • Method

    GraphMixer uses an MLP-based link encoder, neighbor mean-pooling node encoder, and MLP link classifier without RNNs or self-attention.

  • Results

    GraphMixer outperforms baselines and achieves state-of-the-art temporal link-prediction performance with faster convergence and better generalization.

  • Takeaways & Limitations

    The results motivate rethinking the importance of simpler neural architectures and input data structures.

  • Takeaways & Limitations

    Experiments use a sub-sampled GDELT dataset and generally rely on 1-hop neighbors, with one-hot node representations when node features are unavailable.

Abstract

from arXiv · show

Recurrent neural network (RNN) and self-attention mechanism (SAM) are the de facto methods to extract spatial-temporal information for temporal graph learning. Interestingly, we found that although both RNN and SAM could lead to a good performance, in practice neither of them is always necessary. In this paper, we propose GraphMixer, a conceptually and technically simple architecture that consists of three components: (1) a link-encoder that is only based on multi-layer perceptrons (MLP) to summarize the information from temporal links, (2) a node-encoder that is only based on neighbor mean-pooling to summarize node information, and (3) an MLP-based link classifier that performs link prediction based on the outputs of the encoders. Despite its simplicity, GraphMixer attains an outstanding performance on temporal link prediction benchmarks with faster convergence and better generalization performance. These results motivate us to rethink the importance of simpler model architecture.

1 INTRODUCTION

Temporal graph learning commonly relies on RNNs or self-attention, but GraphMixer asks whether these components are indispensable and proposes a simpler alternative. The architecture achieves strong temporal link-prediction results while converging faster and generalizing better.

  • Motivation: Temporal graph learning supports applications such as predicting future user-ad clicks from timestamped interactions.Temporal link prediction models evolving node interactions in settings such as recommender systems.
  • Research question: RNNs and self-attention are de facto standards, yet their architectural complexity makes it difficult to identify which components are truly necessary.The paper frames this uncertainty as whether RNN and SAM components are indispensable for temporal graph learning.
  • Proposed architecture: GraphMixer uses MLPs and neighbor mean-pooling without RNNs or self-attention.Its design is intended as a conceptually and technically simple architecture for temporal graph learning.
  • Results: GraphMixer achieves state-of-the-art performance across average precision, AUC, Recall@K, and MRR on real-world temporal graph datasets.It does so against baselines equipped with RNNs and self-attention, while using fewer model parameters and hyper-parameters.
  • Analysis: The study identifies simplicity of the architecture and input data, fixed time encoding, and a link encoder as factors contributing to GraphMixer’s success.The authors connect these factors to performance, convergence, distinguishability of temporal sequences, and generalization in the cited discussion.

2 PRELIMINARY AND EXISTING WORKS

The paper introduces temporal link prediction and reviews representative temporal graph-learning baselines. These methods differ in how they encode temporal and spatial information, using RNNs, self-attention, graph attention, temporal snapshots, random walks, or collaborative attention.

  • Preliminary: Temporal link prediction aims to determine whether two nodes will interact at a future timestamp using prior temporal graph information.The preliminary setup uses graph structure, node features, and link features observed before the prediction time.
  • Existing works: Existing temporal graph-learning methods are often conceptually and technically complicated, motivating comparison with representative closely related approaches.The paper selects four representative methods for detailed comparison and also considers additional baselines.
  • RNN-based methods: JODIE uses RNNs to update node representations after interactions, then applies time-dependent evolution and an MLP to predict node-pair links.Its dynamic node embedding scales the latest interaction representation according to elapsed time before pairwise prediction.
  • Self-attention methods: DySAT constructs snapshot graphs, applies GAT independently to capture spatial information, and uses a Transformer across timestamps for temporal information.Link prediction is subsequently performed with an MLP over the resulting node representations.
  • Self-attention methods: TGAT combines raw node features with trainable time encodings and applies self-attention to capture spatial and temporal information simultaneously.Its time encoding is described as z(t) = cos(tw+b).
  • Additional baselines: CAWs combines temporal random-walk motifs, RNN walk encoding, and self-attention over multiple walks, while TGSRec uses collaborative attention for temporal recommendation.These baselines extend the architectural complexity of representative temporal graph-learning approaches.

3 GRAPHMIXER: A CONCEPTUALLY AND TECHNICALLY SIMPLE METHOD

GraphMixer is a conceptually and technically simple temporal link-prediction architecture that summarizes temporal links with an MLP-based link-encoder, nodes with neighbor mean-pooling, and predictions with an MLP classifier. Its design uses recent temporal interactions, time encoding, zero-padding, and an undirected 1-hop-neighbor view instead of more complex multi-hop, RNN, or self-attention mechanisms.

  • GraphMixer architecture: GraphMixer has three modules: a link-encoder, a neighbor-mean-pooling node-encoder, and an MLP link classifier.The classifier predicts link existence from the two encoders’ outputs.
  • Link-encoder: The link-encoder retains the top K most recent temporal links and distinguishes timestamps with time encoding and link information with an MLP-mixer.Temporal link records are sorted by timestamp; ties retain raw-input order.
  • Link-encoder: The time-encoding function maps timestamp t to cos(tω), producing similar encodings for similar timestamps while later timestamps converge toward +1.The encoding is represented across vector dimensions, with cosine values varying by dimension.
  • Link-encoder: The MLP-mixer applies mean-pooling to the padded temporal-link matrix T2(t0), producing a single temporal encoding t2(t0).Zero-padding also conveys interaction frequency: more padded dimensions indicate fewer temporal linked neighbors.
  • Node-encoder: The node-encoder uses 1-hop neighbors from a recent time window to summarize node identity and features, with one-hot representations when node features are unavailable.In practice, the method uses 1-hop neighbors and a dataset-dependent window T.
  • Comparison to existing methods: GraphMixer treats temporal graphs as undirected and selects only 1-hop most recent neighbors, unlike methods using directed information flow, multi-hop sampling, or RNN-maintained histories.The undirected view makes recent-neighbor similarity available for link prediction.

4 EXPERIMENTS

Experiments evaluate GraphMixer across real-world temporal graph datasets, comparing it with baselines, training behavior, time-encoding choices, and link-encoder alternatives. The studies also examine input-data simplicity and the role of node and link information.

  • Experimental Setup: Experiments use five real-world datasets with chronological 70%/15%/15% train, validation, and test splits.GDELT additionally has variants that remove link features or both node and link features.
  • Main Empirical Results: GraphMixer outperforms all baselines on every dataset in average precision, supporting that neither RNN nor SAM is necessary for temporal link prediction.GraphMixer-N alone is insufficient on Wiki, MOOC, and LastFM, while node information benefits the link-encoder.
  • Main Empirical Results: GraphMixer reaches high training average precision within a few epochs with smoother curves and has a smaller, smoother generalization gap than baselines.Figure 3 compares training average precision and generalization gap over the first 100 epochs.
  • Fixed Time-Encoding Function: GraphMixer has a smoother training loss landscape, while replacing baselines’ trainable time encoding with the fixed function can flatten their landscapes and improve performance.Table 2 compares average precision before and after replacing trainable time encoding with fixed time encoding.
  • MLP-Mixer in the Link-Encoder: Self-attention alternatives degrade performance relative to the MLP-mixer, especially with mean-pooling, which cannot distinguish identical timestamp-feature sequences or explicitly capture sequence length.The comparison evaluates full and 1-hop self-attention alongside sum- and mean-pooling alternatives.
  • Key Factors to Better Performance: Simpler input data aligned with labels allows a simple network to capture the underlying mapping and can improve generalization.Input-data variants test the contributions of node and link information.

5 CONCLUSION

GraphMixer is a conceptually and technically simple architecture for temporal link prediction that outperforms all baselines while converging faster and generalizing better. An extensive study identifies factors behind its success and highlights simpler architectures and input structures.

  • GraphMixer is proposed as a conceptually and technically simple architecture for temporal link prediction.
  • GraphMixer outperforms all baselines while achieving faster convergence and better generalization ability.
  • An extensive study identifies three key factors contributing to GraphMixer’s success.
  • The findings highlight the importance of simpler neural architectures and input data structures.

A.1 HARDWARE SPECIFICATION AND ENVIRONMENT

The experiments use a unified temporal graph learning framework, specified hardware and software, multiple datasets, and standardized training and evaluation procedures. GraphMixer uses default framework settings with two additional hyperparameters.

  • Experiments run on an Intel i9-10850K machine with an Nvidia RTX 3090 GPU and 64GB RAM, using Python 3.8, PyTorch 1.12.1, and CUDA 11.6.
  • The datasets include Reddit, Wikipedia, LastFM, and other temporal interaction benchmarks with link features derived from posts, edits, or listening information.
  • GraphMixer is implemented under the TGL framework with default hyperparameters including learning rate 0.0001, weight decay 10^-6, batch size 600, and hidden dimension 100.
  • GraphMixer has two introduced hyperparameters: the number of 1-hop most recent neighbors K and the time-slot size T.
  • GraphMixer and baselines use a unified mini-batch and data-preparation process that samples positive and negative node pairs and neighboring nodes.

B.1 RECENT METHODS THAT WE DO NOT COMPARE WITH

The paper omits several related temporal graph methods because some implementations cannot reproduce reported results and the authors consider their baseline set sufficient. It also clarifies that complexity is not inherently bad and uses Figure 8 to compare training average precision and generalization gap.

  • GraphMixer is not compared with some related temporal graph algorithms because official implementations are unavailable or reported results cannot be reproduced.
  • The authors consider their many recent baselines sufficient to verify GraphMixer’s success.
  • Related methods include data augmentation, transformers, contrastive learning, temporal-aware sampling, and recurrent message propagation.
  • The authors do not claim that conceptually or technically complicated methods are bad; simpler methods may be preferred when performance is similar.
  • Existing methods can require nontrivial concepts, node memories, temporal random walks, or multiple augmentation strategies that complicate implementation and tuning.
  • Figure 8 compares link-prediction training average precision and generalization gap during the first 100 training epochs.

C.2 TRANSDUCTIVE LEARNING WITH RECALL@K AND MRR AS EVALUATION METRIC

The evaluation supplements average precision and AUC with Recall@K and MRR, reporting GraphMixer’s performance across datasets and comparing training time and model parameters. Per-epoch preprocessing adds overhead but can be cached because neighbor selection is deterministic.

  • Recall@K and MRR evaluate whether positive temporal links rank above 100 sampled negative destination nodes.
  • GraphMixer achieves outstanding performance across all datasets, especially on the denser LastFM and GDELT graphs.
  • Recall@K and MRR reveal weaker behavior for some baselines, including TGN on LastFM and TGAT on LastFM and GDELT.
  • GraphMixer takes significantly less computation time than CAWs, TGSRec, and DDGCL, while being close to or slightly faster than several other baselines.
  • Per-epoch data preparation takes 41 seconds on Reddit, 9 on Wiki, 20 on MOOC, 48 on LastFM, and 71 on GDELT.
  • Caching preprocessed data removes repeated preparation after the first epoch because neighbor selection is deterministic and input data do not change.

C.4 TRANSDUCTIVE LEARNING WITH AUC AS EVALUATION METRIC

GraphMixer outperforms all baselines on AUC across datasets, with especially large gains on LastFM, GDELT-ne, and GDELT-e. These results support the claim that neither RNN nor SAM is necessary for temporal graph link prediction.

  • GraphMixer outperforms all baselines on every dataset under AUC evaluation.
  • More than 1% gain occurs over all baselines on LastFM, GDELT-ne, and GDELT-e.
  • Around 2% gain occurs over non-RNN DySAT and TGAT on Wiki.
  • Around 11% gain occurs over non-RNN DySAT and TGAT on GDELT-ne.
  • GraphMixer-L’s results indicate that its time-encoding function can preprocess timestamps into meaningful vectors on MOOC, LastFM, and GDELT-ne.

C.5 BASELINES WITH UNDIRECTED TEMPORAL GRAPH

The section tests baseline methods with undirected temporal graphs and compares average precision and AUC. Using undirected temporal graphs does not substantially improve baseline performance because the relevant information is already captured by their architectures or sampling methods.

  • Using an undirected temporal graph cannot improve baseline performance substantially.
  • Baseline architectures or sampling methods already implicitly capture the information provided by an undirected temporal graph.
  • The comparison evaluates baselines using average precision and AUC scores with an undirected temporal graph.

D DISCUSSION ABOUT MODEL PERFORMANCE ON LASTFM

GraphMixer’s large LastFM advantage is attributed to a composite effect of multiple dataset and modeling factors. LastFM has unusually large average time gaps, node degree, and maximum timestamps relative to the other datasets.

  • GraphMixer’s large margin over baselines on LastFM is attributed to a composite effect of multiple factors.
  • Larger average time-gap: LastFM’s average time gap is 106, compared with 4 for Reddit, 17 for Wiki, 3.6 for MOOC, and 0.1 for GDELT.
  • Larger average node degree: LastFM’s average node degree is 653, compared with 61 for Reddit, 17 for Wiki, 57 for MOOC, and 216 for GDELT.
  • RNN- and SAM-based baselines may be less ideal with large time gaps, dense graphs, and large timestamps, whereas GraphMixer is less affected by these conditions.
  • Larger maximum timestamp: LastFM’s maximum timestamp is 137 million, compared with 0.2 million for GDELT and 2.6 million for Reddit, Wiki, and MOOC.

E.1 LOSS LANDSCAPE ON WIKI

The section presents training loss landscapes for Wiki and other temporal graph datasets using contour and surface visualizations. It also includes fixed-time-encoding landscapes for TGAT, TGN, and JODIE.

  • Wiki training loss landscapes are compared using contour and surface visualizations.
  • Reddit training loss landscapes are shown with both contour and surface comparisons.
  • MOOC training loss landscapes are shown with both contour and surface comparisons.
  • LastFM training loss landscapes are shown with both contour and surface comparisons.
  • GDELT-ne and GDELT-e training loss landscapes are each shown with contour and surface comparisons.
  • Additional contour and surface landscapes show TGAT, TGN, and JODIE with fixed time-encoding functions.
Loading 2302.11636v1…