PaperlayerView exact revision on arXiv

Source-linked AI summary

TTGBench: Benchmarking Topological Evolution and Semantic Drift in Text-attributed Temporal Graphs

Longfei Ma, Zemin Liu, Fei Wu

arXiv:2609.08226v1cs.AI

TL;DR

Existing temporal graph benchmarks emphasize structural evolution, provide limited semantic-drift evaluation, and often rely on repeated edges that can inflate performance. TTGBench introduces six text-rich datasets with Dual Volatility and evaluates 17 TGNN- and LLM-based methods, revealing complementary strengths rather than a paradigm that models both dimensions well.

  • Problem

    Existing benchmarks largely emphasize temporal link prediction, restrict semantic evaluation to simple binary classification, and contain repeated edges that can overestimate performance.

  • Method

    TTGBench jointly evaluates structural and semantic evolution using six real-world text-attributed datasets with Dual Volatility, including evolving semantic labels and textual attributes.

  • Results

    17-method evaluation reveals a capability divide: TGNN-based predictors excel at temporal link prediction but struggle with temporal node classification, whereas LLM-based predictors show the opposite trend.

  • Takeaways & Limitations

    Current paradigms do not simultaneously model structural and semantic evolution, motivating models that unify both forms of temporal learning.

  • Takeaways & Limitations

    Developing efficient and scalable architectures remains essential for practical deployment.

Abstract

from arXiv · show

Temporal graph learning models the evolution of dynamic systems, where both structural interactions and semantic states change over time. However, existing benchmarks primarily emphasize structural evolution via temporal link prediction (TLP), while support for semantic evolution remains limited. Although temporal node classification (TNC) is sometimes included, it is typically restricted to simplistic binary settings that fail to capture realistic semantic drift. Moreover, commonly used datasets exhibit high link repetition, leading to inflated performance estimates and obscuring true model capability. To address these limitations, we introduce \textbf{TTGBench}, a new benchmark that jointly evaluates structural and semantic evolution. TTGBench comprises six real-world, text-rich datasets characterized by \emph{Dual Volatility}, enabling rigorous and fair evaluation of existing models. Notably, it is the first benchmark to support both multi-class and multi-label TNC, filling a critical gap in evaluating temporal semantic drift. We conduct a comprehensive evaluation of 17 state-of-the-art methods across Temporal Graph Neural Networks (TGNNs) and Large Language Model (LLM)-based paradigms. The results reveal a clear \emph{capability divide} between the two paradigms: TGNN-based methods excel at structural prediction but fail at semantic tracking, whereas LLM-based predictors show the opposite trend. Through in-depth analysis, we uncover their fundamental limitations and provide insights for developing more comprehensive temporal graph models.

1 Introduction

TTGBench addresses benchmark gaps by jointly evaluating structural evolution and semantic drift under realistic dual volatility. Its evaluation of 17 methods reveals a capability divide: TGNN-based predictors favor structural prediction, while LLM-based predictors favor semantic tracking.

  • Benchmark limitations: Existing benchmarks emphasize temporal link prediction, while semantic drift is often reduced to simple binary temporal node classification.Rich, dynamically evolving semantic labels remain scarce.
  • Benchmark limitations: Repeated edges can inflate benchmark results, with some methods exceeding 98% and obscuring capability in high-novelty environments.The issue arises because many existing datasets contain substantial edge repetition.
  • TTGBench: TTGBench jointly evaluates structural and semantic evolution across six real-world datasets exhibiting high link novelty and frequently changing, complex labels.These properties are termed Dual Volatility.
  • TTGBench: TTGBench supports multi-class and multi-label temporal node classification, enabling more diverse evaluation of semantic dynamics.The benchmark also includes rich textual attributes across its datasets.
  • Evaluation findings: 17 evaluated methods reveal a capability divide: TGNN-Predictors excel at temporal link prediction but struggle with temporal node classification, whereas LLM-Predictors show the opposite trend.The findings expose limitations in simultaneously modeling structural and semantic evolution.
  • Evaluation findings: Analyses identify structural deficiencies in LLM-based predictors and architectural bottlenecks in TGNNs for semantic tracking, alongside performance-efficiency trade-offs.These analyses are intended to inform development of more effective temporal graph models.

2 Related Work

Prior work established datasets and evaluation pipelines for temporal graph learning, introduced text-attributed temporal graphs, and began exploring LLM-based approaches. However, the supplied related-work passages indicate that existing efforts remain focused largely on structural or static-graph settings.

  • Temporal graph benchmarks: DGB, DyGLib, TGB, and DGraph provide diverse datasets and standardized training and evaluation pipelines.These benchmarks helped establish unified evaluation practices for temporal graph research.
  • Temporal graph benchmarks: DTGB introduced text-attributed temporal graph datasets, while TGB-Seq highlighted excessive edge repetition and proposed higher-novelty datasets.The passage positions novelty as an explicit benchmark-design concern.
  • LLMs for temporal graph learning: LLM-based temporal graph work includes LLM-as-Enhancer methods and TGTalker, but most LLM graph-learning studies focus on static graphs.TGTalker uses LLMs as predictors through in-context learning.

3 Task Formulation

TTGBench evaluates temporal graphs through two continuous-time tasks: predicting future interactions for structural evolution and future node semantics for semantic evolution. Its TNC formulation supports both multi-class and multi-label dynamics.

  • A text-attributed temporal graph combines timestamped interactions with edge text and time-dependent node labels representing instantaneous preferences.
  • Temporal Link Prediction (TLP): TTGBench defines TLP as predicting whether a future interaction between nodes u and v will occur from graph history through time t.The model outputs a link probability via a sigmoid function.
  • Temporal Node Classification (TNC): TNC predicts node u’s semantic label at a future time from the historical graph, treating labels as dynamically evolving rather than quasi-static binary outcomes.
  • Temporal Node Classification (TNC): TTGBench supports both multi-class and multi-label TNC, enabling evaluation of complex semantic dynamics.
  • The benchmark’s dataset summary reports node class counts using C.

4 The Proposed Datasets: Unveiling Dual Volatility

TTGBench contains six real-world, text-attributed temporal graph datasets designed around Dual Volatility: rapidly changing topology and complex semantic drift. Analyses show fluctuating interaction density, unstable hub concentration, frequent label changes, and heterogeneous label transitions.

  • Six real-world, text-attributed datasets span culinary feedback, movie reviews, book records, beer ratings, and online shopping interactions.
  • Dual Volatility combines Structural Volatility from novel, rapidly changing interactions with Semantic Volatility from frequent, complex label evolution.
  • Structural Volatility: Interaction volumes repeatedly expand and contract over time, requiring models to adapt across dense and sparse periods.
  • Structural Volatility: The top 5% of active nodes shift between centralized and decentralized activity regimes, making learned degree patterns quickly obsolete.
  • Semantic Volatility: Semantic drift is often high in FOOD and IMDB, with ratios above 0.8, but remains substantial and more moderate in Amazon-Kindle at around 0.3.
  • Semantic Volatility: Label-transition heatmaps contain asymmetric, high-density clusters, indicating abrupt and heterogeneous changes that defeat simple copying or static transition rules.

5 Evaluated Baselines

The benchmark evaluates 17 methods across TGNN-based and LLM-based predictor paradigms. The baselines range from structural temporal models and LLM-enhanced hybrids to in-context and supervised LLM predictors.

  • Seventeen representative algorithms are benchmarked and grouped into TGNN-Predictors and LLM-Predictors according to their prediction backbone.
  • TGNN-Predictors: TGNN-Predictors operate directly on graph structures through message passing or temporal aggregation, including pure TGNNs and LLM-enhanced hybrids.
  • TGNN-Predictors: Nine pure TGNN baselines cover memory-based, attention-based, walk-based, and hybrid mechanisms for modeling topological evolution.
  • TGNN-Predictors: LLM-as-Enhancer methods use frozen LLMs to enrich semantic representations while retaining TGNNs as the final predictors.
  • LLM-Predictors: LLM-Predictors bypass traditional message passing by converting graph information into sequential text or embedding-based prompts for direct prediction.
  • LLM-Predictors: The LLM-Predictor baselines include training-free in-context methods and supervised fine-tuning methods adapted to temporal settings.

6 Experiments and Analysis

Experiments compare temporal link prediction (TLP) and temporal node classification (TNC) across TGNN- and LLM-based methods, revealing a capability divide and distinct performance–efficiency trade-offs.

  • Main Results: TGNN-based predictors excel at TLP but degrade substantially on TNC, whereas LLM-based predictors show the opposite pattern.The divergence indicates that each paradigm captures mainly one facet of temporal information.
  • Main Results: EdgeBank performs near random (AUROC ≈50) across datasets, showing that naive memorization is ineffective under high novelty.The result supports learning evolving temporal patterns rather than relying on repeated historical interactions.
  • Main Results: Memory-based TGNNs such as TGN, JODIE, and DyRep rank among the strongest TLP methods under limited interaction repetition.Their evolving node states capture cumulative temporal effects, but require additional computational and memory overhead.
  • Main Results: Structural inputs improve LLM-Predictor TLP performance, yet even the best LLM-Predictors remain significantly behind TGNNs.SFT outperforms ICL, and structure-aware aggregation partially compensates for limited structural understanding.
  • Why Do TGNNs Fail at Semantic Tracking?: LLM-enhanced TGNNs remain weak on TNC, while TGNN performance deteriorates sharply as semantic-change frequency increases.LLM-Predictors maintain stable or improved performance under faster semantic drift, reflecting greater semantic adaptability.
  • Efficiency Analysis: GraphGPT requires nearly 600,000 seconds of training time and 13.5 teraFLOPs, while all methods scale approximately linearly from 1M to 5M interactions.Memory-based TGNNs also incur higher overhead because they continuously update recurrent node states.

7 Conclusions

TTGBench jointly evaluates structural and semantic evolution using six text-rich datasets with dual volatility and extensive experiments on 17 methods. The findings expose opposing architectural biases and motivate models that jointly capture both dynamics efficiently.

  • Conclusions: TTGBench provides a challenging setting for modeling both structural novelty and semantic dynamics across six text-rich datasets.Its experiments evaluate 17 state-of-the-art methods.
  • Conclusions: TGNN-based methods excel at structural modeling but struggle with semantic tracking, whereas LLM-based predictors show the opposite trend.The paper attributes this capability divide to intrinsic architectural biases and identifies performance–efficiency trade-offs.
  • Future Directions: Future work should develop unified models, improve LLM structural reasoning and TGNN semantic adaptability, and design efficient scalable architectures.These directions target the current gap between structural and semantic modeling and practical deployment.

A.1 Dataset Description

The appendix describes six temporal, text-rich datasets spanning reviews, recipes, books, movies, beers, and Kindle products. Their labels support temporal node classification in multi-class or multi-label settings, while LibraryThing supports only link prediction.

  • Dataset Overview: The six datasets cover FOOD, IMDB, LibraryThing, BeerAdvocate, Amazon-Kindle, and Ratebeer across diverse review-based domains.Table 4 compares these datasets with existing temporal graph datasets.
  • Dataset Descriptions: FOOD represents users and recipes as nodes with timestamped review edges and supports analysis of evolving culinary trends and user preferences.Recipe text includes cooking instructions, ingredients, and category information.
  • Dataset Descriptions: IMDB uses movie genres as users’ temporal interest labels, making it a multi-label temporal node-classification benchmark.Movies and reviews provide textual and temporal information.
  • Dataset Descriptions: LibraryThing contains timestamped user–book review interactions but lacks item category labels, so it is used only for temporal link prediction.Its interactions capture evolving literary preferences.
  • Dataset Descriptions: BeerAdvocate and Ratebeer use single beer-style labels for multi-class temporal node classification, while Amazon-Kindle uses product categories similarly.Amazon-Kindle spans user–product reviews from 1996 to September 2023.

A.2 Dataset Analysis

TTGBench contains heterogeneous user-generated edge texts and imbalanced node labels, creating challenges for extracting temporally relevant semantics and evaluating classification fairly.

  • Edge Text Lengths: Edge-text lengths are right-skewed: most interactions are brief, while a small fraction contain long-tail text.The distributions also differ across datasets.
  • Edge Text Lengths: Short texts require models to integrate information over extended temporal windows, whereas long texts contain multiple cues requiring structure-relevant extraction and temporal alignment.These demands complicate temporal graph learning across datasets.
  • Node Label Distribution: Several datasets exhibit substantial class imbalance, motivating the use of multiple imbalance-aware metrics for temporal node classification.The label distributions are summarized in Table 5.

B Methods Details

The benchmark evaluates traditional TGNNs, text-enhanced TGNNs, and LLM-as-Predictor methods through temporal graph serialization, adaptation, prompting, and task-specific training procedures.

  • TGNNs: Traditional TGNNs model temporal interactions using recurrent state updates, temporal attention, dynamic memory, causal walks, or temporal neighbor representations.The described methods include JODIE, DyRep, TGAT, TGN, CAWN, and related architectures.
  • LLM-enhanced TGNNs: Text-enhanced methods use frozen LLMs to enrich semantic representations while retaining TGNNs as the final predictors.LKD4DyTAG uses distillation, while CROSS summarizes interaction histories before semantic-structure co-encoding.
  • LLM-as-Predictor: LLM-as-Predictor methods serialize temporal graphs into LLM-readable sequences and use either supervised fine-tuning or in-context learning.ICL methods use chronological (u, v, t) triplets, whereas SFT methods use BFS-derived multi-hop neighbor sequences.
  • LLM-as-Predictor: SFT methods adapt static-graph approaches to temporal graphs using time-sensitive neighbors, training-set aggregation, and efficiency-motivated omissions.These adaptations cannot fully capture all temporal graph properties because temporal graphs are more complex than static graphs.
  • Training and evaluation: The evaluation trains TGNN-Predictors with TLP-pretrained checkpoints and uses task-specific weighted losses for binary, multi-label, and multi-class classification.TLP uses binary cross-entropy; FOOD and IMDB use weighted binary cross-entropy, while other multi-class datasets use weighted cross-entropy.

C.2 More Experiment Results and Analysis

Additional experiments examine evaluation settings, textual ablations, training strategies, and scalability across temporal link prediction and temporal node classification.

  • TLP evaluation: The supplementary experiments extend TLP evaluation across metrics, inductive settings, and historical or inductive negative-sampling strategies.Results are reported for small- and large-scale datasets under transductive and inductive protocols.
  • Text ablation: Text substantially improves TGNN performance on TLP but has limited impact on TNC, whereas LLM-Predictors benefit more from text on TNC than on TLP.This asymmetric pattern is reported in the detailed text-ablation comparisons.
  • Training strategy: End-to-end fine-tuning provides no significant TNC improvement over linear probing for TGNNs.The analysis attributes this to semantic gradients being insufficient to overcome structural bias learned during TLP pretraining.
  • Scalability: All methods show approximately linear runtime growth as edge counts increase on Amazon-Kindle, while LLM-Predictors incur higher time costs from LLM inference.The scalability evaluation uses subsamples ranging from 1M to 5M edges.
Loading 2609.08226v1…