PaperlayerView exact revision on arXiv

Source-linked AI summary

TemporalGraphLLM: Temporal Graph Neural Networks with Large Language Models for Dynamic Text-Attributed Graphs

Moran Beladev, Or Eitan, Gilad Katz, Lior Rokach

arXiv:2609.31881v1cs.LG

TL;DR

Dynamic text-attributed graphs call for models that handle evolving relationships and text together, but TGNNs and LLMs each have complementary limitations. TemporalGraphLLM injects time-aware TGNN embeddings into an LLM and jointly trains the components; it reports consistent gains across graph tasks, with smaller improvements for text generation.

  • Problem

    Dynamic text-attributed graphs evolve in both structure and textual attributes, while TGNNs struggle with textual reasoning and LLMs with structured graph reasoning.

    Source: §1 · p. 1

  • Method

    TemporalGraphLLM pairs a TGNN with an LLM by injecting projected, time-aware TGNN embeddings into the LLM stream and jointly training the components with temporal instruction tuning.

    Source: §1 · p. 2

  • Results

    Across datasets, TGNN backbones, and LLM sizes, TemporalGraphLLM consistently outperforms TGNN-only and LLM-only baselines on link prediction and edge classification, with smaller but consistent improvements in edge text generation.

    Source: §4.5 · p. 8

  • Takeaways & Limitations

    The framework combines structural and textual information for dynamic text-attributed graph tasks and is reported to outperform GNN-only and LLM-only baselines across diverse graph domains.

    Source: §5 · p. 8

    The evaluation focuses on open-weight LLMs; API-only models are excluded because they cannot be matched under the same training protocol and are comparable only in zero-shot mode.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Dynamic text-attributed graphs (DTAGs), where nodes, edges, and textual attributes evolve over time, are crucial in applications such as social networks, citation graphs, and knowledge graphs. However, existing approaches struggle to jointly model the temporal evolution of graph structures and the semantic richness of textual attributes. While Temporal Graph Neural Networks (TGNNs) capture evolving node relationships, they often lack contextual text reasoning. Conversely, Large Language Models (LLMs) excel in textual understanding but struggle with structured graph reasoning in temporal settings. To bridge this gap, we propose TemporalGraphLLM, a novel framework that can integrate any temporal GNN with an LLM for enhanced reasoning in DTAGs. Our approach fine-tunes LLMs using graph-time-aware instruction tuning and novel temporal GNNs injection to replace dedicated added tokens with graph embeddings. TemporalGraphLLM effectively leverages pretrained TGNNs within an LLM framework to achieve state-of-the-art performance on edge classification, link prediction, and edge-based text generation tasks. Extensive evaluation on real-world dynamic graph datasets demonstrates state-of-the-art performance. Our findings highlight the synergistic potential of LLMs and TGNNs, opening new directions for learning on evolving graphs.

1. Introduction

Dynamic text-attributed graphs require reasoning across evolving structure, text, and time, but TGNNs and LLMs each leave part of that challenge unmet. TemporalGraphLLM combines them through temporal embedding injection and joint training, and is reported to outperform existing approaches across several tasks and graph domains.

1. Introduction

  • Time-sensitive, context-aware text generation requires joint reasoning over structural, textual, and temporal signals as graph connections and attributes change.The introduction illustrates this with user reviews in a user-product graph.Source: §1 · p. 1
  • TGNNs capture evolving interactions but struggle with textual reasoning, while LLMs understand semantics but lack effective structured-graph reasoning.The introduction identifies integrating temporal node representations with LLM reasoning as a remaining gap for evolving graphs.Source: §1 · pp. 1–2
  • The framework pairs any TGNN with any LLM by replacing placeholder-token representations with time-aware TGNN embeddings through learned projections.It also supports end-to-end joint training with temporally aware instruction tuning and a multi-objective loss coupling structural and temporal learning with text generation.Source: §1 · p. 2
  • TemporalGraphLLM is reported to significantly outperform existing approaches on temporal edge classification, link prediction, and edge text generation across diverse dynamic graph domains.The introduction names email networks, social graphs, e-commerce graphs, and evolving knowledge graphs.Source: §1 · p. 2

2. Related Work

Prior LLM–GNN work spans enhancement, prediction, and embedding alignment, while temporal graph research includes methods that leave either LLM reasoning or time underused. TemporalGraphLLM’s stated distinction is joint TGNN–LLM training with time-aware embeddings injected into the LLM.

2. Related Work

  • Prior LLM–GNN integration is organized into enhancement, prediction, and alignment approaches for combining textual and structural cues.Enhancement enriches GNN representations with text; prediction uses LLMs for graph tasks; alignment addresses consistency between graph and language representations.Source: §2 · p. 2
  • Temporal graph approaches include LLM distillation into frozen TGNNs, temporal-rule extraction, forecasting, and temporal knowledge-graph question answering.The paper contrasts these approaches with its co-training of a temporal GNN and LLM, injecting time-sensitive node embeddings directly into the LLM.Source: §2 · p. 2
  • Temporal GNN baselines capture temporal patterns without LLM reasoning, while DTGB reports that most text-attributed graph methods ignore time and naive LLM-embedding appends offer limited benefit.Source: §2 · p. 2
  • TemporalGraphLLM jointly trains a TGNN and LLM, replacing LLM placeholder representations with projected time-aware node embeddings under a multi-objective loss.The stated objective unifies structural, temporal, and textual supervision.Source: §2 · p. 3

3. The Proposed Approach: TemporalGraphLLM

TemporalGraphLLM combines time-aware node representations with LLM text context for edge classification, link prediction, and edge text generation. It injects projected TGNN embeddings into prompts and jointly fine-tunes the models, using temporally sampled neighbors and task-appropriate supervision.

3.1. Dynamic Text-Attributed Graphs – Notations

  • The notation defines a temporal subgraph through timestamp T, with textual descriptions for nodes and textual relations, category labels, and timestamps for edges.Source: §3.1 · p. 3

3.2. TemporalGraphLLM

  • TemporalGraphLLM integrates a temporal GNN with an LLM, combining graph and text context for edge classification, link prediction, and edge text generation.The framework supports any temporal GNN backbone and jointly fine-tunes the TGNN and LLM.Source: §3.2 · p. 3

3.2.1. TEMPORAL GNN EMBEDDING

  • A temporal GNN forms time-aware node embeddings from each queried node’s interaction history, using time encodings and stacked layers to capture temporal dynamics and multi-hop structure.The framework can use TGAT, GraphMixer, or DyGFormer, while remaining agnostic to the backbone choice.Source: §3.2.1 · p. 3

3.2.2. PROMPT ENCODING

  • Projected node embeddings replace prompt placeholders alongside node and edge text, while recency-weighted sampling adds temporal neighbors to the prompt.The sampling comparison reports consistently higher performance for recency-aware than random sampling across TGNN backbones.Source: §3.2.2 · pp. 4–5

3.2.3. EDGE PREDICTION VIA LLM-GNN FINE-TUNING

  • Edge prediction fuses LLM context with TGNN embeddings through an MLP, while next-token prediction trains the decoder to output labels or edge text.Text generation uses only the language-modeling loss, with λ1 = 0 and λ2 = 1.Source: §3.2.3 · p. 5

3.2.4. NEGATIVE SAMPLING FOR THE LINK PREDICTION TASK

  • For link prediction, each observed edge is paired with an unobserved candidate node from the same temporal window, yielding a plausible negative absent from the interaction history.The candidate must be a node the source has not interacted with before the positive edge’s timestamp.Source: §3.2.4 · pp. 5–6

4. Experimentation

The experiments evaluate TemporalGraphLLM on dynamic graph datasets across three prediction and generation tasks, comparing TGNN, LLM, and fused models. Fusion outperforms both baseline types on link prediction and edge classification; ablations examine temporal-neighbor count and loss balance.

4.1. Datasets; 4.3. Baselines; 4.4. Metrics

  • The study uses datasets of entity interactions and compares TGNNs, instruction-tuned LLMs, and their fusion, with open-weight LLMs selected for controlled fine-tuning and reproducibility.TGNN baselines use the same fixed Llama-3-1B node-text embeddings; link prediction, classification, and generation use structural and text-similarity metrics.Source: §4.1 · p. 6, §4.3 · p. 7, §4.4 · p. 7

4.2. Tasks

  • The evaluation covers future link prediction, future edge-category classification, and generation of future interaction text from prior interactions.Link prediction uses an inductive setup that can include unseen nodes; classification predicts a future interaction category.Source: §4.2 · p. 6

4.5. Results

  • TemporalGraphLLM consistently outperforms TGNN-only and LLM-only baselines on link prediction and edge classification, while generation gains are smaller but consistent.The authors attribute the smaller generation gains to language modeling and the grounding already provided by temporal prompts; TGNN states add long-range temporal structure.Source: §4.5 · p. 8
  • Paired Wilcoxon tests find the best TemporalGraphLLM variant significantly outperforms all baselines on link prediction and edge classification (p < 0.05).Kruskal–Wallis tests find significant method differences on both tasks (p < 0.01).Source: §4.5 · p. 8

4.6. Analysis and Ablation Study

  • Performance improves up to k = 10 temporal neighbors and then saturates; using none causes a large drop, so k = 10 is the default.The authors identify temporal-neighborhood context as critical.Source: §4.6 · p. 8
  • Performance is stable across tested loss weights, with a slight advantage for structural weighting (λ1, λ2) = (2, 1); structure-only and text-only variants underperform.The tested settings vary the balance between structural and next-token supervision.Source: §4.6 · p. 8

5. Conclusions and Future Work

TemporalGraphLLM integrates TGNNs and LLMs for dynamic text-attributed graph tasks, injecting time-aware node embeddings into the LLM token stream. The paper reports consistent improvements over GNN-only and LLM-only baselines and proposes extensions involving joint generation and reinforcement learning.

5. Conclusions and Future Work

  • The framework injects temporally aware node embeddings into the LLM token stream to fuse structural and textual information.Across diverse graph domains, it consistently outperforms GNN-only and LLM-only baselines.Source: §5 · p. 8
  • Future work proposes jointly predicting edge descriptions and labels, with reasoning sequences before classification to target interpretability and accuracy.The proposal aims to capture temporal and relational context prior to classification.Source: §5 · p. 9
  • The authors also propose reinforcement learning to optimize temporal-path selection using rewards for correct predictions and generated edge descriptions.They describe this as a route toward more robust, context-aware graph reasoning.Source: §5 · p. 9

A. Prompt Templates

The paper uses task-specific prompt templates for edge-category classification, link prediction, and edge-text generation. They combine node descriptions, recent temporal interactions, and task instructions.

A. Prompt Templates

B. Datasets Details

The dataset collection spans email, political-event, Stack Exchange, business-review, and movie-review interactions. Their nodes, edge text, labels, and temporal ordering reflect different domains.

B. Datasets Details

  • Enron represents employees and timestamp-ordered emails, with employee-role text, raw email text, and 10 edge categories.Source: B. Datasets Details · p. 11
  • GDELT4 tracks timestamp-ordered political events between entities, with edge text describing relationships.Source: B. Datasets Details · p. 11
  • Stack elec. contains timestamp-ordered answers and comments linking users to questions, categorized by usefulness.Node text includes user self-introductions and question titles and bodies; edges carry raw answer or comment text.Source: B. Datasets Details · p. 11
  • Googlemap CT contains timestamp-ordered user reviews of Connecticut businesses, with business text and ratings from 1 to 5.Source: B. Datasets Details · p. 11
  • Amazon movies contains timestamp-ordered user reviews of movie and TV products, with ratings from 1 to 5.Source: B. Datasets Details · p. 11

C. Experimentation Setup

Experiments use chronological data splits and a shared evaluation setup, with parameter-efficient LLM fine-tuning and specified TGNN configurations.

C. Experimentation Setup

  • Datasets are split chronologically into train, validation, and test sets at a 7:1.5:1.5 ratio, with identical splits used across models.Experiments run on an NVIDIA A40 GPU with 48GB memory; the LLM context length is 2,048 tokens.Source: C. Experimentation Setup · p. 11
  • LLMs are fine-tuned for one epoch using LoRA with rank 16 and scaling factor 16, 4-bit quantization, and early stopping on held-out validation data.The setup uses batch size 4, gradient accumulation over 16 steps, and a learning rate of 5 × 10−5.Source: C. Experimentation Setup · p. 11
  • TGNN backbones use two layers, 768-dimensional node embeddings, 100-dimensional time encodings, dropout 0.1, and 20 neighbors.For link prediction and edge classification, λ1=λ2=1; for edge text generation, λ1=0 and λ2=1.Source: C. Experimentation Setup · p. 11

D. Computational Efficiency: Wall-Clock Training Time (Google)

The Google-dataset comparison measures end-to-end training time under identical settings for the LLM-only and TGNN-integrated models. TGNN integration adds minor overhead for smaller LLMs but reduces training time for larger backbones.

D. Computational Efficiency: Wall-Clock Training Time (Google)

  • TGNN integration adds minor overhead for smaller LLMs but reduces total training time for larger backbones.The reported pattern is consistent with faster convergence and more stable optimization.Source: Table 5 · p. 13
  • The LLM-only and TGNN×LLM models use identical batch size, sequence length, and optimization hyperparameters, so differences reflect integration and convergence behavior.Training time is measured from the first forward pass through the final checkpoint.Source: D. Computational Efficiency:… · p. 13

E. Qualitative Examples of Generated Edge Descriptions

Generated edge descriptions adapt to their dataset domains, ranging from technical discussions and event labels to reviews and corporate communications.

Enron (Corporate Communication)

  • Enron examples use formal internal-business communication, including meeting summaries and memo notices.Across the examples, generated descriptions adapt to different domains with relevant, coherent, context-sensitive text.Source: E. Qualitative Examples of Generated… · pp. 13–14

StackOverflow (Technical Discussion)

  • For StackOverflow, the generated technical discussion uses conversational phrasing while comparing resistor choices.The example is described as aligned with typical StackExchange interactions.Source: E. Qualitative Examples of Generated… · p. 14

GDELT (Event Classification)

Amazon Movies (User Review) / Google Maps (Review Text)

F. Ablation Study

The ablations show gains from recency-aware neighbor sampling, a moderate temporal-neighbor context, and joint structural and textual supervision.

Effect of the Number of Temporal Neighbors (k)

Effect of Loss Weighting (λ1, λ2)

  • 0.6211 on Enron and 0.6172 on Googlemap CT: DyGFormer+Llama3-1B reaches the highest F1(W) scores with structural/textual loss weights (2, 1).Purely textual and structure-only variants underperform; equal weighting (1, 1) provides a strong baseline.Source: F. Ablation Study · p. 14, Table 8 · p. 15

Temporal vs. Random Neighbor Sampling

  • 0.5660 versus 0.5494 weighted F1 and 0.4239 versus 0.2391 macro F1: temporal sampling outperforms random sampling for DyGFormer+Llama3-1B on Enron.Temporal sampling consistently improves performance across all TGNN backbones at k=5.Source: F. Ablation Study · p. 14, Table 6 · p. 15

Related papers

Loading 2609.31881v1…