Source-linked AI summary
TH-GNN: Heterogeneous Temporal Graph Neural Networks for LLM-Agent Shilling Attack Detection
Shivam Swarup, Divya Prakash Shrivastava, Rakesh Thakur
TL;DR
LLM-generated shilling profiles combine realistic text, plausible ratings, and coordinated timing that expose blind spots in text-only and graph-only detectors. TH-GNN jointly models heterogeneous graph structure, multimodal text, and temporal burstiness. Across five attack families and four datasets, it achieves a 0.870 grand-mean F1 and a +10.9 pp Agent4SR margin over the strongest text-only baseline, widening to +11.5 pp at the lowest injection rate.
Problem
Existing text-only and graph-only detectors miss complementary semantic, structural, and temporal signals in realistic LLM-agent shilling attacks.
Method
TH-GNN uses a heterogeneous temporal graph transformer with cross-modal review and item text fusion and a GRU over log inter-arrival times.
Results
0.870 grand-mean F1 across five attack families and four datasets, with a +10.9 pp Agent4SR margin over SemanticShield and +11.5 pp at the lowest injection rate.
Takeaways & Limitations
Jointly modeling temporal, structural, and semantic signals improves detection of sophisticated LLM-driven shilling attacks.
Takeaways & Limitations
The temporal stream requires per-event timestamps, and sustaining accuracy against evolving LLM-generated profiles requires periodic retraining on fresh adversarial samples.
Abstract
from arXiv · showhide
LLM agents can now generate realistic shilling profiles, fluent reviews, and coherent ratings at scale, systematically defeating recommender-system defenses. Text-only detectors that flag semantic drift in review embeddings are blind to graph structure and temporal coordination, while graph-only detectors that exploit neighborhood anomalies cannot reason over review semantics or the cross-modal inconsistencies produced by LLM-generated content. We propose TH-GNN, a heterogeneous temporal graph neural network with a two-layer Heterogeneous Graph Transformer backbone that applies per-type and per-relation attention augmented with learnable sinusoidal temporal encodings on every edge. Cross-modal attention fuses structural user embeddings with frozen RoBERTa representations of reviews and item descriptions, while a GRU operating over log inter-arrival times captures temporal burstiness. Evaluated across five attack families and four benchmark datasets, TH-GNN achieves a grand-mean F1 score of 0.870, outperforming the strongest text-only baseline on Agent4SR attacks by 10.9 percentage points and 11.5 percentage points at the lowest injection rate. These results demonstrate the effectiveness of jointly modeling temporal, structural, and semantic signals for detecting sophisticated LLM-driven shilling attacks.
1 Introduction
LLM agents create coordinated fake profiles that evade any single rating, text, or graph anomaly signal. TH-GNN jointly models heterogeneous structure, review semantics, and temporal burstiness, achieving broad gains over prior detectors.
- LLM agents generate fluent, coherent profiles with persona-consistent ratings, realistic filler items, and reviews embedded within the genuine-user manifold.
- No individual rating, textual, or neighborhood anomaly signal reliably flags coordinated LLM-generated profiles.
- TH-GNN combines heterogeneous graph attention, learnable temporal edge encodings, cross-modal RoBERTa attention, and a burstiness GRU.
- 0.870 grand-mean F1 across 20 dataset–attack configurations, with TH-GNN above every baseline on every row.
- +10.9 pp against SemanticShield on Agent4SR and +11.5 pp at the 0.5% injection rate.
- The evaluation spans five attack families and four diverse datasets, with margins widening at low injection rates.
2 Related Work
Prior shilling and deceptive-content detectors rely mainly on handcrafted statistics, isolated-user sequences, graph structure, or text embeddings. The surveyed gaps motivate TH-GNN’s joint heterogeneous, multimodal, and temporal design and its evaluation against LLM-agent attacks.
- Statistical and supervised detectors capture coarse injection fingerprints but can overfit known attack signatures and degrade on unseen profiles.
- Sequence-based deep models recover temporal rating patterns but treat users independently and miss collective relational anomalies.
- Graph detectors improve structural-attack detection but Anti-FakeU lacks review-text and edge-timestamp representations.
- Textual deception detectors exploit linguistic fingerprints, but neural generators can violate the distributional assumptions behind those classifiers.
- SemanticShield compares aggregated review embeddings with the genuine-user centroid but degrades without review text and ignores graph and temporal coordination.
- Systematic evaluation of detection strategies against agent-generated profiles across datasets and injection rates remains absent from prior literature.
- TH-GNN closes the three detector gaps through heterogeneous graph encoding, cross-modal fusion, and temporal coordination modeling.
3 Methodology
TH-GNN formulates shilling detection on a heterogeneous temporal user–item–review graph and combines structural, textual, and temporal signals. Its architecture uses typed temporal graph attention, cross-modal text fusion, burstiness modeling, and focal-loss training for sparse attacks.
- Problem formulation: Users, items, and reviews form typed nodes connected by writes, about, and rates edges, with each review event’s edges sharing a timestamp.Users receive binary fake-profile labels, and the detector outputs per-user shilling scores and thresholded predictions.
- Graph construction: The unified data pipeline applies 5-core filtering and chronological 70 / 10 / 20 % train/validation/test windows shared across detectors and attacks.User and item features include interaction statistics, while dataset-specific catalogue or demographic features are appended when available.
- HGT encoder: A two-layer HGT maps node types to a shared space and learns type- and relation-specific attention and message transformations over incoming heterogeneous edges.The structural encoder produces per-user embeddings, with temporal bias injected directly into attention logits without changing node feature dimensions.
- Temporal encoding: Learnable sinusoidal encodings represent elapsed edge time, using normalized day-scale deltas and parameters initialized so the temporal term begins at zero.The encoding is added as a scalar bias to each attention logit and is learned with frequencies, phases, a projection vector, and bias.
- Multimodal fusion: Cross-modal attention fuses structural user embeddings with frozen RoBERTa review and item representations, while a GRU encodes log inter-review gaps as burstiness features.The fused representation and burstiness vector are concatenated and passed to a two-layer MLP classifier; review nodes use scalar placeholders because semantic content is encoded separately.
- Training objective: At a moderately confident prediction of p_t=0.5, focal loss reduces easy-negative contribution by 4× relative to cross-entropy, concentrating gradients on rare or ambiguous fake profiles.The model uses γ=2.0 and α=0.25, is trained end-to-end with Adam, and freezes RoBERTa weights throughout.
4 Experiments
Experiments evaluate TH-GNN across diverse datasets, attack families, baselines, and downstream recommendation outcomes. TH-GNN consistently outperforms baselines, with especially large advantages for LLM-agent attacks and low injection rates.
- Datasets: Four benchmarks span rating-only and text-rich settings, sparse and dense graphs, and short and long temporal extents.All datasets use 5-core filtering and timestamp-based 70/10/20 splits shared across detectors and attacks.
- Attack setup: Five attacks cover random, bandwagon, GAN-based AUSH, GNN-targeted GraphAttack, and LLM-agent Agent4SR profiles.Injection rates are swept over 0.5%, 1%, and 5%, with each fake profile targeting five popular items under a 10% catalogue budget.
- Main results: TH-GNN achieves a grand-mean F1 of 0.870 across 20 dataset–attack configurations and remains below 4.5% FAR throughout.Table 2 reports F1, detection rate, and false-alarm rate averaged over three seeds and three injection rates.
- Main results: On Agent4SR, TH-GNN reaches mean F1 = 0.825 versus SemanticShield’s 0.716, a margin of +10.9 pp.The advantage reflects detection of temporal coordination despite fluent reviews and plausible rating histories.
- Downstream recommendation quality: TH-GNN filtering restores ML-1M GraphAttack NDCG@10 from 0.170 without defense to 0.219, or 97.8% of clean performance.The method ranks first in both NDCG@10 and HR@10 across all 20 dataset–attack conditions.
- Low-budget regime: At 0.5% injection, Agent4SR F1 is 0.806 for TH-GNN versus 0.691 for SemanticShield, a margin of +11.5 pp.The cited setting contains only five fake profiles per 1,000 real users; graph-temporal signals become more salient as injected mass shrinks.
5 Conclusion
TH-GNN improves shilling detection by jointly modeling heterogeneous graph structure, multimodal text, and temporal burstiness, addressing distinct failure modes of prior detectors. Across five attack families and four datasets, it achieves strong performance, with margins widening at the lowest injection rate; deployment requires timestamps and ongoing retraining as generated profiles evolve.
- TH-GNN jointly models heterogeneous graphs, multimodal text, and temporal burstiness to address orthogonal failure modes of prior detectors.Its streams include a two-layer HGT with learnable sinusoidal temporal encoding, cross-modal attention over review and item text, and a GRU over log inter-arrival times.
- +10.9 pp is TH-GNN’s margin over the strongest text-only baseline on LLM-agent attacks across five attack families and four datasets.The grand-mean F1 is 0.870, and the margin widens to +11.5 pp at the lowest injection rate.
- +11.5 pp is the margin at the lowest injection rate, precisely where single-modality detectors are most vulnerable.
- TH-GNN requires per-event timestamps; without them, platforms reduce the model to text-and-graph detection, with an estimated −5.1 pp cost.Periodic retraining on fresh adversarial samples is also necessary as LLM-generated profiles evolve.