PaperlayerView exact revision on arXiv

Source-linked AI summary

ZeroGAR: Benchmarking the Adversarial Robustness of Zero-Shot Graph Models

Zhongjian Zhang, Xiao Wang, Busheng Zhang, Bo Yan, Xingtong Yu, Yue Gao, Jia Li, Chuan Shi

arXiv:2609.33314v1cs.LGcs.AI

TL;DR

Zero-shot graph models are usually evaluated on clean graphs, while existing robustness benchmarks mainly focus on supervised settings, leaving their resilience to attacks on unseen targets unclear. ZeroGAR evaluates models across transfer settings, datasets, attack types, and budgets, and finds that clean performance does not ensure robustness, with vulnerabilities varying by model type and existing defenses not consistently helping.

  • Problem

    Existing ZGM evaluations mostly use clean graphs, and supervised robustness benchmarks provide limited evidence about robustness transfer to unseen target graphs.

    Source: Introduction · pp. 1–2

  • Method

    ZeroGAR benchmarks 13 representative ZGMs across 8 graph datasets in 4 domains, testing in-domain and cross-domain transfer under structural, textual, and node injection attacks and examining existing defenses.

    Source: Introduction · p. 2

  • Results

    GNN-based methods are more vulnerable to structural and node injection attacks, LLM-based methods to textual attacks, stronger LLMs show a structure–text robustness trade-off, and existing defenses do not consistently improve robustness.

    Source: Summary and Future Work · p. 7

  • Takeaways & Limitations

    ZeroGAR provides a standardized testbed for developing robustness methods specifically designed for zero-shot deployment on unseen graphs.

    Source: Conclusion · p. 8

Abstract

from arXiv · show

Zero-shot graph models (ZGMs), which learn transferable knowledge from source graphs and directly apply to unseen target graphs without any adaptation, have achieved promising performance and attracted considerable attention. Despite their proliferation, existing ZGMs are predominantly evaluated on clean graphs, while existing graph robustness benchmarks mainly focus on supervised settings, leaving a fundamental question largely unexplored: How robust are ZGMs when their unseen target graphs are exposed to adversarial manipulation? In this paper, we answer this question by proposing ZeroGAR, the first systematic benchmark for evaluating the adversarial robustness of ZGMs. ZeroGAR evaluates 13 representative ZGMs from 3 different paradigms on 8 graph datasets across 4 domains, covering both in-domain and cross-domain transfer under structural, textual, and node injection attacks with multiple perturbation budgets. It further investigates whether existing graph defenses remain effective in the zero-shot setting. Extensive experiments reveal that strong clean zero-shot performance does not guarantee adversarial robustness, with three key findings: (1) Vulnerability patterns are related to model prediction mechanisms: GNN-based methods are particularly vulnerable to structural and node injection attacks, whereas LLM-based methods are more vulnerable to textual attacks; (2) Stronger LLM backbones introduce a structure-text robustness trade-off; (3) Existing graph defense methods do not consistently improve zero-shot robustness and may compromise clean performance. We hope that ZeroGAR will facilitate rapid, equitable evaluation and inspire further innovative research in ZGM security.

Introduction

Zero-shot graph models transfer knowledge from source graphs to unseen targets, but their adversarial robustness remains unclear because existing evaluations focus on clean graphs and robustness benchmarks largely use supervised settings. ZeroGAR benchmarks this gap across model paradigms, datasets, transfer settings, and attack types, finding distinct vulnerability patterns and no consistent benefit from existing defenses.

Introduction

  • ZGMs transfer knowledge from source graphs to unseen targets; their three paradigms are GNN-based predictors, graph-textualizing LLMs, and graph-tokenizing LLMs.GNNPreds compare node and class representations or train classifiers; GTextLLMs express graph information in natural-language contexts, while GTokenLLMs encode it as graph tokens for LLMs.Source: Introduction · p. 1
  • Existing ZGM evaluations predominantly use clean target graphs, while robustness benchmarks mainly use supervised settings, leaving robustness transfer to unseen graphs underexplored.Zero-shot models must handle unseen target graphs without labeled data or adaptation.Source: Introduction · pp. 1–2, Table 1 · p. 2
  • ZeroGAR evaluates 13 ZGMs across 8 text-attributed graph datasets and 4 domains, covering in-domain and cross-domain transfer, three attack surfaces, multiple budgets, and existing defenses.The attack surfaces are graph structure, raw node text, and node injection.Source: Introduction · p. 2
  • GNNPreds show an 18.37% average relative accuracy drop under structural attacks, versus 3.10% for GTextLLMs and 2.45% for GTokenLLMs; textual attacks cause larger drops for LLM-based methods.Textual attacks cause drops of 21.19% for GTextLLMs and 20.27% for GTokenLLMs, compared with 8.28% for GNNPreds; stronger LLM backbones trade greater resistance to corrupted structure for sensitivity to misleading text, while defenses do not consistently help.Source: Introduction · p. 2

Preliminaries

The preliminaries define text-attributed graphs and node classification, describe zero-shot learning as training on source graphs and applying the model directly to an unseen target graph, and frame attacks as input perturbations intended to degrade predictions.

Text-Attributed Graphs

  • A text-attributed graph contains nodes, edges, and node texts; node classification predicts target-node labels from their text and graph structure.The adjacency matrix records whether each node pair has an edge.Source: Text-Attributed Graphs · p. 2

Zero-shot Graph Learning

  • Zero-shot graph learning trains a model on source graphs, then applies it directly to an unseen target graph without labeled target nodes or task-specific parameter updates.Training may be supervised or unsupervised; the model predicts target-node labels from candidate classes.Source: Zero-shot Graph Learning · p. 2

Graph Adversarial Attacks and Defenses

  • An adversarial attacker manipulates the target graph to degrade predictions, using structural or textual changes or malicious-node injection; defenses aim to improve target-node prediction performance.The permitted perturbations are those that are allowed and effective; defenses may train a robust model or purify the attacked graph.Source: Graph Adversarial Attacks and Defenses · pp. 2–3

ZeroGAR Benchmark

ZeroGAR benchmarks zero-shot graph models across diverse text-attributed datasets, transfer settings, attack surfaces, victim paradigms, and robustness metrics. Its design combines evasion attacks under grey-box knowledge with controlled evaluation settings.

Raw Data Collection

  • ZeroGAR’s benchmark covers text-attributed graphs from four domains, selected for domain diversity and variation in scale, connectivity, text, and label space.The eight datasets span academic, e-commerce, social, and web graphs.Source: Raw Data Collection · p. 3

Attack Types

  • Attacks are inference-time evasion under grey-box knowledge, targeting edges, node text, or injected nodes at low, medium, or high budgets.Structural attacks use surrogate gradients to select edge changes; textual attacks rewrite selected node texts; injection attacks add adversarial textual nodes and connections.Source: Attack Types · pp. 3–4

Victim Models

  • The benchmark evaluates GNN predictors, graph-textualizing LLMs, and graph-tokenizing LLMs using representative models from each paradigm.GNN predictors use contrastive similarity or learned binary classifiers; the two LLM paradigms respectively prompt with textual graph descriptions or projected graph tokens.Source: Victim Models · p. 4

Evaluation Settings and Metrics

  • In-domain evaluation transfers between datasets in the same domain, while cross-domain evaluation uses source graphs from different domains than the targets.The listed targets are Cora, Citeseer, History, Child, Instagram, and WikiCS.Source: Evaluation Settings and Metrics · p. 4
  • Victim models share test nodes, candidate class descriptions, and perturbation budgets, and are evaluated primarily with accuracy and relative accuracy drop.Attack success rate is also used, with its results reported in the appendix.Source: Evaluation Settings and Metrics · p. 4

Experiments

The experiments assess clean zero-shot performance, three attack surfaces, and existing graph defenses. Clean accuracy does not guarantee robustness: vulnerability patterns differ by model type, stronger LLM backbones trade structural robustness for textual sensitivity, and defenses may harm clean performance.

Performance on Clean Graphs

  • Parameter-free similarity prediction transfers better than binary classifiers within GNNPreds, while GTextLLMs generally rank above GTokenLLMs on clean graphs.GraphCLIP ranks first overall among the reported GNNPreds; the evaluated GTokenLLMs use the relatively weaker Vicuna-7B backbone.Source: Performance on Clean Graphs · pp. 4–5

Robustness against Structural Attacks

  • GNNPreds have an average structural-attack RDA of 18.37%, versus 3.10% for GTextLLMs and 2.45% for GTokenLLMs.GNNPreds aggregate perturbed neighborhood information, whereas LLM-based models typically retain clean central-node text; stronger LLM backbones show almost no average structural degradation, versus 13.06% for Vicuna-7B.Source: Robustness against Structural Attacks · p. 5, Table 4 · p. 5

Robustness against Textual Attacks

  • Textual attacks cause average relative accuracy drops of 21.19% for GTextLLMs and 20.27% for GTokenLLMs, compared with 8.28% for GNNPreds.Stronger LLM backbones are more vulnerable to textual attacks; GTokenLLMs’ text perturbations can affect both central-node text and graph tokens generated from it.Source: Robustness against Textual Attacks · pp. 5–6, Table 5 · p. 6

Robustness against Node Injection Attacks

  • Node injection yields average relative accuracy drops of 19.22% for GNNPreds, 0.69% for GTextLLMs, and 7.01% for GTokenLLMs.This pattern is closer to structural attacks because injection perturbs neighborhood information while preserving central-node text.Source: Robustness against Node Injection… · p. 6, Table 6 · p. 7

Effectiveness of Existing Graph Defenses

Summary and Future Work

  • Across attacks, all ZGMs degrade; GNNPreds are more vulnerable to structural and node injection attacks, while LLM-based models are more vulnerable to textual attacks.Stronger LLM backbones improve robustness to corrupted structural context but increase sensitivity to misleading textual semantics; existing defenses do not consistently improve robustness and may compromise clean performance.Source: Summary and Future Work · p. 7
  • The authors recommend reducing adversarial neighborhood influence for GNNPreds, verifying text against graph context for LLM-based models, and preserving clean zero-shot performance in defenses.They also identify dynamically balancing structural and textual evidence as a future direction.Source: Summary and Future Work · p. 7

Conclusion

ZeroGAR benchmarks adversarial robustness of zero-shot graph models on unseen target graphs and finds that clean performance does not ensure robustness. Its results distinguish vulnerabilities across model types and point to a standardized testbed for zero-shot robustness research.

Conclusion

  • GNN-based methods are more vulnerable to structural and node injection attacks, while LLM-based methods are more vulnerable to textual attacks.Stronger LLMs introduce a structure–text robustness trade-off, and existing graph defenses do not consistently improve zero-shot robustness and may compromise clean performance.Source: Conclusion · pp. 7–8
  • ZeroGAR is the first systematic benchmark of adversarial robustness for zero-shot graph models on unseen target graphs.It evaluates 13 representative models across diverse text-attributed graphs, transfer settings, and attack types, and examines existing defenses.Source: Conclusion · p. 7
  • The authors hope ZeroGAR will provide a standardized testbed for developing robustness methods for zero-shot deployment on unseen graphs.Source: Conclusion · p. 8

Related Work

Zero-shot graph models use different ways to provide graph information to predictors, while prior robustness evaluations have largely centered on supervised settings. This leaves their robustness on unseen target graphs without target-domain supervision insufficiently understood.

Zero-Shot Graph Models

  • ZGMs include GNN-based predictors, graph-textualizing LLMs, and graph-tokenizing LLMs, which differ in how they represent graph information for prediction.GNNPreds compare node and class representations or use binary classifiers; GTextLLMs prompt LLMs with textualized graphs, whereas GTokenLLMs pass projected graph tokens to LLMs.Source: Zero-Shot Graph Models · p. 10

Graph Robustness Benchmark

  • Existing graph robustness evaluations cover GNNs and LLM-enhanced graph learning under several attacks, but mainly assess supervised settings.Examples include GRB, LLM4RGNN, TrustGLM, and evaluations of graph-aware LLMs under poisoning and evasion attacks.Source: Graph Robustness Benchmark · p. 10
  • Prior robustness benchmarks therefore provide limited insight into whether ZGMs remain robust when transferred to unseen target graphs without target-domain supervision.Source: Graph Robustness Benchmark · p. 10

Additional Details on ZeroGAR

ZeroGAR uses publicly available text-attributed graph datasets and benchmarks 13 representative models across three model families. The materials describe dataset examples and model-specific approaches.

Datasets

  • The datasets are publicly available, with graph data, text, labels, and train, validation, and test masks stored in PyTorch .pt format.Unless otherwise specified, the datasets are released under the MIT license.Source: Datasets · p. 10
  • Dataset examples span citation, product, book, Wikipedia, and social graphs, with node-level text and labels described for the datasets.Cora nodes are research papers with title and abstract text; product nodes represent items, while social-network nodes represent users.Source: Datasets · pp. 10–11

Details of Victim Models

  • The benchmark includes four GNN-based Predictors, four Graph-Textualizing LLMs, and five Graph-Tokenizing LLMs.Source: Details of Victim Models · p. 11
  • The model descriptions distinguish approaches including contrastive node–class matching, natural-language graph inputs, and projected graph tokens processed by LLMs.Examples include G2P2 and ZeroG, textualized-input baselines such as Vicuna-7B and G1, and token-based methods such as GraphGPT, LLaGA, TEA-GLM, GOFA, and UniGTE.Source: Details of Victim Models · p. 11

Additional Details on Implementations

The implementation details specify attack configurations, model settings, and prompts used in ZeroGAR. The experiments use PyTorch Geometric and an NVIDIA A800 80GB GPU.

Hyperparameter Settings

  • The structural attack uses SGA with structure perturbation enabled and feature perturbation disabled; node injection uses WTGIA with ATDGIA and optimized injected-node features.The textual attack rewrites text using Qwen3.6-27B; injected features use zero initialization and FGSM-style binary flipping.Source: Hyperparameter Settings · p. 11
  • Model-specific settings include training schedules, architectures, and inference configurations for methods such as G2P2, ZeroG, GraphCLIP, OFA, and LLaGA.For example, G2P2 uses SBERT-384 node features, while LLaGA uses a frozen Vicuna-7B-v1.5-16K backbone and tunes only its multimodal projector.Source: Hyperparameter Settings · p. 12

Prompt Template for the Text-based Attack

Text Prompt for GTextLLM

Computing Environment and Resources

  • ZeroGAR is implemented with PyTorch Geometric, and experiments use an NVIDIA A800 80GB GPU.The reported environment also includes Ubuntu Linux 5.15.0-102-generic and an Intel Xeon Platinum 8358 CPU.Source: Computing Environment and Resources · p. 12

Additional Evaluation Results for ZeroGAR

The additional evaluation reports ASR results for structural, textual, and node injection attacks, which are consistent with the main paper. It also compares TextFooler with LLM-based textual attacks, finding limited effectiveness from word-level attacks and adopting LLM-based attacks for worst-case robustness evaluation.

Results for the ASR Metric

  • The ASR metric measures the proportion of initially correctly classified test nodes that become misclassified after an attack.It is computed over test nodes, using their ground-truth labels and clean and adversarial predictions.Source: Results for the ASR Metric · p. 12
  • ASR results for structural, textual, and node injection attacks are consistent with the observations in the main paper.Tables 7–9 report ASR across perturbation sizes; lower ASR indicates stronger robustness.Source: Results for the ASR Metric · p. 13, Table 7 · p. 14, Table 8 · p. 14

Results against TextFooler Attack

  • TextFooler is the primary word-level attack because its selected-word replacements preserve the original semantics and provide subtle lexical changes.BAE, PWWS, and HotFlip are included as additional baselines.Source: Results against TextFooler Attack · p. 13
  • Text-based attacks overfit to the surrogate model and embedding type, limiting their effectiveness, while LLM-based textual attacks generally degrade performance across backbones and text encoders.The paper adopts LLM-based textual attacks to evaluate worst-case robustness.Source: Results against TextFooler Attack · p. 13
Loading 2609.33314v1…