Source-linked AI summary
Are LLM-Enhanced GNNs Privacy-Safe?
Longzhu He, Zelang Wen, Chaozhuo Li, Sen Su
TL;DR
Privacy risks in LLM-enhanced GNNs remain largely underexplored despite their utility gains and the known vulnerability of GNN outputs to inference attacks. The paper systematically evaluates these risks across datasets, models, and attack objectives, finding greater vulnerability than shallow baselines and a privacy-utility trade-off under differential privacy.
Problem
The paper addresses the limited evidence on privacy vulnerabilities in LLM-enhanced GNNs, where black-box outputs may expose sensitive information.
Method
The study evaluates six real-world text-attributed graph datasets and 42 victim model configurations under six attacks covering link, label, and membership inference, plus differential-privacy defenses.
Results
LLM-enhanced GNNs consistently show greater vulnerability than shallow baselines because semantic enrichment amplifies privacy-sensitive signals in the embedding space.
Takeaways & Limitations
Differential privacy partially mitigates attack risks but causes significant utility degradation, revealing a fundamental privacy-utility trade-off.
Takeaways & Limitations
The threat model assumes black-box access and may include auxiliary prior knowledge, while the terminology covers both LLM- and pretrained-LM-augmented GNNs.
Abstract
from arXiv · showhide
Large language models (LLMs) have recently advanced graph neural networks (GNNs) by enriching node representations with semantic information, giving rise to LLM-enhanced GNNs that achieve substantial performance gains. However, their vulnerability to privacy attacks, in which adversaries infer sensitive information from model outputs, remains largely underexplored. To bridge this gap, we present a systematic evaluation of privacy risks in LLM-enhanced GNNs through a unified framework consisting of five stages: (1) dataset preparation, (2) victim model training, (3) privacy attack, (4) risk assessment, and (5) defense analysis. Specifically, we conduct experiments on six real-world text-attributed graph datasets covering diverse domains. We consider six representative privacy attack methods targeting three fundamental threats, namely link, label, and membership inference, and construct 42 victim model configurations by combining multiple LLM-based feature enhancers with representative GNN backbones. Extensive experiments show that, despite their utility improvements, LLM-enhanced GNNs consistently exhibit increased vulnerability to privacy attacks compared to shallow text representation baselines. Further analysis reveals that semantic enrichment amplifies link-, label-, and membership-related signals in the embedding space, making them more exploitable by inference attacks. Finally, we evaluate differential privacy as a defense strategy and show that, while it can partially mitigate privacy risks, it introduces significant utility degradation, highlighting a fundamental privacy-utility trade-off in LLM-enhanced graph learning. Overall, this work provides a comprehensive understanding of privacy risks in LLM-enhanced GNNs and offers practical insights for developing more secure and trustworthy graph learning systems.
1 Introduction
LLM-enhanced GNNs enrich graph node representations with semantic information and improve performance, but their privacy risks remain underexplored. This study introduces a unified evaluation framework and systematically examines these risks across diverse models, datasets, and attacks.
- Motivation: LLM-enhanced GNNs integrate language-model-derived semantic features with graph structure to enrich node representations.They include explanation-based and embedding-based approaches alongside GNN backbones.
- Research gap: Privacy vulnerabilities in LLM-enhanced GNNs remain largely underexplored despite evidence that conventional GNN outputs can leak sensitive information.The richer semantic features may increase the information exposed through model outputs.
- Framework: The proposed evaluation framework covers dataset preparation, victim model training, privacy attack, risk assessment, and defense analysis.It enables end-to-end analysis of privacy risks and mitigation strategies.
- Experimental scope: The study evaluates six real-world text-attributed graph datasets, 42 victim model configurations, and six attack methods targeting link, label, and membership inference.The configurations combine multiple LLM-based feature enhancers with widely used GNN backbones.
- Contributions: The paper identifies privacy risks in LLM-enhanced GNNs as an important but overlooked problem and conducts a comprehensive empirical study.Its contribution spans multiple datasets, diverse attacks, and defense analysis.
2 Preliminaries
The preliminaries define text-attributed graphs and standard GNN message passing, then describe how language models enhance textual node features before graph learning.
- Text-attributed graphs: A text-attributed graph combines nodes, edges, textual attributes, and an adjacency matrix to represent structured topology with unstructured semantics.The paper focuses on node classification using both graph structure and textual node attributes.
- Graph neural networks: GNNs learn node embeddings by aggregating information from neighboring nodes and updating each representation through a learnable transformation.Neighborhood aggregation can use permutation-invariant operators such as mean, sum, or max pooling.
- Graph neural networks: At the input layer, each node representation is initialized with its original feature vector.Subsequent layers propagate and transform neighborhood information.
- LLM-enhanced GNNs: LLM-enhanced GNNs use language models to encode textual node attributes, adding semantic information that complements conventional GNN representations.These methods are categorized according to whether LLMs generate additional textual information.
- LLM-enhanced GNNs: Explanation-based methods generate high-level semantic text before LM encoding, whereas embedding-based methods directly produce or fine-tune task-specific embeddings.Embedding-based variants either use LLM embeddings directly or fine-tune conventional pretrained language models.
- Terminology: The paper uses LLM-enhanced GNNs broadly to include GNNs augmented by either large language models or pretrained language models.This terminology follows prior work.
3 Threat Model
The threat model assumes a black-box adversary that queries node posteriors and may use auxiliary knowledge. It evaluates link, label, and membership inference as representative privacy risks.
- Threat model: The adversary is characterized by its knowledge, capabilities, and objective under the paper’s privacy threat model.The model distinguishes what the adversary can access from what it aims to infer.
- Adversary access: The adversary has black-box query access to node posteriors but no access to the target model’s internal parameters or architecture.Auxiliary prior knowledge and additional capabilities may support the attack.
- Adversary objectives: The evaluation targets link, label, and membership inference, corresponding respectively to graph structure, node semantics, and training participation.These objectives are treated as representative forms of privacy leakage in graph learning.
- Link inference: Link inference predicts whether an edge exists between two nodes from their model posteriors.The target is a potentially sensitive structural relationship.
- Label inference: Label inference predicts a target node’s true class label from its model posterior.The inferred label may expose sensitive information such as private interests.
- Membership inference: Membership inference determines whether a target node was included in the model’s training set.This can reveal participation in sensitive medical or user-specific datasets.
4 Evaluation Framework
The framework evaluates privacy risks in LLM-enhanced GNNs across datasets, model configurations, attack surfaces, risk metrics, and differential-privacy defenses. It combines diverse feature enhancers and GNN backbones with attacks targeting link, label, and membership inference.
- The evaluation proceeds through dataset preparation, victim model training, privacy attack, risk assessment, and defense analysis.
- Dataset Preparation and Victim Model Training: The study combines six real-world text-attributed graph datasets with 42 victim models built from language-model feature enhancers and seven GNN backbones.The backbones include GCN, SAGE, GAT, GIN, APPNP, SGC, and SSGC.
- Privacy Attack: The attack stage covers link, label, and membership inference using six representative methods under different threat-model assumptions.Link attacks use posterior distances; label attacks include homophily guessing and node infiltration; membership attacks use shadow-model or posterior-based classifiers.
- Risk Assessment: Membership attacks are evaluated with accuracy and AUC, link attacks with AUC, and label attacks with accuracy to quantify privacy leakage.
- Defense Analysis: Differential privacy is evaluated as a model-agnostic defense by perturbing node representations or related information across embedding-, link-, and label-level privacy.The study considers several classical local differential-privacy mechanisms.
- Utility Evaluation: Node classification accuracy (%) is reported across datasets for GCN-based configurations, with the highest and second-highest results visually distinguished.
5 Evaluation
Across six real-world text-attributed graph datasets, the evaluation compares LLM-enhanced GNNs with shallow embeddings across utility, three privacy attacks, and differential-privacy defenses. LLM-enhanced models improve node-classification utility but consistently increase link, label, and membership leakage, while defenses reduce attacks at substantial utility cost.
- Evaluation Setup: Five research questions evaluate utility, membership inference, link inference, label inference, and defense effectiveness.Experiments use repeated runs with standard train/validation/test splits and averaged performance.
- Utility Evaluation: LLM-enhanced GNNs consistently outperform the Shallow baseline in node classification accuracy (%).The baseline uses BoW and Word2Vec shallow text embeddings.
- Link Inference: LLM-enhanced GNNs generally achieve higher link-attack AUC (%) than Shallow across MLA and ELA, with similar vulnerability across variants.On Tape-Arxiv23, Linq increases geometric separation between connected and unconnected pairs, making link relations more distinguishable.
- Label Inference: LLM-enhanced GNNs show stronger label leakage, measured by attack accuracy (%), than Shallow across HGA and NIA.Linq also produces a clearer gap between intra-class and inter-class embedding distances, indicating more structured label-aware representations.
- Membership Inference: LLM-enhanced GNNs consistently exhibit higher membership leakage than Shallow across MIA and NMA, although the increase varies by feature enhancer.Linq creates a larger confidence gap between member and non-member nodes on Tape-Arxiv23.
- Defense Analysis: Differential-privacy defenses partially reduce attacks but cause significant utility degradation across embedding-, link-, and label-level protection.Stronger embedding perturbation lowers attack AUC and utility; RR and k-RR likewise reduce attack performance while weakening utility or classification accuracy.
6 Related Work
Related work covers LLM-enhanced GNNs for semantic text modeling and three principal privacy threats in graph learning: membership, link, and label inference.
- LLM-Enhanced GNNs: LLM-enhanced GNNs integrate language models with GNNs to encode rich textual attributes and improve node representations and model utility.Methods are categorized as explanation-based or embedding-based approaches.
- LLM-Enhanced GNNs: Explanation-based methods generate semantic descriptions with LLMs before encoding them through a language model.
- LLM-Enhanced GNNs: Embedding-based methods directly encode textual attributes using LLMs or fine-tuned language models.
- Privacy Attacks on GNNs: Membership inference determines whether a node appeared in training by exploiting behavioral differences between member and non-member samples.
- Privacy Attacks on GNNs: Link inference recovers sensitive graph structure from node embeddings, while label inference estimates private node labels from class-discriminative information.
7 Conclusion
The paper systematically evaluates privacy risks in LLM-enhanced GNNs across multiple threats, datasets, and model configurations. It finds greater vulnerability than shallow baselines, while differential privacy reduces risk at substantial utility cost.
- LLM-enhanced GNNs are consistently more vulnerable than shallow baselines because semantic enrichment amplifies privacy-sensitive signals in embedding space.
- The evaluation spans link, label, and membership inference using six real-world text-attributed graph datasets and 42 victim model configurations.
- Differential privacy partially mitigates privacy risks but introduces significant utility degradation, revealing a privacy-utility trade-off.
- The findings provide practical insights for developing more secure and trustworthy graph learning systems.