Source-linked AI summary
Adapting Membership Inference Attacks to GNN for Graph Classification: Approaches and Implications
Bang Wu, Xiangwen Yang, Shirui Pan, Xingliang Yuan
TL;DR
Prior GNN membership attacks examine graph components, leaving inference about entire graph records in graph classification unexplored. This paper proposes training-based and threshold-based attacks using black-box model outputs. The attacks are effective across evaluated settings, with overfitting the most significant factor and different transferability-performance trade-offs between attack types.
Problem
Prior GNN membership-inference work targets graph components, while how to infer membership of an entire graph record in graph classification remains unexplored.
Source: §I · pp. 1–2Method
The paper proposes black-box training-based attacks that classify confidence scores and threshold-based attacks that compare a confidence value with a threshold.
Source: §I · p. 2Results
Both attack types are effective; overfitting is the most significant factor, while training-based attacks transfer more strongly and threshold-based attacks perform better but transfer less well.
Source: §I · p. 2, §VI · p. 10Takeaways & Limitations
Graph-level GNN membership inference can expose training-record membership, and its attack performance is more closely associated with overfitting than with node-level GNN attack patterns.
Source: §I · p. 2
Abstract
from arXiv · showhide
Graph Neural Networks (GNNs) are widely adopted to analyse non-Euclidean data, such as chemical networks, brain networks, and social networks, modelling complex relationships and interdependency between objects. Recently, Membership Inference Attack (MIA) against GNNs raises severe privacy concerns, where training data can be leaked from trained GNN models. However, prior studies focus on inferring the membership of only the components in a graph, e.g., an individual node or edge. How to infer the membership of an entire graph record is yet to be explored. In this paper, we take the first step in MIA against GNNs for graph-level classification. Our objective is to infer whether a graph sample has been used for training a GNN model. We present and implement two types of attacks, i.e., training-based attacks and threshold-based attacks from different adversarial capabilities. We perform comprehensive experiments to evaluate our attacks in seven real-world datasets using five representative GNN models. Both our attacks are shown effective and can achieve high performance, i.e., reaching over 0.7 attack F1 scores in most cases. Furthermore, we analyse the implications behind the MIA against GNNs. Our findings confirm that GNNs can be even more vulnerable to MIA than the models with non-graph structures. And unlike the node-level classifier, MIAs on graph-level classification tasks are more co-related with the overfitting level of GNNs rather than the statistic property of their training graphs.
I. INTRODUCTION
Prior GNN membership attacks target individual graph components, leaving whole-graph membership in graph classification unexplored. This paper proposes black-box graph-level attacks and evaluates their performance, vulnerability factors, and transferability.
I. INTRODUCTION
- Earlier GNN membership attacks infer membership of graph components, such as nodes or links, rather than entire graph records.The paper identifies graph-level classification as an unexamined setting for whole-record membership inference.Source: §I · p. 1
- The study evaluates attack effectiveness across GNN methods and datasets, varying overfitting and examining training-graph statistics and node-level comparisons.The experiments also investigate transferability and the implications of graph-level versus node-level attacks.Source: §I · p. 2
- The proposed black-box attacks are the first GNN membership inference attacks for graph-level classification.They target whether an arbitrary graph record was used to train the model.Source: §I · p. 2
- GNN membership attacks can be more vulnerable than attacks against non-graph models in certain applications.Source: §I · p. 2
- Overfitting is the most significant factor for both training-based and threshold-based attacks, unlike prior node-level GNN attacks.This aligns with prior DNN MIA observations.Source: §I · p. 2
- Training-based attacks transfer strongly across GNN types and shadow datasets, while threshold-based attacks achieve higher performance but transfer less well.Source: §I · p. 2
II. RELATED WORK
Prior MIA research covers models trained on Euclidean data and GNN attacks on individual nodes, links, or subgraphs. The vulnerability of graph-level GNN classifiers to membership inference remains unexplored.
II. RELATED WORK
- MIAs have been studied across machine-learning models trained on Euclidean data, including posterior-based, white-box, generative-model, low-cost, and label-only attacks.The paper positions its work as an investigation of GNN classifiers trained on graph-structured data.Source: §II · p. 2
- Prior node-level GNN attacks infer node or link membership using target-node, neighbor, or pairwise posterior information.Some work also infers basic graph properties or subgraph membership from graph embeddings, but not membership of an entire graph.Source: §II · p. 3
- The cited prior GNN attacks do not establish how graph-level classifiers expose the membership of whole graph records.Source: §II · p. 3
III. PROBLEM FORMALISATION
This section formalizes graph classification and the membership-inference objective, then specifies the attacker's access and background-knowledge assumptions.
III. PROBLEM FORMALISATION
- The formalization section introduces the proposed MIAs and their attack assumptions.Source: §III · p. 3
A. Problem Definition
- Graph classification maps an individual graph to a categorical label in the label set Y.Source: A. Problem Definition · p. 3
- The victim is a graph classifier trained on labeled graphs, and the privacy target is membership of its training data.Source: A. Problem Definition · p. 3
- The attack aims to determine whether a specific graph record was included in the target model's training set.The attack model outputs 1 for a member record and 0 otherwise.Source: A. Problem Definition · p. 3
B. Attack Assumptions
- The attacker has black-box access to the target model, without access to its parameters or internal representations.Only queries and model outputs for chosen inputs are accessible.Source: B. Attack Assumptions · p. 3
- The attacker may use a shadow dataset from the target's domain, though the paper also considers a different-domain shadow dataset.Source: B. Attack Assumptions · p. 3
- The attacker is assumed to know the GNN type and training algorithm, with the model-type assumption later relaxed.Source: B. Attack Assumptions · p. 3
IV. THE PROPOSED ATTACKS
The paper develops black-box graph-level membership inference through training-based and non-training-based approaches. The training-based attack learns to classify records from confidence scores labeled using a shadow model.
A. Training-based Attacks
- Training-based attacks train a binary classifier to infer whether a graph record was used to train the target model.The attack uses a shadow model to generate confidence scores labeled “in” or “out” for attack-model training.Source: A. Training-based Attacks · pp. 3–4
- The attacker splits a shadow dataset into member and non-member sets, trains a shadow model on members, and gathers labeled confidence scores from both sets.Source: A. Training-based Attacks · p. 4
- When confidence-vector dimensions differ, the attack retains the top k confidence values to form a reduced vector.Source: A. Training-based Attacks · p. 4
B. Threshold-based Attacks
Threshold-based attacks infer membership without training an attack model by comparing a model-confidence measure with a threshold. The paper considers confidence scores, losses, and distances as measures.
B. Threshold-based Attacks
- A threshold-based attack compares a confidence value with a threshold to infer membership, avoiding attack-model training when resources are limited.Source: B. Threshold-based Attacks · pp. 4–5
- The attack measures confidence using the highest prediction score, loss, or distance from a one-hot output vector.The evaluated losses include MSE and cross-entropy; the distance measures include Cityblock and Canberra.Source: B. Threshold-based Attacks · pp. 4–5
- Higher thresholds favor precision, while lower thresholds favor recall.Source: B. Threshold-based Attacks · p. 5
V. EVALUATION AND ANALYSIS
The evaluation asks whether GNN types are vulnerable to membership inference, how attack performance transfers with less attacker knowledge, and which factors affect it. It compares model overfitting with graph properties and other graph-based attacks.
V. EVALUATION AND ANALYSIS
- The evaluation examines whether and how different GNN types are vulnerable to membership inference attacks.Source: §V · p. 5
- It tests attack transferability and how performance changes when attackers have less knowledge of the target model.Source: §V · p. 5
- The evaluation varies parameters to assess attack factors, including whether overfitting is the only factor affecting performance.Source: §V · p. 5
- It compares the influence of target-model performance and training-dataset graph statistics, and contrasts graph-level attacks with other graph-based attacks.Source: §V · p. 5
A. Experimental Setup
The experiments evaluate attacks across seven real-world datasets and multiple GNN architectures, using a worst-case non-overlapping shadow dataset setup. Results report strong training-based attacks in some settings, while threshold-based performance depends on the chosen metric and threshold.
A. Experimental Setup
- The evaluation uses seven real-world datasets, five representative GNN types, an MLP baseline, and 15 experimental runs.The target and shadow datasets are equally split with no overlap, a stated worst-case attacker scenario; Table II summarizes dataset statistics.Source: A. Experimental Setup · p. 5
1) Performance Overview:
- More than 0.85 F1 Score is reported for training-based attacks on DD and CIFAR10, and attacks exceed 65% precision on most datasets.The evaluation compares attack performance across datasets and GNN methods; Table III also reports target-model accuracy.Source: 1) Performance Overview · p. 6, Table III · p. 6
- GateGCN and GCN are reported as more vulnerable than GIN, while distilled 10-dimensional confidence vectors achieve higher attack rates than other vector sizes.Table V compares attack-model F1 scores for different input vectors.Source: 1) Performance Overview · p. 6, Table V · p. 7
- Threshold-based attacks are effective only at some matrix thresholds, and cross-entropy is the only confidence metric reported to achieve satisfying performance.Table IV compares threshold-based attack F1 scores across models and six datasets.Source: 1) Performance Overview · p. 6, Table IV · p. 7
2) Comparison to Non-graph-structure Model:
The paper compares graph-level GNN attacks with an MLP baseline and examines transfer across model types and shadow datasets. Training-based attacks transfer more consistently, while threshold-based attacks depend more on domain knowledge.
2) Comparison to Non-graph-structure Model:
- On DD, GateGCN attacks reach 0.885 F1 versus 0.552 F1 for MLP, and GNNs outperform MLP on most datasets.All MLP models are more robust than GateGCN and GCN models in the reported comparison.Source: 2) Comparison to Non-graph-structure… · p. 7
3) Transferability:
- Most training-based attacks transfer across GNN methods with F1 reductions within 0.1.The comparison tests target and shadow models using different GNN methods.Source: 3) Transferability · p. 8
- Training-based attacks remain effective across shadow datasets, indicating that shadow-dataset knowledge affects them less.The experiments include cross-domain shadow datasets.Source: 3) Transferability · p. 8
- Threshold-based attacks transfer poorly across domains because optimal thresholds vary by domain and require domain knowledge.The attackers select thresholds using threshold distributions obtained by querying target models with records from the same or a similar domain.Source: 3) Transferability · p. 8
4) Comparison between Two Proposed Attacks:
Threshold-based attacks generally perform better but transfer less well than training-based attacks. Across both attack types, overfitting is associated with stronger attacks; deeper architectures show slightly greater robustness.
4) Comparison between Two Proposed Attacks:
- Threshold-based attacks achieve the highest performance in most settings but have poorer transferability than training-based attacks.They also require fewer computation resources because attack models do not need to be trained, though threshold selection can significantly affect performance.Source: 4) Comparison between Two Proposed… · p. 8
1) Effect of the Target Model Property:
- Higher target-model overfitting is associated with stronger training-based attacks: increasing epochs raises both train-test gaps and attack F1 scores.The train-test gap is used to measure overfitting.Source: 1) Effect of the Target Model Property · p. 8
- Threshold-based attacks are also positively correlated with overfitting and are more sensitive to its fluctuation than training-based attacks.The comparison varies training epochs and tracks the target model's train-test gap and attack performance.Source: 1) Effect of the Target Model Property · p. 8
- Deeper DeepGCN models have lower train-test gaps and higher attack F1 scores, yet are described as slightly more robust.The authors relate higher training loss in deeper models to lower confidence on member data, which reduces attack performance.Source: 1) Effect of the Target Model Property · p. 9
2) Effect of the Target Graph Data Property:
The reported findings connect graph-classification vulnerability to class count and average graph degree, while also identifying model overfitting as strongly related to attack effectiveness.
2) Effect of the Target Graph Data Property:
- Graph-classification GNNs with more classes or training graphs with larger average degrees are reported as more vulnerable to MIAs.The paper says higher-dimensional confidence vectors provide more information, while higher-degree graphs may increase models’ confidence in member records.Source: 2) Effect of the Target Graph Data… · pp. 9–10
- Overfitting is highly related to graph-level attack effectiveness, while training-graph statistics are reported not to affect attacks.The results report Spearman correlations for graph-level classification on PROTEIN full and node-level classification on CORA.Source: 2) Effect of the Target Graph Data… · p. 9
D. Comparison to MIAs on Node-level GNNs
Graph-level MIAs are more correlated with model properties, especially overfitting, whereas node-level MIAs depend more on graph data properties. The paper also identifies defenses and label-only attacks as future directions.
D. Comparison to MIAs on Node-level GNNs
- Graph-level MIAs correlate more with model properties, while node-level MIAs depend more on graph-data properties.The paper links this difference to graph-level attacks using the entire graph’s final posterior and node-level attacks exploiting correlations with neighboring nodes.Source: D. Comparison to MIAs on Node-level GNNs · p. 10
- Graph-level attack effectiveness relies on target-model overfitting, unlike node-level attacks that use posterior information from neighboring nodes.The comparison describes node-level attacks as exploiting correlations between target nodes and their neighbors.Source: D. Comparison to MIAs on Node-level GNNs · p. 10
VI. CONCLUSION AND FUTURE DIRECTIONS
- Defenses against GNN MIAs and attacks using label-only exposures remain future work.The authors identify both as future directions.Source: §VI · p. 10