Graph Neural Networks for Vulnerability Detection: A Counterfactual Explanation

1 Introduction

• Vulnerability detection aims to automatically identify potentially vulnerable code and support software security and reliability.The introduction notes that vulnerabilities threaten data integrity, user privacy, and cybersecurity.Source: §1 · p. 1
• Factual explainers identify features or subgraphs that contribute to a prediction, but do not answer how changing code structure would affect the detector’s decision.The resulting uncertainty can leave developers checking variables manually to find the vulnerability’s actual cause.Source: §1 · pp. 2–3
• In the FFmpeg example, removing data-flow edges changes the prediction, and the minimal change identifies the chunksize flow as the root cause of a buffer overflow.This points developers to inspect the chunksize value for potential errors.Source: §1 · pp. 2–3
• CFExplainer learns a differentiable edge mask to find a minimal code-graph perturbation that flips a vulnerable prediction to non-vulnerable.The authors describe this as the first explainer to introduce counterfactual reasoning for GNN-based vulnerability detection.Source: §1 · p. 3
• Experiments on four GNN-based vulnerability detection systems show CFExplainer outperforms state-of-the-art factual reasoning-based explainers.The paper evaluates CFExplainer for explaining GNN decisions and helping developers discover vulnerability causes.Source: §1 · p. 3

2.1 GNN-Based Vulnerability Detection Model

• GNNs model code graphs through repeated neighbor aggregation and node updates, then pool node representations into a graph representation for classification.The classifier produces probabilities for vulnerable and non-vulnerable classes.Source: §2.1 · p. 3
• GNN training minimizes binary cross-entropy, and testing assigns the class with the highest predicted probability.The prediction is calculated from the code graph representation.Source: §2.1 · p. 4
• The study investigates GCN, GGNN, GIN, and GraphConv, which use different aggregation and update functions to capture structural code information.Their described mechanisms include gated information flow, graph-isomorphism updates, and higher-order structures at multiple scales.Source: §2.1 · p. 4

2.2 Model Explainability: The Problem

• The explainability problem assumes a trained GNN in a black-box setting, where the explainer can query predictions but cannot access model parameters, training data, or layer gradients.Source: §2.2 · p. 4
• Factual reasoning explains a prediction by identifying key input features, such as a compact subgraph that preserves the original predicted label.The cited example optimizes the probability of predicting that original label from the subgraph.Source: §2.2 · p. 4
• Counterfactual reasoning seeks a minimally perturbed code graph classified differently from the original, with graph distance able to count removed edges.Source: §2.2 · p. 4

3 Proposed CFExplainer

• CFExplainer combines code-graph perturbation, counterfactual reasoning, and explanation generation to explain GNN vulnerability predictions.Its stated goal is to identify a minimal graph perturbation that flips the prediction.Source: §3 · p. 4, §3.2 · p. 5

3.1 Code Graph Perturbation

• Because vulnerability can involve structural relations, CFExplainer perturbs graph edges and learns a differentiable mask rather than using costly greedy search.The binary mask is relaxed to continuous values so the perturbed adjacency matrix can be optimized by gradient descent.Source: §3.1 · pp. 4–5

3.2 Counterfactual Reasoning Framework

• The reasoning framework optimizes prediction and distance losses to flip the model’s output while keeping the perturbed graph close to the original.The trade-off parameter α can prioritize changing the prediction or minimizing perturbation; the framework updates the mask while holding the GNN parameters fixed.Source: §3.2 · p. 5

3.3 Counterfactual Explanation Generation

• After optimization, CFExplainer removes edges with the lowest mask values to form a counterfactual subgraph whose removal reverses the vulnerability prediction.It can also search combinations within that subgraph to produce diverse explanations and actionable options.Source: §3.3 · p. 6

4 Experimental Setup

• The experimental section presents the dataset, baseline explainers, and implementation details before evaluating CFExplainer.It also introduces two types of metrics for quantitative assessment.Source: §4 · p. 6
• The section uses two types of evaluation metrics to assess the effectiveness of CFExplainer quantitatively.Source: §4 · p. 6
• Implementation details are included alongside the dataset and baseline descriptions in the experimental setup.Source: §4 · p. 6

4.1 Dataset

• Big-Vul contains 188,636 C/C++ functions from 348 GitHub projects, including 10,900 vulnerable functions and 91 vulnerability types.The data spans 2002–2019 and is linked to the public CVE database.Source: §4.1 · p. 6
• Unlike Devign and Reveal, Big-Vul provides statement-level code changes from commits, which support ground-truth evaluation of explanation quality.Source: §4.1 · p. 6
• The authors clean the code, parse snippets into PDGs, undersample non-vulnerable examples for balance, and randomly split data 8:1:1.Explanations are generated only for test-set predictions.Source: §4.1 · p. 6

4.2 Baselines

• The comparison includes GNNExplainer, PGExplainer, SubgraphX, GNN-LRP, DeepLIFT-Graph, and GradCam-Graph.Source: §4.2 · p. 6
• The baselines span mutual-information subgraph methods, Monte Carlo tree search, graph-walk decomposition, and gradient-based feature importance.DeepLIFT-Graph and GradCam-Graph adapt image-classification explainers for GNNs.Source: §4.2 · p. 6
• PGExplainer, GNN-LRP, DeepLIFT-Graph, and GradCam-Graph are not black-box explainers because they require model parameters, training data, or gradients.Baseline hyperparameters follow prior implementations.Source: §4.2 · p. 6

4.3 Implementation Details

• The implementation has two components: training GNN-based vulnerability detectors and generating explanations for their predictions.Source: §4.3 · p. 6

4.3.1 GNN-Based Vulnerability Detection.

• Four two-layer GNN detectors use GCN, GGNN, GIN, or GraphConv cores, GraphCodeBERT node embeddings, graph mean pooling, and an MLP classifier.They are trained for 50 epochs with Adam, binary cross-entropy, a learning rate of 0.005, and batch size 64; all achieve over 70% accuracy, over 55% precision, over 40% recall, and over 50% F1.Source: §4.3.1 · p. 7

4.3.2 Explanation Generation.

• CFExplainer is trained separately for each code snippet for 800 epochs, and evaluation explains only correctly detected vulnerable snippets.The default KM is 8, matching the factual explainers’ explanation-subgraph size for fair comparison.Source: §4.3.2 · p. 7

4.4 Evaluating the Explainability

• Explanation quality is evaluated with vulnerability-oriented and model-oriented metrics.Source: §4.4 · p. 7

4.4.1 Vulnerability-Oriented Evaluation Metric.

• Statements deleted or modified in Big-Vul fix commits form ground truth; accuracy, precision, recall, and F1 assess explanation overlap, with scores averaged across snippets.A statement is labeled positive if it is deleted or modified in the fixed version; explanation precision measures relevance, while recall measures coverage of ground-truth statements.Source: §4.4.1 · p. 7

4.4.2 Model-Oriented Evaluation Metric.

• Probability of Necessity measures the share of explanations whose removal changes the corresponding code snippet’s vulnerability prediction.The metric addresses whether explanations influence detector decisions, beyond matching vulnerability root causes.Source: §4.4.2 · pp. 7–8

5.1 RQ1: Vulnerability-Oriented Evaluation

• Accuracy improved by 24.32%, 12.03%, 28.22%, and 14.29% across GCN, GGNN, GIN, and GraphConv, respectively, over factual explainers.CFExplainer outperformed baseline explainers in most vulnerability-oriented evaluation scenarios.Source: §5.1 · p. 8

5.2 RQ2: Model-Oriented Evaluation

• CFExplainer outperformed factual explainers in model-oriented evaluation, with its PN curve encompassing baseline curves across K_M settings.The evaluation assesses whether explanations are necessary for supporting detection predictions.Source: §5.2 · pp. 8–9, Figure 3 · p. 9

5.3 RQ3: Influence of Hyper-parameter 𝛼

• The optimal α settings were 0.9 for GCN and GraphConv, 0.8 for GGNN, and 0.5 for GIN.Increasing α encourages more counterfactual predictions, but beyond an optimum, larger perturbations may miss the most critical prediction factor.Source: §5.3 · p. 9, Figure 4 · p. 9

5.4 Case Study

• In the tcpdump case study, CFExplainer identified counterfactuals implicating buffer operations and the null-termination operation in a reported NFS parser over-read.The fix limits copied data to NFSX_V3FHMAX and null-terminates temp; some factual explanations missed the cause or included unrelated statements.Source: §5.4 · pp. 9–10, Figure 5 · p. 10

• CFExplainer depends on a reliable GNN detector; biased patterns or incorrect predictions for perturbed instances can undermine its effectiveness.The authors recommend using it with detection models that exhibit ideal performance.Source: §6 · p. 10
• The current perturbation approach is based mainly on graph theory and does not specifically account for vulnerability features.The authors plan to investigate vulnerability-tailored perturbation algorithms.Source: §6 · p. 10

7.1 Vulnerability Detection and Localization

• Deep-learning vulnerability detectors represent code semantics but do not provide fine-grained explanations of which lines cause detected vulnerabilities.Fault-localization techniques can require extensive test suites or time-consuming testing, while deep-learning line-level methods require large training samples and may lack explainability.Source: §7.1 · pp. 10–11

7.2 Explainability in Software Engineering

• Software-engineering explainability includes global, neuron-level, structural, and factual approaches, alongside code simplification methods that preserve predictions.Prior counterfactual work perturbed plain-text code, whereas CFExplainer perturbs graph inputs.Source: §7.2 · p. 11

7.3 Counterfactual Reasoning in GNNs

• Prior GNN counterfactual methods address graph neighborhoods, decision regions, autoencoders, and specialized domains; the authors position CFExplainer as the first for code graphs and vulnerability detection.The cited domains include molecular graphs and brain networks.Source: §7.3 · p. 11

8 Conclusion

• CFExplainer outperforms state-of-the-art factual reasoning-based explainers in experiments on four GNN-based vulnerability detection models.Source: §8 · p. 11
• CFExplainer identifies minimal code-graph perturbations that alter predictions, producing explanations intended to identify vulnerability root causes and offer developers actionable fixing insights.Source: §8 · p. 11
• Counterfactual reasoning in software engineering remains at an early stage, with potential applications including bug detection, code search, and code clone detection.Source: §8 · p. 11
• The paper makes its experimental data and code available in the NaturalCC counterfactual-vulnerability-detection repository.Source: §8 · p. 11