Source-linked AI summary
Probing Model Signal-Awareness via Prediction-Preserving Input Minimization
Sahil Suneja, Yunhui Zheng, Yufan Zhuang, Jim Laredo, Alessandro Morari
TL;DR
AI-for-code models can achieve strong vulnerability-detection performance without necessarily learning genuine vulnerability signals. P2IM minimizes inputs while preserving predictions and uses SAR to test whether reduced snippets retain the original vulnerability; Recall falls sharply from the high 90s to below 60 under this evaluation.
Problem
Method
P2IM treats the trained detector as a black box, iteratively minimizes compilable test samples while preserving predictions, and verifies the reduced snippets’ vulnerability profiles.
Source: §1 · pp. 1–2Results
Recall drops from the high 90s to below 60 with SAR across three neural architectures and multiple datasets, suggesting reliance on noise or dataset nuances.
Source: §1 · pp. 1–2, §7 · p. 9Takeaways & Limitations
SAR provides a signal-aware supplement to traditional model metrics and is expected to match Recall when models capture task-specific signals.
Source: §1 · p. 2SAR is bounded by the quality of the checker used to verify vulnerabilities in minimized subprograms, although P2IM is not tied to a specific checker.
Source: §4.1 · p. 7, §5.3 · p. 8
Abstract
from arXiv · showhide
This work explores the signal awareness of AI models for source code understanding. Using a software vulnerability detection use case, we evaluate the models' ability to capture the correct vulnerability signals to produce their predictions. Our prediction-preserving input minimization (P2IM) approach systematically reduces the original source code to a minimal snippet which a model needs to maintain its prediction. The model's reliance on incorrect signals is then uncovered when the vulnerability in the original code is missing in the minimal snippet, both of which the model however predicts as being vulnerable. We measure the signal awareness of models using a new metric we propose- Signal-aware Recall (SAR). We apply P2IM on three different neural network architectures across multiple datasets. The results show a sharp drop in the model's Recall from the high 90s to sub-60s with the new metric, highlighting that the models are presumably picking up a lot of noise or dataset nuances while learning their vulnerability detection logic. Although the drop in model performance may be perceived as an adversarial attack, but this isn't P2IM's objective. The idea is rather to uncover the signal-awareness of a black-box model in a data-driven manner via controlled queries. SAR's purpose is to measure the impact of task-agnostic model training, and not to suggest a shortcoming in the Recall metric. The expectation, in fact, is for SAR to match Recall in the ideal scenario where the model truly captures task-specific signals.
1 INTRODUCTION
The paper introduces signal awareness as a missing dimension of source-code model quality and presents P2IM and SAR to test whether vulnerability detectors rely on task-relevant signals. Across architectures and datasets, models retain vulnerable predictions on reduced code while their Recall drops sharply under signal-aware evaluation.
Approach
- P2IM reduces vulnerable source samples to minimal prediction-preserving snippets and checks whether the original vulnerability remains, exposing reliance on incorrect signals.The approach queries an untouched black-box model and verifies reduced samples with a bug checker while preserving compilability.Source: §1 · pp. 1–2
Evaluation
- The work evaluates CNN, RNN, and GNN vulnerability detectors across three datasets, including a real-world Github dataset.The architectures operate on code as images, token sequences, and graphs, respectively.Source: §1 · p. 1
Results
- Recall drops from the high 90s to the low 60s under SAR, suggesting models often use features irrelevant to vulnerability detection.More than 85% of samples can be reduced while maintaining their vulnerable prediction across datasets and models.Source: §1 · pp. 1–2
Implications
- SAR is intended to supplement traditional metrics and should match Recall when models truly capture task-specific signals.The authors frame the gap between SAR and Recall as evidence motivating more task-aware training, not as a defect in Recall.Source: §1 · p. 2
Approach
- P2IM uses Delta Debugging to accelerate minimization and differs from instance-level interpretability methods by testing whether highlighted code contains the correct task-relevant signal.It is intended to measure signal awareness through controlled black-box queries, not to craft adversarial attacks.Source: §1 · p. 2
Contributions
- P2IM is black-box and model-agnostic, emits valid compilable code for external verification, and introduces Signal-aware Recall as a comparison metric.These properties support evaluation across classic machine-learning and neural-network models.Source: §1 · p. 2
2 MOTIVATION
The motivation is that strong conventional performance does not establish that code models learn genuine vulnerability signals. The paper therefore probes reduced variants to quantify signal awareness and distinguish task-relevant learning from dataset-specific cues.
2 MOTIVATION
- A model can achieve a strong buggy-versus-healthy separator by exploiting dataset nuances such as sample length, variable names, or programming constructs rather than vulnerability signals.Such performance can create a false sense of security despite apparent classification success.Source: §2 · p. 2
- Signal awareness matters for trusting AI vulnerability detectors alongside or against traditional analyzers and for comparing model quality beyond statistical metrics.The paper treats it as a measure that can guide improvements across model evolutions.Source: §2 · p. 2
- A 99 F1 GNN model flips its prediction on slightly different but semantically identical code variants, illustrating weak robustness.The paper distinguishes these observations from its main goal of measuring task-agnostic training effects.Source: §2 · p. 3
- The probing procedure queries the original test set with iteratively reduced samples and counts whether prediction-preserving minima retain the original vulnerability.Those counts provide the basis for SAR.Source: §2 · p. 3
- The authors hope exposing the SAR–Recall gap will motivate more task-aware learning for source-code understanding.They present this as a future research direction rather than a demonstrated training intervention.Source: §2 · p. 3
3 BACKGROUND
The background covers three code representations used by AI-for-code models and introduces Delta Debugging as the minimization foundation for P2IM. CNNs process embedded code as matrices, RNNs process token sequences, and GNNs process code graphs before DD finds outcome-preserving minima.
3.1 AI-for-code Models
- CNNs process embedded source-code tokens as real-valued matrices, adapting image-oriented convolution and pooling to vulnerability detection.Convolution extracts increasingly complex patterns, while pooling downsamples features.Source: §3.1 · p. 3
- RNNs model source code as token sequences, maintaining working memory and usually producing one final representation for vulnerability detection.Their gates update memory using current inputs and prior state.Source: §3.1 · p. 3
- GNNs operate on code graphs such as abstract syntax, data-flow, and control-flow graphs through message passing, aggregation, and updating.This graph structure is a natural fit for source code and has supported vulnerability-detection applications.Source: §3.1 · pp. 3–4
3.2 Delta Debugging
- Delta Debugging minimizes an input while preserving an oracle outcome, producing a 1-minimal sequence from which no single element can be removed.It was originally developed to simplify failure-inducing browser inputs for diagnosis and repair.Source: §3.2 · p. 4
- DD repeatedly tests subsets and complements, increases partition granularity when needed, and retains any outcome-preserving reduction.In favorable cases, this behaves like a binary search for a minimal input.Source: §3.2 · p. 4
- P2IM replaces DD’s failure oracle with a prediction model and then checks the resulting 1-minimal subprogram for the original vulnerability.This translates failure-inducing input simplification into signal-awareness measurement.Source: §3.2 · p. 4
4 DESIGN
The design treats vulnerability-detection models as black boxes and uses P2IM to test whether their predictions depend on vulnerability-relevant signals. It iteratively minimizes vulnerable samples while preserving predictions, then measures signal awareness conservatively against checker quality.
4 DESIGN
- P2IM frames source-code model analysis as testing whether black-box vulnerability detectors learn vulnerability-relevant signals.The approach addresses the implicit logic of AI-for-code models rather than relying on directly perceptible reasoning.Source: §4 · p. 4
4.1 P2IM Workflow
- P2IM iteratively removes tokens from a vulnerable test sample and queries the model until reaching a 1-minimal snippet whose single-token removals change the prediction.The workflow starts from a model-predicted vulnerable test-set sample and preserves the vulnerability prediction throughout reduction.Source: §4.1 · p. 4
- SAR gives an upper-bound estimate of signal awareness because P2IM credits the model based on the reached 1-minimal, even if it is not the global minimum.The conservative measurement still exposes models that fail to pick up correct signals during learning.Source: §4.1 · p. 7
- SAR depends on the checker used to verify bugs in reduced programs, with quality varying across dataset labels, line-based matching, and static analysis.Infer worked well for the paper’s settings, while line-based matching provides a looser, more model-favoring bound; P2IM itself is not tied to Infer.Source: §4.1 · p. 7
5 EVALUATION
The evaluation applies P2IM to three code representations across three datasets and compares conventional Recall with Signal-aware Recall (SAR). Despite strong standard performance, SAR reveals substantial reliance on signals that may not be vulnerability-relevant.
5 EVALUATION
- P2IM evaluates trained black-box models by reducing each true-positive sample to a 1-minimal version and checking whether its original bug remains.True positives whose minimal versions retain the bug become signal-aware TP’; those whose bugs disappear become signal-agnostic FN’.Source: §5 · p. 7
- SAR is defined as TP’ / (TP’ + FN’ + FN), contrasting with Recall = TP / (TP + FN) to quantify task-relevant signal learning.Both metrics operate over the same vulnerable samples, enabling comparison of reported Recall with signal-aware performance.Source: §5 · p. 7
5.1 Datasets
- The evaluation uses Juliet, balanced s-bAbI, and a 6,728-function Github dataset, with CNN, RNN, and GNN models operating on different code representations.The models frame vulnerability detection as binary classification; the Github data pairs before-fix vulnerable functions with corresponding patched functions.Source: §5.1.1 · p. 7, §5.1.2 · p. 7, §5.1.3 · p. 7
5.3 Results
- 95+ Recall drops below 60 across models on s-bAbI when P2IM tests signal awareness, with similar results on the other datasets.The s-bAbI dataset targets only buffer-overflow vulnerabilities, yet the Recall-to-SAR gap remains substantial.Source: §5.3 · p. 8
- P2IM reduces 39%-46% of samples without changing model predictions, while more than 85% of samples can be reduced while maintaining vulnerable predictions across datasets and models.Source: §5.3 · p. 8
- Valid-compilation constraints prevent some reductions and limit certainty about signal capture; for CNN+Github, all 13% of unreduced true positives were due to compilation failures.For CNN+Juliet, 89.1% of the 14.1% unreduced true positives had Windows headers unavailable in the Linux environment.Source: §5.3 · p. 8
- On synthetic datasets, over 90% overlap across models for TP and signal-aware TP’ suggests shared dataset-specific nuances, whereas overlap is lower on real-world data.The real-world dataset also shows lower performance than synthetic datasets, alongside greater data variety and fewer artificial nuances.Source: §5.3 · p. 8, Table 4 · p. 9
- Adding SAR to standard metrics enables measurement of how much model learning is task-aware and supports fairer comparisons across model evolutions.The proposed metric is intended as a supplement to Recall, not as a claim that Recall itself is deficient.Source: §5 · p. 7, §5.3 · p. 9
6 RELATED WORK
Prior work uses program slicing and Delta Debugging to isolate relevant code or failures, while explanation methods probe model internals or approximate local decision boundaries. P2IM instead treats models as black boxes, produces minimal compilable representations, and introduces SAR to quantify signal awareness.
- Program slicing identifies subsets of code that introduce defects, and this work applies the same relevant-subset intuition to black-box vulnerability models.Source: §6 · p. 9
- Delta Debugging reduces tests needed for failure isolation and is repurposed here to systematically generate model-tested program subsets.Source: §6 · p. 9
- Unlike a parallel Delta Debugging study focused on qualitative reduced-code properties, this work quantifies signal-agnostic training effects through SAR.Source: §6 · p. 9
- P2IM also fits the metamorphic-testing paradigm by comparing predictions on original and minimized inputs, with violations when vulnerability status changes.Source: §6 · p. 9
- P2IM complements gradient, local surrogate, and graph-explanation methods by using black-box queries to produce precise minimal, compilable code without feature-threshold approximation.Source: §6 · p. 9
- P2IM is distinguished from prior reliability metrics by directly probing whether true vulnerability signals remain in models’ minimal representations, motivating SAR as a new signal-awareness metric.Source: §6 · p. 9
7 CONCLUSION
The paper presents P2IM and SAR to evaluate whether AI-for-code vulnerability models rely on vulnerability-related signals. Across architectures and datasets, the sharp performance drop suggests reliance on noise or dataset nuances, while SAR complements traditional metrics for comparing model evolution.
- P2IM reduces programs to prediction-preserving minimal snippets, and SAR measures whether those snippets retain the original vulnerability rather than merely the model’s vulnerable prediction.Source: §7 · p. 9
- Across three state-of-the-art neural models and multiple datasets, SAR shows a sharp performance drop, suggesting reliance on noise or dataset nuances instead of vulnerability-related signals.Source: §7 · p. 9
- SAR augments traditional performance measures by assessing task-relevant learning for fairer comparison and guidance across model evolutions.Source: §7 · p. 9