Source-linked AI summary
Reva: Real-Time Vulnerability Detection through Contextual Probing of Large Language Models
Amine Lbath, Massih-Reza Amini
TL;DR
Coding agents outpace developer review, while agentic vulnerability detection takes minutes per function and faster direct-verdict detectors are less accurate. REVA supplies an LLM with security context and uses a trained probe to read the function’s internal representations in a single forward pass. On PrimeVul’s held-out test set, REVA on GLM-5.3-Flash is on par with VulTriage and more accurate than VulAgent on the same LLM, taking about one second per function.
Problem
Coding agents outpace developer review, while agentic vulnerability detection takes minutes per function and faster direct-verdict detectors are less accurate.
Source: §I · p. 1Method
REVA supplies an LLM with security context and uses a trained probe to read the function’s internal representations in a single forward pass.
Source: §I · p. 1Results
On PrimeVul’s held-out test set, REVA on GLM-5.3-Flash is on par with VulTriage and more accurate than VulAgent on the same LLM, taking about one second per function.
Source: §I · p. 1Takeaways & Limitations
REVA’s results support using LLM internal representations for real-time screening of functions as they are written.
Source: §VI · p. 4For GLM-5.3-Flash, test accuracy is 6 points below cross-validation accuracy (60.0% against 66.2%), a gap the authors say likely reflects PrimeVul’s chronological shift toward newer code.
Source: D. RQ3: Ablation and Robustness · p. 4
Abstract
from arXiv · showhide
A vulnerability costs more to fix the later it is discovered. As coding agents generate code faster than developers can review it, software needs a first line of defense screening each new function in real time. LLM agents have recently become capable of finding real-world vulnerabilities by working as a security expert would: they run analysis tools, explore the surrounding code, test hypotheses and weigh the evidence before forming their own judgement. This reasoning, however, spans many model calls, so that each verdict takes minutes of compute, far too long for real-time screening. Faster detectors, which fine-tune deep learning models or prompt an LLM for a direct verdict, are less accurate: they judge a function without the context an agent gathers, and an LLM's generated verdict is unreliable even when given some of it. We present REVA (REal-time Vulnerability Analysis), an AI vulnerability detector that brings the security context and code understanding components behind agentic detectors into a single forward pass of an LLM. For each new function, REVA gives the LLM a pre-computed threat model of its module, lightweight static analysis findings and call-graph information. REVA leverages a trained probe to read the LLM's internal representations of the new function in this context. On PrimeVul's held-out test set, REVA on GLM-5.3-Flash reaches 60.0% accuracy in about one second per function. This matches VulTriage (59.2%), the state-of-the-art agentic detector, run on the same LLM, while being over 250 times faster and less than a tenth as costly. On the same prompt, the probe is 14% to 17% more accurate than the LLM's own direct verdict. Across three open LLMs, REVA is at least as accurate as VulTriage, and about 10% more accurate on two of them. An LLM's internal representations thus encode much of what agentic detectors obtain through reasoning.
I. INTRODUCTION
REVA targets real-time vulnerability screening by combining security context with an LLM’s internal representations in a single forward pass. On PrimeVul, it performs comparably to or better than agentic detectors while taking about one second per function.
I. INTRODUCTION
- Because remediation costs grow later in the software lifecycle and coding agents outpace developer review, the paper motivates screening each new function as it is generated.The stated goal is a first line of defense for real-time screening.Source: §I · p. 1
- Agentic detectors can find real-world vulnerabilities using tools and code analysis, but their multi-call reasoning takes minutes per function.Faster detectors that fine-tune models or request direct LLM verdicts are considerably less accurate.Source: §I · p. 1
- REVA combines a module threat model, static-analysis findings, call-graph information, and file context with a probe that maps code-token representations to a risk score in one forward pass.The LLM processes the prompt without generating text; the probe reads its internal representations.Source: §I · p. 1
- On PrimeVul’s held-out test set, REVA on GLM-5.3-Flash is on par with VulTriage and 9% to 13% more accurate than both agentic detectors on two other open LLMs.It runs in about one second per function, over 250 times faster than VulTriage.Source: §I · p. 1
- The paper compares REVA with agentic detectors on three LLMs and PrimeVul’s held-out test set, measuring accuracy, cost, and robustness.It also outlines an agenda for screening every generated function and escalating likely vulnerabilities to agents that construct proofs of vulnerability.Source: §I · p. 1
II. BACKGROUND AND RELATED WORK
The background contrasts multi-step agentic detection and direct LLM verdicts with REVA’s contextual probe of real-world code. It also highlights benchmark-pair checks as a way to detect spurious classification behavior.
II. BACKGROUND AND RELATED WORK
- Figure 1 depicts REVA combining a function with security context, retaining code-token hidden states across probed layers, and pooling them into a risk score.The caption describes a single forward pass up to the deepest of four probed layers.Source: Figure 1 · p. 2
- On PrimeVul, GPT-4 with chain-of-thought labels both versions of a pair correctly in only 12.9% of cases, while REVA reads internal representations rather than the LLM’s answer or reasoning.The passage also notes that non-agentic LLM verdicts can reason inconsistently about security.Source: §II · p. 2
- REVA replaces multi-step agentic reasoning with a prompt containing key context and a single probed forward pass.Agentic approaches add knowledge and reasoning, but their calls can be costly and their context can exceed the model’s window.Source: §II · p. 2
- Unlike earlier code probes that read code alone or use toy datasets, REVA probes real-world functions together with targeted security context.Prior work used linear probes to separate secure from insecure code or predict insecure or incorrect generated code.Source: §II · p. 2
- Scoring vulnerable and benign counterparts independently, then checking whether both receive correct labels, tests whether a detector relies on shared spurious features.The passage notes that paired functions can share code metrics or semantics unrelated to the vulnerability.Source: §II · p. 2
III. CONTEXTUAL PROBING WITH REVA
REVA supplies each new function with project and module security context, then uses one LLM pass and an attention probe to produce a risk score. The context is prepared ahead of time, while function-specific processing includes context extraction and static analysis.
III. CONTEXTUAL PROBING WITH REVA
- The prompt combines each new function with a project and module threat model, Semgrep findings, call-site and helper information, and file-level repository context.The threat model describes the project, attacker-controlled inputs, trust boundaries, and the module’s role, inputs, sinks, and guards.Source: §III · p. 2
- Repository context is available for 84% of training functions; otherwise, the prompt contains the function and its Semgrep findings.The median prompt has 3,840 tokens, including 601 code tokens.Source: §III · p. 2
- REVA stops the LLM after the deepest of four probed intermediate layers, selected by cross-validation, and generates no output token.The four layers are evenly spaced from 38% to 85% of model depth.Source: §III · p. 2
- At each probed layer, REVA retains only function-token hidden states, normalizes and projects them to 1,024 dimensions, then concatenates the layers.Source: §III · p. 2
- An eight-head attention probe pools token states into a risk score and is trained on labeled functions with classification and vulnerable-over-benign ranking losses.Training pairs supply the ranking signal, but REVA scores each function independently.Source: §III · p. 2
- With Muse-Glimmer-30B on one H200 GPU, the practical pipeline takes 0.93 seconds per function, including 0.46 seconds for context extraction.The LLM and Semgrep rules are loaded once, and each project’s threat model is built ahead of time.Source: §III · p. 2
IV. PRELIMINARY RESULTS
The preliminary evaluation is organized around REVA’s accuracy against agentic detectors, its computational savings, and the components and robustness of the approach.
IV. PRELIMINARY RESULTS
- RQ1 asks how accurately REVA performs compared with agentic detectors running on the same LLM.Source: §IV · p. 2
- RQ2 asks how much computation REVA saves.Source: §IV · p. 2
- RQ3 examines which parts of REVA matter and whether its accuracy is robust to leakage and selection.Source: §IV · p. 2
A. Experimental Setup
The experiments evaluate REVA on PrimeVul using held-out testing, cross-validation, and same-LLM agentic baselines. Accuracy is measured over individual functions, with failed detector outputs counted as benign.
A. Experimental Setup
- PrimeVul pairs each vulnerable C/C++ function with a benign version of the same function and has 3,789 training, 480 validation, and 435 test pairs.Source: A. Experimental Setup · p. 2
- Accuracy is scored over all functions independently; pair outcomes are also categorized as both correct, both flagged, both cleared, or reversed.These pair categories are P-C, P-V, P-B, and P-R, respectively.Source: A. Experimental Setup · p. 3
- Design choices use training and validation data, while the test set remains unseen until scoring the final configuration.Cross-validation combines five training folds, keeping functions from the same commit together, with validation accuracy from a probe trained on the full training set.Source: A. Experimental Setup · p. 3
- REVA is compared with VulAgent and VulTriage run on the same three open-weight LLMs; functions receiving no verdict count as benign.Published GPT-4o results are included only for context.Source: A. Experimental Setup · p. 3
B. RQ1: Accuracy
On PrimeVul, REVA matches or exceeds the agentic detectors across the three evaluated LLMs, with accuracy and false-positive comparisons favoring REVA in most same-model comparisons. Its accuracy varies less across LLMs than VulTriage's.
B. RQ1: Accuracy
- 60.0% accuracy for REVA on GLM-5.3-Flash compares with 59.2% for VulTriage and 52.8% for VulAgent on the same LLM.REVA's accuracy confidence interval is [57.8, 62.2]; its 0.8-point lead over VulTriage has a confidence interval of [−2.1, +3.8].Source: B. RQ1: Accuracy · p. 3
- 5.1 to 6.7 points more accurate than both agentic detectors, REVA performs on Qwen3.8-27B and Muse-Glimmer-30B, with all four confidence intervals excluding zero.Source: B. RQ1: Accuracy · p. 3
- Across all three LLMs, REVA has a lower false positive rate than VulTriage and reverses fewer vulnerable-benign pairs; on two LLMs it also labels more pairs entirely correctly.The authors interpret these comparisons as more reliable separation of vulnerable and benign functions, suggesting the probe captures vulnerability rather than surface features shared with secured counterparts.Source: B. RQ1: Accuracy · p. 3
- REVA is at least as accurate as both agentic detectors on each LLM and is 100 to 6,000 times faster; its accuracy varies by 1.1 points across LLMs versus 5.6 for VulTriage.VulTriage on GPT-4o reports 63.6% accuracy, but the authors could not replicate that result.Source: B. RQ1: Accuracy · p. 3
C. RQ2: Computational Cost
REVA reduces the time and cost of function-level screening relative to agentic detectors run on the same LLMs. It processes prompts without generating tokens, but requires access to LLM hidden states.
C. RQ2: Computational Cost
- 0.88 s to 1.33 s per function for REVA compares with 108 s to 5,614 s for agentic detectors on the same LLMs.The measurements use parallel load and therefore overstate single-request latency.Source: C. RQ2: Computational Cost · p. 3
- 3 to 70 times fewer tokens are processed by REVA than by agentic detectors; REVA reads about 3,100 to 3,400 tokens per function and generates none.Source: C. RQ2: Computational Cost · p. 3
- $0.9 to $1.3 per 1,000 functions is REVA's estimated cost on a rented H200 GPU, compared with $9 to $61 for VulTriage and $63 to $307 for VulAgent.Because REVA needs LLM hidden states, it cannot run behind an API.Source: C. RQ2: Computational Cost · p. 3
D. RQ3: Ablation and Robustness
Ablations show that the probe and its attention pooling matter to REVA's accuracy, while repository context has no significant effect on PrimeVul. Robustness checks find limited loss on unseen projects but a larger chronological test-set gap.
D. RQ3: Ablation and Robustness
- Chance-level accuracy results when the same LLM gives a Yes/No verdict instead of being probed; static analyzers alone also score near chance at 50.8% to 51.3%.On the test set, the direct verdict is 7.4 to 8.7 points below the probe.Source: D. RQ3: Ablation and Robustness · pp. 3–4
- 4.2 points of accuracy are lost when attention pooling is replaced by averaging token states, suggesting vulnerability evidence lies in a few tokens.Single-layer probing, omitting standardization or the ranking term, and changing repository context each alter accuracy by 0.4 to 1.9 points; none of these changes is significant.Source: D. RQ3: Ablation and Robustness · p. 4
- 1.3 to 2.7 points is the cross-validation accuracy drop when projects are kept out of both training and evaluation folds; choosing probed layers on training folds changes accuracy by less than half a point.Source: D. RQ3: Ablation and Robustness · p. 4
- 6 points lower is test-set accuracy than cross-validation accuracy for GLM-5.3-Flash: 60.0% versus 66.2%.The authors suggest the chronological split's shift toward newer code likely explains the gap.Source: D. RQ3: Ablation and Robustness · p. 4
V. FUTURE PLANS
Future work will test REVA across larger open LLMs and coding-agent workflows, then assess its generalization beyond PrimeVul.
V. FUTURE PLANS
- REVA will be compared with both agentic detectors on larger open LLMs, including tests of whether short self-generated analysis adds information.The success criterion is matching or exceeding the detectors on the same LLM at every scale.Source: §V · p. 4
- A project- and function-dependent threshold will route only high-risk functions to an agentic detector or proof-of-vulnerability check.The planned success criterion is finding more vulnerabilities than the agentic detector alone for a given escalation budget.Source: §V · p. 4
- REVA will be evaluated on other datasets, interprocedural vulnerabilities, and vulnerabilities disclosed after each LLM’s training cutoff.The plan also studies fine-tuning the LLM with the same objective; the success criterion is retaining accuracy on vulnerabilities the LLM could not have seen.Source: §V · p. 4
VI. CONCLUSION
REVA probes an LLM’s contextual representations in one forward pass and matches or exceeds agentic detectors on PrimeVul’s held-out test set.
VI. CONCLUSION
- REVA probes an LLM’s internal representations of a function and its security context in a single forward pass.On PrimeVul’s held-out test set, it matches or exceeds agentic detectors run on the same LLM, with fewer false positives, in about one second per function and at a small fraction of their cost.Source: §VI · p. 4
- An LLM’s internal representations carry more information about vulnerabilities than its verdicts.Reading those representations enables real-time screening of every function as it is written.Source: §VI · p. 4
- REVA matches or exceeds agentic detectors on the same LLM while producing fewer false positives.The reported comparison is on PrimeVul’s held-out test set.Source: §VI · p. 4