Source-linked AI summary
VulnSAGE: Adjudicating SAST Reports by Connecting LLM Reasoning to Program Structure
Jinbao Chen, Zixiang Zhou, Yu Zhang, Yihong Wu
TL;DR
SAST adjudication requires project-specific context beyond the reported operation and a decision basis that program analysis can check. VulnSAGE uses report-centered slicing to supply dependencies and explanatory structure, then checks CLAIM facts' cited program relations before deriving verdicts. Across 187 labeled reports, VulnSAGE rejects 89 of 97 false alarms and retains 85 of 90 real defects; it reproduces 104 defects from open-source alerts.
Problem
SAST adjudication requires project-specific context beyond the reported operation and a decision basis that program analysis can check.
Source: §1 · p. 1Method
VulnSAGE uses report-centered slicing to supply dependencies and explanatory structure, then checks CLAIM facts' cited program relations before deriving verdicts.
Source: §1 · p. 2Results
Across 187 labeled reports, VulnSAGE rejects 89 of 97 false alarms and retains 85 of 90 real defects; it reproduces 104 defects from open-source alerts.
Source: §1 · p. 2Takeaways & Limitations
On open-source projects, reproductions include ten previously unknown defects confirmed by upstream maintainers, nine of which are already fixed.
Source: §8 · p. 24Decision coverage depends on upstream reports and available PSEG relations; unresolved macros, indirect calls, or external-library behavior can prevent retrieval or admission.
Source: §7 · p. 23
Abstract
from arXiv · showhide
Static Application Security Testing (SAST) reports flag operations whose safety depends on project-specific context, and maintainers must decide which reports describe real defects. Large language models (LLMs) can interpret this context, but adjudicating a report with them requires the decisive code, which often lies far from the reported line, and a decision basis that program analysis can check. We present VulnSAGE, an adjudication system for SAST reports that connects LLM reasoning to program structure in both directions. Its report-centered slicer recovers the dependencies of the reported operation together with the program structure that explains them, such as the caller object on which a helper acts. CLAIM, an intermediate representation for report judgments, records each model interpretation as a semantic fact that cites the program relations it assumes. Program analysis checks these premises, and the verdict follows from the facts whose premises hold. Both mechanisms share a program graph, which lets the model cite the code it reads and lets the checker query those citations. Statically supported true positives then guide proof-of-concept generation and runtime reproduction. Across 187 labeled reports from four corpora covering five vulnerability families, VulnSAGE rejects 89 of 97 false alarms and retains 85 of 90 real defects. Under a common PoC procedure, 25 of its positive verdicts reproduce at runtime, compared with three for the LLM-based adjudicator LLM4SA. Applied to 45,440 alerts raised by five SAST tools on the 200 most-starred C/C++ projects on GitHub, it reproduces 104 defects at runtime. Maintainers have confirmed 24 of the 26 defects they have responded to, including ten previously unknown defects, nine of which are already fixed.
1 Introduction
SAST reports require project context beyond the flagged operation, while LLM judgments need a basis that program analysis can check. VulnSAGE connects those needs through report-centered slicing and CLAIM, then carries statically supported findings toward runtime reproduction.
1 Introduction
- SAST adjudication matters because accepting false alarms consumes review effort, whereas dismissing real defects leaves programs vulnerable.The paper studies 45,440 alerts raised by five SAST tools on 200 highly starred C/C++ GitHub projects.Source: §1 · p. 1
- The challenge is that decisive context, such as caller-selected constants or helper effects, may lie beyond the reported operation and be omitted from analyzer reports.LLMs can interpret project context once they see it, but existing LLM triage retrieves context and adopts the model's verdict.Source: §1 · pp. 1–2
- Its report-centered slicer recovers dependencies and explanatory program structure, while CLAIM records model interpretations with structural premises that program analysis checks.Verdicts follow from admitted facts whose cited relations hold; missing support yields an uncertain outcome.Source: §1 · p. 2
- For statically supported true positives, VulnSAGE carries trigger conditions into PoC construction and execution to expose the reported behavior.A generated BigTIFF input reproduces a confirmed libtiff allocation/copy mismatch under AddressSanitizer.Source: §1 · p. 2
- VulnSAGE rejects 89 of 97 false alarms and retains 85 of 90 real defects across 187 labeled reports; 25 positive verdicts reproduce, versus three for LLM4SA.On 45,440 open-source alerts, it reproduces 104 defects; maintainers confirmed 24 of 26 defects they responded to, including ten previously unknown defects.Source: §1 · p. 2
2 Problem Setting and Motivation
VulnSAGE adjudicates source-available C/C++ reports by combining project-specific context with checkable program relations. Its design moves from report input, through contextual interpretation and structural admission, toward runtime reproduction of supported findings.
2.1 Report Input and Outcomes
- Reports concern five vulnerability families and receive FP, TP-S, uncertain, or dynamically reproduced TP-D outcomes according to available support and runtime observation.An optional trace supplies additional locations, and the enclosing procedure refines the reported target.Source: §2.1 · p. 3
2.2 Report-Specific Context
- The three examples require distinct context: libav caller constants constrain failure guards, nginx table values determine copy length and capacity, and helper bindings identify the initialized object.The initial slice recovers dependencies with explanatory structure before model reasoning; removing it lowers accuracy by 14.81 to 27.08 percentage points.Source: §2.2 · pp. 3–4, Figure 1 · p. 4
2.3 Semantic Facts and Program Premises
- CLAIM attaches each semantic interpretation to program relations it assumes, and structural admission determines whether those premises support the verdict.The helper example separates the model's semantic judgment about initialization from checkable bindings and member relations.Source: §2.3 · pp. 4–5
2.4 Static Support and Runtime Reproduction
- For statically supported true positives, trigger conditions guide PoC construction and execution toward an observation correlated with the reported behavior.A BigTIFF input and configured invocation reproduced the previously unknown libtiff behavior under AddressSanitizer.Source: §2.4 · p. 5
2.5 Design Requirements
- The design requires recovering report-relevant dependencies with source identity, deciding from facts whose premises pass admission, and seeking runtime reproduction for supported true positives.Source: §2.5 · p. 5
3 Design
VulnSAGE links the program context shown to an LLM with the program relations used to check its judgments. A shared PSEG supports both report-focused retrieval and structural admission before verdicts are computed.
3 Design
- PSEG-backed slicing recovers report-relevant code, while CLAIM records interpretations with cited relations that structural admission checks before they support a verdict.PSEG nodes retain source identity, so the model can cite code it reads and the checker can query those relations.Source: §3.1 · p. 5
- For the helper report, slicing and targeted retrieval expose the caller-to-parameter binding and initializer call; admission checks the cited relations before the ledger returns FP.A TP-S verdict proceeds to PoC construction and execution, where a correlated runtime observation yields TP-D.Source: §3.1 · p. 6
3.2 Report-Centered Slicing
Report-centered slicing collects dependencies together with explanatory relations so that the model receives code relevant to the warning and retains source identity for its claims. PSEG supplies a shared graph for retrieval and relation checks.
3.2 Report-Centered Slicing
- The slicer follows explanatory relations as well as dependencies, seeds traversal from report trace locations, and returns source excerpts whose elements retain identity for CLAIM citations.Explanatory relations include declarations, guards, call bindings, and type or member information that give dependencies meaning for a report.Source: §3.2 · p. 6
3.2.1 PSEG Structure and Construction.
- PSEG stores program structure and report-relevant relations as labeled edges over shared nodes, enabling bounded traversal to recover context and CLAIM queries to inspect those same relations.Its attributes retain source locations and code, while edge groups include syntax, control flow, dependence, calls, symbols, and context.Source: §3.2.1 · p. 6, Table 1 · p. 7
3.2.2 Criteria and Traversal.
- Report locations resolve to seed nodes, and bounded traversal follows enabled relations backward for sources and context or forward for uses, subject to call-boundary policies.Depth is three hops for a primary location without a trace and one for trace seeds; results are grouped by source line and retain resolution gaps.Source: §3.2.2 · pp. 7–8
3.2.3 Source Reconstruction and Targeted Retrieval.
- The ContextAgent extends the initial slice with targeted source or graph queries when interpretation leaves a report-relevant question unresolved.In the helper case, retrieval traces the caller argument to the parameter and initializer use, letting CLAIM cite the binding and member relations.Source: §3.2.3 · pp. 8–9
3.3 CLAIM Intermediate Representation
CLAIM separates model-authored security interpretations from program relations that analysis can verify. Its fact structure and neutral relation vocabulary connect the interpretation to report-relevant code, and admission checks those premises before they support a verdict.
3.3 CLAIM Intermediate Representation
- CLAIM pairs each semantic judgment with cited structural premises, and the evidence ledger bases verdicts on facts whose premises pass admission.The model selects relations relevant to its interpretation; analysis checks them, and missing support yields uncertain.Source: §3.3 · p. 9
3.3.1 Syntax and Fact Fields.
- Each CLAIM groups facts, premise bundles, and source references; fact fields identify the asserted side, roles, closure, and supporting premises.The fact schema distinguishes TP source-to-sink explanations from FP mechanisms that rule out the reported behavior.Source: §3.3.1 · pp. 9–10
- Ten family templates constrain permitted semantic roles, with one TP and one FP template for each of the five vulnerability families.The templates are defined per family and side, independently of projects and benchmarks.Source: Table 3 · p. 10, §3.3.1 · p. 10
3.3.2 Structural Premises.
- CLAIM’s neutral atoms cite source endpoints and program relations that the checker resolves as PSEG queries, including data, declaration, call, and assignment relations.The relation vocabulary includes auxiliary identity and order checks, but a fact needs a structural relation beyond merely locating or ordering elements.Source: §3.3.2 · p. 11, Table 4 · p. 11
- The helper example grounds an FP judgment in checked bindings between the caller’s object, the helper parameter, and the initializer’s member argument.The null-dereference TP example instead uses assignment and data-reaching relations to support a source-to-sink carrier.Source: §3.3.2 · p. 12
3.4 Admission Semantics and Verdict
Admission resolves and checks every cited premise, then requires the fact to match the vulnerability-family template and connect to the report. The ledger uses admitted support and retrieval gates to produce TP-S, FP, or uncertain verdicts.
3.4.1 Premise Interpretation.
- A cited atom is supported, refuted, unavailable, or unresolved; identity and source-order checks can locate elements but do not establish structural connections.Endpoint resolution uses source location and, when available, function, symbol, access path, callee, and argument position.Source: §3.4.1 · pp. 12–13
3.4.2 Admission Judgment.
- Admission requires a matching family template, the candidate side, report anchoring, and neutral structural support, while every cited premise must pass.A refuted, unavailable, or unresolved atom blocks admission; identity and order relations cannot admit a fact alone.Source: §3.4.2 · p. 13
- Report anchoring requires a cited endpoint at the reported location or object, making a checked helper binding relevant only when it reaches the reported member.The helper instance’s actual argument must carry the caller’s object whose member appears at the report target.Source: §3.4.2 · p. 13
3.4.3 Repair and Verdict.
- When a helper binding is refuted, its fact fails admission even if its other premises pass; one repair round can revise citations and references without changing the candidate side.A failed fact supplies no verdict support, so the outcome depends on other admitted facts and required context.Source: §3.4.3 · p. 14
- TP-S or FP requires resolved report location, retrieved local source, and admitted support for that side; otherwise the ledger returns uncertain.Admitted facts cannot support both sides, and a failed repair leaves no support from the rejected fact.Source: §3.4.3 · p. 14
- Each decided verdict rests on a semantic fact whose cited atoms pass in PSEG, while the model affects the verdict only through its facts, citations, and retrieval requests.Admission and verdict computation are functions of CLAIM, PSEG, the report, and retrieved context.Source: §3.4.3 · p. 14
3.5 Runtime Reproduction
VulnSAGE carries statically supported true positives into runtime tests that target the reported behavior. A correlated sanitizer report or crash strengthens the static basis with a reproduction, while a clean or unsuccessful run leaves the static verdict unchanged.
3.5 Runtime Reproduction
- For each TP-S, the system hands the report target, admitted static basis, trigger conditions, and source context to PoC generation and runtime testing.The PoCAgent builds an input or harness, can retrieve missing execution information, and runs the test against a selected revision and configuration.Source: §3.5 · p. 14
- A runtime result correlates when a sanitizer report or crash has a stack frame at the reported operation and the selected project build runs the specified invocation.The frame may match the reported file and line or its function.Source: §3.5 · p. 14
- A correlated observation strengthens the static TP basis with reproduction, but a clean or unsuccessful execution preserves the static verdict because the input may not reach the required state.Source: §3.5 · p. 15
4 Implementation
The implementation builds PSEG over Joern’s code property graph and coordinates report interpretation, context retrieval, CLAIM checking, and runtime testing. Reports are processed against their associated source revisions.
4 Implementation
- Report normalization extracts the vulnerability family, source location, diagnostic text, and optional trace, and analysis uses the report’s associated source revision.Multifinding analyzer reports are processed independently.Source: §4 · p. 15
- PSEG overlays implicit program relations on Joern’s CPG, while its service performs location queries, traversals, and relation checks and returns source references and resolution status.Overlay passes add relations such as argument correspondence and switch-label control flow; source and project searches supplement graph queries.Source: §4 · p. 15
- The Supervisor routes work among the MasterAgent and ContextAgent; the MasterAgent builds CLAIMs, the ContextAgent retrieves context, and the PoCAgent constructs tests for TP-S.Schema validation precedes admission, and the ledger computes verdicts from admitted facts and retrieved context.Source: §4 · p. 15
5 Evaluation
The evaluation measures adjudication quality, the effects of report-centered slicing and CLAIM admission, runtime reproduction, and maintainer confirmation across labeled reports and large-scale C/C++ scans.
5.1 Experimental Setup
- The evaluation combines Juliet test cases with D2A-libtiff, SATE-V, and ARVO reports from real projects, using matching source revisions for real-world labels.D2A-libtiff reports were re-adjudicated against source and, where available, upstream fixes; Juliet covers all five vulnerability families.Source: §5.1 · pp. 15–17
5.2 RQ1: Adjudication Quality
- VulnSAGE rejects 89 of 97 false alarms and retains 85 of 90 real defects, while deciding at least 96% of reports in each set.On D2A-libtiff, it rejects 38 of 43 false alarms and retains four of five real defects; 25 of 35 retained defects across three real-world sets reproduce at runtime.Source: §5.2 · p. 17, Table 6 · p. 17
5.3 RQ2: Report-Centered Slicing and CLAIM Admission
- Without pre-slicing, all-report accuracy falls from 87.50% to 60.42% on D2A-libtiff and from 92.59% to 77.78% on SATE-V.Without admission, unsupported facts promote 14 D2A-libtiff false alarms and reject three SATE-V real defects, versus four and one with admission.Source: §5.3 · p. 18, Table 7 · p. 18
5.4 RQ3: Reproducible Verdicts
- Under a common PoC procedure, 25 VulnSAGE positives reproduce at runtime, compared with three for LLM4SA.In ARVO finding 46653, VulnSAGE’s context preserves the path origin and execution conditions needed for a PoC that triggers the out-of-bounds read; LLM4SA’s attempt does not.Source: §5.4 · p. 18, §5.4.1 · p. 19, Table 8 · p. 19
5.5.1 Scan Results.
- Of 45,440 alerts, static adjudication assigns 1,040 to TP-S and filters out 97.71%; root-cause merging yields 486 candidate defects.PoC generation advances 104 of 154 candidates runnable in the container to TP-D, a rate of 67.53%.Source: §5.5.1 · p. 20, Table 9 · p. 20
5.5.2 Upstream Confirmation.
- Maintainers confirm 24 of the 26 reproduced defects they have responded to, including ten previously unknown defects; nine of those ten are fixed.The ten previously unknown defects came from all five SAST tools.Source: §5.5.2 · p. 20
5.5.3 Case Study: libtiff 4.5.0.
- A libtiff BigTIFF input sets StripByteCounts to 0x100000005, which narrows to 5 for allocation before a later raw-strip copy causes an AddressSanitizer heap-buffer-overflow.The configured tiff2pdf memory-limit option reaches the vulnerable path, advancing the report from TP-S to TP-D.Source: §5.5.3 · pp. 21–22, Figure 4 · p. 21
5.5.4 Case Study: 7-Zip-zstd.
- In 7-Zip-zstd, releasing the last counted reference before a success-branch size read through a raw pointer produces an eight-byte heap-use-after-free under AddressSanitizer.Valid Zstandard files reproduce the read; the maintainers fixed it by moving the release below the size query in all five single-stream handlers sharing the code.Source: §5.5.4 · p. 22, Figure 5 · p. 22
6 Related Work
Related work uses LLMs to classify or investigate security alerts, combines LLMs with program analysis, and generates or adapts runtime tests. VulnSAGE emphasizes verdicts grounded in checked program relations and reproduction guided by an admitted static basis.
LLM-assisted report adjudication
- LLift, LLM4SA, ZeroFalse, and an industrial study aim to classify alerts; VulnSAGE instead targets the basis for each verdict and carries supported reports into runtime reproduction.LLift uses postconstraint-guided reasoning, LLM4SA uses domain knowledge and representative cases, and ZeroFalse adds flow traces and CWE-specific knowledge [Iranmanesh et al. 2025].Source: §6 · pp. 22–23
Integration of LLMs and program analysis
- IRIS, GPTScan, and InferROI combine LLMs with program analysis, while CLAIM has the model interpret an existing report and admission check its cited relations.The verdict follows from admitted facts.Source: §6 · p. 23
Program slicing and representations
- Unlike slicing and graph-based approaches that extract vulnerability-focused context, VulnSAGE derives slice criteria from the report and preserves source identity so CLAIM facts can cite the code read by the model.Prior approaches include dependence graphs, CPGs, and LLMxCPG [Lekssays et al. 2025].Source: §6 · p. 23
Runtime reproduction and PoC generation
- Runtime-generation approaches address exploit synthesis, dynamic verification, PoC adaptation, or hybrid triage; VulnSAGE guides reproduction from an existing SAST report and its admitted static TP basis.Examples include NodeMedic-FINE, ProtocolGuard, SyzBridge, and Agrawal and Ahi [2026].Source: §6 · p. 23
7 Limitations
VulnSAGE’s limits concern semantic interpretation, report and structural coverage, model variability, and the conditions required for runtime reproduction. These boundaries constrain which reports can be decided and which supported reports can be reproduced.
Semantic interpretation
- A model’s semantic interpretation can be wrong even when its cited structural premises are supported.Semantic models of project contracts could extend direct checking of properties such as capacity or ownership validity.Source: §7 · p. 23
Report and structural coverage
- Unresolved macro expansion, indirect calls, or external-library behavior can leave PSEG relations unavailable and prevent retrieval gates or admission conditions from being met.Coverage also depends on upstream reports and available PSEG relations for matching source revisions.Source: §7 · p. 23
Model variability
- Models, prompting configurations, and program-analysis pipelines can vary, so the reported results describe the evaluated configuration; prior training exposure may affect semantic judgments.Structural admission checks cited relations independently of the model.Source: §7 · p. 23
Runtime reproduction
- Runtime reproduction requires a buildable revision and an input that reaches the reported operation under a suitable configuration.Each reproduction concerns its tested input and configuration; broader input construction and environment support could reproduce more statically supported reports.Source: §7 · p. 23
8 Conclusion
VulnSAGE links report-centered slicing and checked semantic facts so verdicts rest on supported program relations. Its judgments guide inspectable, rerunnable tests, including reproductions of previously unknown defects confirmed by maintainers.
8 Conclusion
- VulnSAGE’s report-centered PSEG slicing gives the model report dependencies together with the structure that explains them.The shared graph supports retrieval of program context and structural checks of the model’s cited relations.Source: §8 · p. 24
- CLAIM records model interpretations as semantic facts with premises that program analysis checks, and verdicts follow from facts whose premises hold.Source: §8 · p. 24
- Maintainers confirmed ten previously unknown defects among the open-source reproductions, and nine of those defects are already fixed.The resulting judgments guide tests that maintainers can inspect and rerun.Source: §8 · p. 24