PaperlayerView exact revision on arXiv

Source-linked AI summary

SemWeaver: Refining LLM-Generated Static Vulnerability Checkers with Analyzer-Internal Evidence

Chengzhi Yi, Shenao Wang, Yanjie Zhao, Xiao Cheng, Haoyu Wang

arXiv:2610.34928v1cs.SE

TL;DR

Patch-driven checker generation and report-based refinement do not necessarily explain the semantic reason a checker misses or misclassifies code. SemWeaver adds analyzer-internal evidence to an iterative editing loop and tests the checker on vulnerable and fixed revisions. On KNighter-derived Linux checkers, it recovers previously silent vulnerable revisions and improves paired discrimination over report-driven refinement alone.

  • Problem

    Patch-driven checkers may capture what changed without encoding why the fix makes the code safe, while compilation and validation feedback report symptoms without explaining the checker’s misunderstanding.

    Source: §1 · p. 2

  • Method

    SemWeaver assembles patch-relevant analyzer evidence with source context and validation feedback, then iteratively edits and validates the checker on both revisions under a retention gate.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    On 39 selected KNighter-derived Linux CSA checkers, SemWeaver recovers previously silent vulnerable revisions and improves paired discrimination over report-driven refinement; native and no-internal-evidence refinement also repair some different cases.

    Source: §1 · p. 3, §5.2 · p. 9, §10 · p. 18

  • Takeaways & Limitations

    The findings show that refinement also requires helping a checker recognize relevant operations in the analyzer’s representation, with repairs such as recognizing lowered callbacks or unwrapping expression casts.

    Source: §7 · p. 16, §10 · p. 18

    The 39 subjects come from one generator’s Linux artifacts, limiting the diversity of checker styles and vulnerability types; broader CodeQL transfer remains future work.

    Source: §8 · p. 17

Abstract

from arXiv · show

Recent work shows that LLMs can synthesize static-analysis checkers from security patches, but a compilable checker is not necessarily a correct one. A patch reveals what changed, not why the change is safe, so generated checkers often match only the surface edit and miss the deeper semantic condition. Existing refinement loops feed back compilation errors and validation reports, yet these signals describe symptoms without explaining what the checker misunderstands about the code. We observe that the analysis framework itself already computes the information needed to close this gap. Control-flow graphs, symbolic values, memory regions and checker callbacks are available inside the analyzer but unused by current refinement strategies. SemWeaver is a plug-in refinement layer that extracts patch-relevant analyzer evidence and assembles it with source context and validation feedback into a structured, provenance-labelled bundle. An iterative editing loop refines the checker on both the vulnerable and fixed revisions, guarded by a retention gate that accepts warning recovery or false-positive reduction while preventing over-pruning. We evaluate SemWeaver on 39 KNighter-derived CSA checkers from Linux kernel fixes. With analyzer evidence, refinement reaches 23 warning-positive vulnerable revisions (vs. 15 for KNighter's loop) and raises version-level $F_1$ from 0.448 to 0.575. An ablation removing internal evidence recovers fewer silent cases and produces different checker edits, indicating that the two evidence configurations address complementary recognition gaps. Additional experiments cover three editor models on a 12-subject subset, a CodeQL query refinement, and controlled source-form variations. A qualitative follow-up on all 11 non-tied comparisons traces each warning-count change to a specific checker repair, such as recognizing a lowered callback or unwrapping an expression cast.

1 Introduction

Static vulnerability checkers must encode the semantic conditions separating real bugs from safe code, but patch-generated checkers can match a code change without understanding why it is safe. SemWeaver adds analyzer-internal evidence to checker refinement and evaluates the approach on Linux CSA checkers.

1 Introduction

  • A patch shows what changed, not why the change makes code safe, so a checker can compile yet report both vulnerable and fixed revisions or detect neither.Suppressing spurious reports can also remove the true-bug trigger, a failure mode termed over-pruning.Source: §1 · p. 2
  • Existing refinement loops return compilation errors and validation reports, but these symptoms do not explain what the checker misunderstands about the code.Missed lowered callbacks, casts, or memory regions can underlie a silent warning.Source: §1 · p. 2
  • Analyzers already expose semantic information—including control-flow graphs, symbolic values, memory regions, and callbacks—that can help explain checker failures.The challenge is selecting relevant evidence and relating it to the patch and checker.Source: §1 · p. 2
  • SemWeaver bundles patch-relevant analyzer views with source context and validation feedback, then iteratively edits and tests the checker on both revisions under a retention gate.The gate accepts warning recovery or fixed-side report reduction while rejecting edits that lose an existing vulnerable-side warning.Source: §1 · p. 2
  • SemWeaver’s native refinement recovers eight initially silent vulnerable revisions, while the no-internal-evidence ablation recovers six and repairs some different cases.The complementary repairs suggest that internal and source-derived evidence address different recognition gaps.Source: §1 · p. 3

2 Background

The background frames vulnerability checking as encoding semantic conditions through a framework’s program representations. Patch-driven generation supplies a before-and-after target, while analyzer internals offer additional evidence about what a checker observes.

2.1 Static Vulnerability Checkers

  • Static vulnerability checkers encode security-relevant conditions as framework rules or queries, using representations such as path-sensitive CSA states or relational CodeQL facts.Effective checkers must express the distinction between a real bug and a safe use through framework callbacks, values, and object representations.Source: §2.1 · p. 3

2.2 LLM-Based Patch-Driven Checker Generation

  • Patch-driven generation treats the vulnerable and fixed revisions as a paired specification: the checker should warn on the former and stay silent on the latter.KNighter already refines generated CSA checkers using compiler and report feedback, but its no-report branch has no report-driven edit.Source: §2.2 · p. 3
  • A patch describes one repair rather than every safe and unsafe use, so SemWeaver adds analyzer context and recovery attempts for initially silent checkers.The added context supplements the patch-driven setting rather than replacing it.Source: §2.2 · p. 3

2.3 Analyzer-Internal Intermediate Results

  • CSA exposes symbolic states, region bindings, control-flow and call-graph representations, while CodeQL materializes queryable program relations.These backend-produced representations can reveal how a checker interacts with the program, and their meaning depends on their source and interface.Source: §2.3 · pp. 3–4

3 Motivating Example

The motivating off-by-one example shows how a checker can recognize a look-ahead access yet miss that the loop bound determines whether it exceeds array capacity. Analyzer evidence supports a refinement that checks the bound-to-capacity relation.

3 Motivating Example

  • The motivating case supplies a concrete example of the gap between recognizing a look-ahead access and understanding its relation to array capacity.The array capacity is abbreviated as N in Figure 1.Source: §3 · p. 4, Figure 1 · p. 4
  • Under the vulnerable bound i < N, the loop can reach i = N − 1 and read links[i + 1] at index N, beyond the array’s capacity N.The patch changes the bound to i < N − 1, limiting the maximum look-ahead index to N − 1.Source: §3 · p. 4
  • The baseline checker checks branch guards but overlooks protection from the loop bound, so it reports both revisions and remains at 1/1 after KNighter refinement.The fixed revision still contains the look-ahead expression, and the report-driven loop finds no actionable feedback in this case.Source: §3 · p. 4
  • SemWeaver adds a capacity-aware condition that compares the maximum look-ahead index with the array extent, warning for the vulnerable bound and staying silent for the fixed bound.The condition relies on type and index relations rather than the patched function name, a particular capacity constant, or statement adjacency; the candidate passes replay at 1/0.Source: §3 · p. 5

4 Approach

SemWeaver refines generated vulnerability detectors by combining patch-relevant analyzer evidence with source context and validation feedback, then editing and validating the detector on both revisions.

4 Approach

  • SemWeaver refines an upstream-generated CSA checker or CodeQL query to recover vulnerable-side warnings or reduce fixed-side reports.Source: §4 · p. 5

4.1 Evidence Sources and Roles

  • Analyzer-internal evidence captures intermediate analysis results—such as control-flow, symbolic-value, region-binding, and callback information—not recoverable from source code alone.Collectors also gather patch and bounded source context, plus paired-validation feedback; CSA observations are from analysis, not concrete program executions.Source: §4.1 · p. 6

4.2 Evidence Selection

  • Rule-guided preflight selects patch-relevant evidence roles and bounds collection rather than passing large, heterogeneous analyzer outputs wholesale.For the CSA comparison, the visible evidence view is frozen before editing and includes up to four function-local CFG witness cards.Source: §4.2 · p. 7

4.3 Evidence Bundle Schema

  • The EvidenceBundle normalizes collected facts into typed records with provenance, scope, source location, payload, and optional compact semantic slices.Provenance distinguishes analyzer-internal, analyzer-output, and source-derived records; missing evidence is retained explicitly.Source: §4.1 · p. 6, §4.3 · p. 8

4.4 Paired-Warning Refinement

  • The editor receives bounded evidence and validation feedback, while a retention gate keeps a compiling candidate that recovers a vulnerable warning or reduces fixed-side reports without losing vulnerable-side warning presence.A new backend uses an adapter for evidence and provenance mapping, execution, and paired validation; broader validation and new native-evidence semantics require additional work.Source: §4.4 · pp. 8–9

5 Evaluation

The evaluation compares SemWeaver with KNighter and examines evidence variants, cross-backend feasibility, and editor configurations. It uses paired validation and warning-presence metrics within patch-related scope.

5 Evaluation

  • The evaluation measures recovery of vulnerable-side warnings and reduction of fixed-side reports, focusing on patch-local and patch-related-file behavior rather than whole-kernel bug discovery.Source: §5 · p. 9

5.1 Research Questions

  • Four research questions cover KNighter comparison, CodeQL feasibility and cost, native versus no-internal evidence, and sensitivity to editor configuration.Source: §5.1 · p. 9

5.2 Experimental Setup

  • The main CSA experiment selects 39 fixed, non-random KNighter-derived checkers; 15 initially warn on both revisions and 24 are silent on both.All 39 enter KNighter refinement and both SemWeaver variants; the study also reuses a 12-subject subset for further experiments.Source: §5.2 · pp. 9–10

5.3 Metrics

  • Patch-discriminating success requires a vulnerable-side warning and fixed-side silence, while report reduction must preserve vulnerable-side warning presence.Precision, recall, and F1 classify each vulnerable revision as positive and its fixed revision as negative by warning presence, covering 78 patch-version decisions; these metrics do not adjudicate individual bug sites.Source: §5.3 · p. 11

6 Results

Across 39 Linux CSA checkers, SemWeaver recovers more silent vulnerable revisions than KNighter, with a trade-off in fixed-side reports. Additional experiments show cross-backend feasibility, complementary evidence effects, decode variation, and checker repairs that warning counts alone cannot characterize.

6.1 RQ1: KNighter-Derived Checkers

  • Native SemWeaver recovers eight of 24 silent starts while retaining 15 initially warning-positive revisions; KNighter recovers none under its no-report policy.Table 3 covers all 39 subjects.Source: §6.1 · p. 12, Table 3 · p. 12
  • Native produces 123 fixed-side reports, versus 68 for no-internal and 69 for KNighter, trading broader recovery for lower precision.Native precision is 0.561, versus 0.618 for no-internal.Source: §6.1 · p. 12
  • On the 15 initially noisy subjects, native reduces fixed reports by 25 (36.2%) against KNighter and 24 (35.3%) against no-internal, while all methods share binary fixed positivity and F1.The reduction is concentrated in G24, whose fixed reports fall from 27 to 2.Source: §6.1 · p. 12, Table 3 · p. 12

6.2 RQ2: Cross-Backend Feasibility

  • Two of three CodeQL decodes refine an archived query from 0/0 to 8/0 vulnerable/fixed rows; the successful edit corrects relative-path matching while retaining other filters.Paired replay measures median query-plus-decode time of 12.57 s initially and 36.51 s after refinement; developer labor and cold installation were not measured.Source: §6.2 · p. 12, Table 4 · p. 13

6.3 RQ3: Contribution of Analyzer Evidence

  • Across 39 subjects, native has six comparative benefits and five disadvantages, while edits show distinct repairs by both evidence variants, including lowered-cleanup recognition and symbolic-region handling.Native adds the lowered cleanup name __free_kfree in G19, changing counts to 15/0 while no-internal remains at 0/0; no-internal recovers symbolic-region handling in G23.Source: §6.3 · p. 13

6.4 RQ4: Sensitivity Across Models

  • On the 12-subject, three-decode model subset, all configurations retain five initial vulnerable warning signals, but results vary across decodes; source replay also exposes sensitivity to equivalent guard rewrites.GLM-5.3-Flash has the highest mean F1, while GLM-5.3 has fewer fixed-side reports in all three decodes; local renames and inert no-ops leave counts unchanged in the three replayed subjects.Source: §6.4 · p. 14, Table 5 · p. 14

6.5 Qualitative Findings

  • The audit of 11 non-tied comparisons links warning-count differences to specific repairs, including loop recognition, lowered cleanup names, symbol lookup, sink recognition, and destination binding.The audit distinguishes source-supported and conditional target reports from unsupported ones without changing primary scores.Source: §6.5 · pp. 14–15
  • Report counts can obscure target relevance: G24 removes 25 reports per side, but its two survivors concern owner destruction outside the intended surviving-owner lifecycle, and some removed target reports are conditional.The qualitative analyses also show that recovered target-line presence can coexist with negative-side noise and that correcting a recognized callback or expression may leave the object, guard, or owning scope wrong.Source: §6.5 · pp. 15–16

7 Discussion

SemWeaver’s improvements are local to the motivating patch: recognizing analyzer representations is an earlier repair task, while warning correctness still depends on scope and path conditions. Its evidence bundles and backend workflow support different repair settings, but broader transfer remains open.

Scope of improvement

  • SemWeaver improves patch-motivated checkers by helping them recognize relevant analyzer representations, while scope and path conditions still determine whether warnings are justified.G19 needs a lowered cleanup callback; G25 needs an expression wrapper unwrapped.Source: §7 · p. 16

Evidence as context compression

  • Provenance-tagged summaries compress patch-relevant source and analyzer facts with validation feedback; native and no-internal arms repair different representation problems.Examples include G19/G25 versus G01/G23/G38.Source: §7 · p. 16

Backend interfaces

  • The paired editing and validation workflow also operates on a CodeQL query, but broader transfer and measured developer effort remain open.That case retains source-derived context and inherited patch-local filters.Source: §7 · p. 16

Failure modes

  • Fewer fixed-side reports do not guarantee correct warnings: examples retain reports outside the intended lifecycle or on explicitly non-NULL paths.A broader cleanup matcher can also report at a callee return while the caller’s variable remains live.Source: §7 · p. 16

Generalization

  • The motivating example survives tested renamings, capacities, and equivalent guards, but a replay of three subjects shows sensitivity to an equivalent guard rewrite.That rewrite silences G01 on both versions and increases G08’s fixed-side reports; transfer to unseen projects and vulnerability instances requires separate evaluation.Source: §7 · p. 16

8 Threats to Validity

The validity discussion limits what the results establish: evidence components are bundled, warning-based metrics are not target-level precision, and the evaluated subjects and auxiliary experiments have bounded scope. Repeated decodes and transformed fixtures are stability checks, not additional independent bugs.

Internal

  • Native context bundles multiple components, limiting attribution to any single evidence type; three independent decodes per configuration partially address model-alias and seed variation.Persisted outputs, hashes, failed attempts, and unavailable observations support reproducibility of the fully automatic scored edits.Source: §8 · p. 17

Construct

  • Version-level warning presence measures paired warnings within patch-related scope, not target-level precision or confirmed false-positive bugs.Final-line matching cannot distinguish objects or feasible from infeasible paths on the same line.Source: §8 · p. 17

External

  • The 39 subjects come from one generator’s Linux artifacts, limiting checker-style and vulnerability-type diversity.The CodeQL experiment covers one patch-local example, and the controlled source replay covers three subjects rather than natural evolution.Source: §8 · p. 17

Conclusion

  • Repeated decodes and transformed fixtures are correlated observations, not additional independent bugs, and do not estimate prevalence across arbitrary checkers.The findings characterize local repairs and failures within the studied cohort and do not establish when internal evidence will help.Source: Conclusion · p. 17

9 Related Work

SemWeaver builds on static-analysis representations, patch evidence, and feedback-guided repair, but focuses on editing existing generated checkers using analyzer context. Related work also uses analyzer context for other tasks, whereas this study examines checker changes and their remaining semantic errors.

Static Analysis and Checker Languages

  • Static-analysis frameworks expose paths, guards, states, and diagnostics, while query languages and semantic patching provide complementary representations; SemWeaver adds no analyzer or query language.It treats generated checker code as the artifact to refine.Source: §9 · p. 17

Vulnerability Detection and Patch Evidence

  • Prior vulnerability detection uses taint, data-flow, graph, AST, and learned representations, while clone- and patch-driven systems use historical fixes as compact witnesses.These approaches motivate patches as a way to scope refinement.Source: §9 · pp. 17–18

LLM-Based Checker Synthesis

  • KNighter is the closest workflow, while SemWeaver adds bounded compiler and checker observations and attempts warning recovery on silent starts.Other cited systems synthesize tools or queries using execution feedback and structured tool access.Source: §9 · p. 18

Feedback-Guided Repair

  • Feedback-guided repair work raises concerns about weak validation proxies and overfitting; this study examines when warnings return at the expected line but retain the wrong object, scope, or path.The follow-up examines those consequences for edited static checkers.Source: §9 · p. 18

Analyzer Context and Alert Triage

  • Prior studies use analyzer context for specifications, taint reasoning, report feasibility, or adjudication, whereas SemWeaver studies its effects on existing checker implementations and their reports.It retains both useful edits and remaining semantic errors.Source: §9 · p. 18

10 Conclusion

The conclusion is that analyzer-internal evidence can guide edits to generated checkers, with evaluation showing recovered silent vulnerable revisions and improved paired discrimination. The qualitative follow-up links warning-count changes to specific repairs, while target-level semantic verification remains future work.

10 Conclusion

  • SemWeaver feeds analyzer-internal evidence into the editing loop, addressing the gap left when existing refinement loops leave analyzer-computed semantic information unused.The evidence helps the model understand why a checker fails rather than only that it fails.Source: §10 · p. 18
  • Evaluation on KNighter-derived Linux checkers reports recovery of previously silent vulnerable revisions and improved paired discrimination over report-driven refinement alone.The qualitative follow-up traces warning-count changes to repairs such as recognizing lowered callbacks and unwrapping expression casts.Source: §10 · p. 18
  • Verifying target-level semantic correctness beyond warning presence remains a direction for future work.Source: §10 · p. 18

Related papers

Loading 2610.34928v1…