PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Evolving Static Analysis via LLM Pseudo-Labeling

Rui Melo, Andre Duarte, Aditya Oke, Shubham Gandhi, Nachiket Kotalwar, Rui Abreu, Lei Li, Carolyn Rose

arXiv:2610.30184v1cs.CRcs.AIcs.SE

TL;DR

Static-analysis rules must keep pace with changing frameworks, APIs, and coding idioms. SESAA evolves readable JOERN queries from scratch using a second LLM’s feedback, without exposing ground-truth labels to the agent. It achieves a mean F1 of 0.742 on the OWASP Benchmark for Python and catches 10 of 19 later CVEs in a real-world pilot.

  • Problem

    Static-analysis rules require manual revision as frameworks and coding idioms change, while existing LLM approaches either report bugs directly or derive rules from already documented vulnerabilities.

    Source: §1 · p. 1

  • Method

    SESAA starts with no rules and iteratively evolves JOERN queries using a second LLM critic’s pseudo-labels, without access to ground-truth labels.

    Source: §1 · p. 2

  • Results

    Mean F1 0.742 across 8 runs on the OWASP Benchmark for Python, above five tools’ defaults and comparable to label-tuned CodeQL; the real-world pilot caught 10 of 19 CVEs.

    Source: §1 · p. 3, §5.5 · p. 17, §9 · p. 20

  • Takeaways & Limitations

    The output is a set of ordinary, readable queries that engineers can inspect, edit, and run independently of the agent.

    Source: §1 · p. 2

    The stronger critic and the agent’s direct corpus exploration confound attribution of gains to pseudo-labeling; the S2 ablation cannot separate their effects.

    Source: §8 · p. 19

Abstract

from arXiv · show

Security bugs often slip past static analysis not because they are impossible to detect, but because detection rules have not yet been written for the code in front of us. New frameworks, APIs, and coding idioms continually create such gaps, leaving developers to discover failures and manually encode new rules. We present SESAA, a framework in which an Large Language Model (LLM) agent starts from no rules and iteratively evolves a set of readable and auditable joern queries. The agent receives feedback only from a second LLM critic, which inspects flagged and unflagged code to identify likely false alarms and suspected misses, without access to ground-truth labels. The agent then uses this feedback to revise its queries. On the OWASP Benchmark for Python, SESAA reaches a mean F1 of 0.742 across 8 independent runs, well above the default configurations of five widely used tools and comparable to the strongest of them once it is tuned with access to ground-truth labels that SESAA never sees. In a pilot on real-world Python projects, queries induced from code available before a vulnerability was introduced detect 10 of 19 subsequent CVEs, including 9 detected by rules written for the same repository. The resulting detector is a set of executable, human-readable queries that engineers can inspect, modify, and run independently. These results suggest that LLM agents can turn static-analysis rule maintenance from a manually authored process into an iterative process of automated discovery and repair.

1. Introduction

Static-analysis rules require expert maintenance as code and APIs change, while narrow and broad rules respectively miss patterns or generate false positives. SESAA starts with no rules and iteratively evolves JOERN queries using a critic’s pseudo-labels, without ground-truth labels; it reaches a mean F1 of 0.742 across eight runs on the OWASP Benchmark for Python.

1. Introduction

  • Static-analysis queries are hand-written by security experts, who must diagnose missed vulnerabilities and false positives and revise rules as frameworks and codebases change.Narrow rules miss new patterns, while broad rules produce false positives.Source: §1 · pp. 1–2
  • Earlier LLM approaches either report bugs directly, leaving little reusable for later scans, or derive rules from documented vulnerabilities.SESAA instead asks whether an agent can observe rule failures and rewrite its own rules.Source: §1 · p. 2
  • SESAA starts with no rules and iteratively builds JOERN queries by editing and running them over unlabeled code.A second LLM critic judges reported findings as likely bugs or false alarms and identifies suspected misses; its judgments guide query revisions.Source: §1 · p. 2, Figure 2 · p. 3
  • Mean F1 0.742 across 8 independent runs on the OWASP Benchmark for Python, above five tools’ default configurations and comparable to label-tuned CodeQL.The agent never sees the withheld labels used to tune CodeQL.Source: §1 · p. 3

2. System Architecture

SESAA pairs an agent that writes and repairs JOERN queries with a critic that assesses findings and suspected misses without ground-truth labels. Its fitness combines critic-estimated precision with coverage signals to guide iterative query evolution.

2. System Architecture

  • The ReAct agent repeatedly mutates JOERN queries, evaluates them on a code corpus, and retains the best-scoring suite; the critic supplies diagnoses without ground-truth labels.The resulting suite is ordinary code engineers can review and run independently of the agent.Source: §2 · p. 3
  • JOERN’s Code Property Graph combines syntax, control flow, and data/control dependencies, allowing queries to express taint flows in editable Scala.The agent alternates reasoning with tool use and edits based on critic diagnoses; the best-fitness suite is retained to limit the cost of harmful mutations.Source: §2 · pp. 3–4

2.1. Pseudo-Labeled Fitness

  • Fitness uses the critic’s report-level verdicts to estimate precision and vulnerability-class coverage as a proxy for recall, without ground-truth labels.The geometric mean balances these proxies; type coverage measures breadth and saturates once each target class has one report.Source: §2.1 · pp. 4–6
  • False-positive tickets let the agent remove a likely false alarm or contest the critic’s judgment, while false-negative tickets prompt it to address suspected missed vulnerabilities.Coverage is the fraction of critic-identified misses that have been resolved, so the signal is limited to the reviewed portion of the corpus.Source: §2.1 · pp. 5–6

3. Illustrative Example

The XPath-injection example illustrates how the critic’s tickets can prompt a semantic query repair. The agent excludes safe escaped and parameterized cases rather than suppressing the query altogether.

3. Illustrative Example

  • The XPath rule’s coarse taint logic flagged benign calls that escaped values or passed them as bound parameters.The critic opened false-positive tickets for these cases, judging them from surrounding code rather than labels.Source: §3 · p. 6
  • After the repair, the XPath group’s estimated precision and the suite’s fitness rose, so the revised suite was retained.A repair can also proceed through contesting a ticket, which the critic may reverse after reconsideration.Source: §3 · pp. 6–7
  • The agent added exclusions for XML-escaped arguments and named-parameter calls, preserving the query while distinguishing these safe cases.The edit records why the constructs are safe in an inline comment.Source: §3 · p. 6, Figure 3 · p. 7

4. Methodology

The methodology evaluates SESAA on the OWASP Benchmark and real-world Python projects, with repeated runs, ablations, and separate protocols for mechanism analysis and introduction-time detection. Baselines include default tools and label-tuned configurations, while ground-truth labels remain withheld from SESAA.

4.1. Evaluation Corpus

  • The OWASP Benchmark for Python has 1,230 labeled test cases across 14 vulnerability categories; the real-world extension uses pre- and post-vulnerability snapshots.The benchmark contains 452 vulnerable and 778 safe cases.Source: §4.1 · p. 7

4.2. Evolution Configuration

  • SESAA starts from stripped detection patterns and is evaluated against default and label-tuned tool configurations; baseline tuning selects the best of nine runs with access to withheld labels.The evaluation uses a Claude Sonnet 5 agent and Claude Opus 4.8 critic; SESAA itself never receives the withheld labels.Source: §4.2 · pp. 7–8, §4.3 · p. 8

4.4. Experimental Settings

  • S1 repeats the main configuration across eight seeds, S2–S4 remove components in five-seed ablations, and S5 tests proactive detection on real-world projects.S2 removes critic feedback, S3 removes FP tickets and contests, and S4 removes FN tickets; S5 uses a different fitness proxy on clean pre-introduction snapshots.Source: §4.4 · pp. 8–9

4.5. Evaluation Metrics

  • Benchmark precision, recall, F1, and Youden’s J are computed per finding against withheld labels that are never visible to the agent.The introduction-time benchmark instead measures whether a suite detects a vulnerability at its introducing commit and whether tickets correct resulting false positives.Source: §4.5 · p. 9

4.6. Analyzing the Ticket Mechanism

  • The ticket analysis checks each opened ticket and repair against withheld labels to assess critic judgments and whether repairs remove false positives or destroy real detections.The protocol separately tests whether tickets identify genuine false positives, repairs respond selectively to tickets, and precision estimates are biased.Source: §4.6 · p. 9, §4.7 · p. 9

4.7. Statistical Tests

  • Mechanism tests use exact enumeration, Fisher’s exact test, a hypergeometric test, Wilcoxon signed-rank tests, and correlation tests; ablation F1 comparisons use Welch’s t-test.A Bayesian effect-size supplement addresses the five-seed ablations, where p-values above 0.05 may reflect low power.Source: §4.7 · p. 10

4.8. Introduction-Time Detection Dataset

  • SZZ-style blame attribution traces CVE fix lines to introducing commits, resolving 36 files across 19 CVEs and 12 Python projects.Cases are matched at function granularity when possible; otherwise, matching falls back to line level.Source: §4.8 · p. 10
  • For S5, detectors evolve on clean pre-introduction trees and are tested against code added at the introducing commit, with a hit requiring a matching CVE finding.The proactive fitness combines coverage of security-sensitive patterns with the critic-judged false-positive rate on the clean corpus.Source: §4.8 · p. 11

5. Results

Across the benchmark and pilot, evolved queries matched the strongest tuned baseline on average and detected vulnerabilities introduced later in real-world projects. Ticket and agent behavior was consistent with selective query repair, although the pilot and ablation results have important scope and power limits.

5.1. RQ1: Can static-analysis queries evolve without ground-truth labels, from scratch?

  • Removing the critic, FP tickets, or FN tickets moved outcomes in predicted directions, but no ablation established an F1 effect; the pooled tests remained underpowered.The ablations used five seeds each.Source: §5.1 · p. 12
  • S1 averaged F1 0.742 ± 0.084 across eight runs; 6/8 seeds beat tuned CodeQL’s 0.70, though the confidence interval overlapped CodeQL’s score.Figure 4 reports ground-truth F1 and Youden’s J by setting; the two J calculations are not strictly comparable because evolved-detector true negatives are approximated.Source: §5.1 · p. 12, Figure 4 · p. 13

5.2. RQ2: Does the ticket mechanism behave as designed?

  • Of 665 solved findings, 568 were genuine false positives and 97 were real vulnerabilities removed by mistake; selective repairs beat ignoring or blindly clearing tickets in every seed.The outcomes were scored against withheld labels across all eight S1 runs.Source: §5.2 · pp. 13–14, Figure 6 · p. 14
  • Across eight seeds, tickets were enriched for genuine false positives, agent repairs followed that signal above chance, and the critic’s precision estimates were conservative.The critic’s tickets point disproportionately at genuine false positives, and its calibration error consistently underestimates detector precision.Source: §5.2 · p. 14

5.3. Agent Behaviour

  • Across 711 tool calls in the committed generation, the agent read and explored before editing, then evaluated; editing existing detectors was more common than writing new ones.The reported counts were 85 calls editing existing detectors and 37 writing new ones.Source: §5.3 · p. 15

5.4. RQ3: Do static-analysis queries evolved without ground-truth labels reach the quality of mature, hand-maintained SAST tools?

  • 0.742 ± 0.084 mean F1 put the evolved detector ahead of tuned CodeQL’s 0.70 and every other panel tool, though the seed interval overlapped CodeQL’s score.The comparison is driven mainly by precision rather than recall.Source: §5.4 · pp. 15–16, Figure 8 · p. 16

5.5. RQ4 (pilot study): Can agents write proactive security rules on clean code that catch vulnerabilities at introduction time?

  • The pilot tested queries evolved on pre-introduction repository snapshots against later introducing commits; repository size and complexity correlated with lower scores in this sample, but were not tested as causes.The lowest-scoring repositories included some of the largest and most complex, while the smallest repository was detected perfectly.Source: §5.5 · p. 17
  • 10 of 19 introduction-time CVEs were caught across the pilot, including 9 by detectors evolved on the same repository and one cross-repository transfer.Catch rates varied heavily by repository, and the fitness signal designed for this setting was inert in most runs.Source: §5.5 · p. 17

6. Related Work

SESAA retains static analysis’s readable, executable queries while evolving them with critic pseudo-labels. This differs from learned classifiers and LLM rule-generation systems whose outputs or supervision take other forms.

Static analysis

  • SESAA keeps deterministic, explainable static-analysis queries as its output, but rewrites them continuously using critic pseudo-labels rather than human maintenance.Source: §6 · p. 17

Vulnerability classifiers

  • Learned vulnerability classifiers can be accurate, but their learned behavior remains in model weights rather than an independently readable, editable, runnable query [64],,,,.Source: §6 · p. 17

LLM agents

  • Unlike agents that perform vulnerability analysis anew on each repository, SESAA directs a second model’s signal at persistent queries, so the analyzer carries over [36], [29].Source: §6 · p. 18

LLM-generated analysis rules

  • Prior LLM rule-generation systems use CVE reports, patches, or specifications as supervision, while SESAA evolves a detector from critic pseudo-labels over unlabeled code [52], [28], [37].Source: §6 · p. 18

Self-improving agents

  • Self-improving agents refine prompts, strategies, or policies, leaving improvements embedded in the model pipeline and subject to the stability–plasticity dilemma [47], [39], [61], [57], [1],.Source: §6 · p. 18

Evolutionary search

  • SESAA applies evolutionary search to analysis logic itself, selecting executable query changes with a critic-pseudo-labeled signal that needs no ground truth, [14] [21],, [35], [10], [39] [47], [1].Source: §6 · p. 18

7. Discussion

The critic’s tickets identify enriched sets of likely false positives and suspected misses, and the agent acts on that feedback. The approach returns reviewable queries, but the evidence has limits: tickets are imperfect, testing does not establish effectiveness on unlabeled production code, and the archive offers only flat, read-only memory.

Mechanism and ablations

  • Tickets are enriched for genuine false positives, and the agent repairs ticketed findings more often than chance; ablations point in the same direction but do not establish that the mechanism drives F1.Removing the critic trades precision for recall; a larger campaign would be needed to pin down the effect size.Source: §7 · p. 18

Ticket reliability

  • Tickets are not accurate labels: 97 of 665 solved findings were real vulnerabilities removed by mistake, and a pooled 61% of FN tickets point at genuine misses.The contest step can challenge a wrong verdict, but the current setup relies on a single critic; independent critics are proposed as a next step.Source: §7 · pp. 18–19

Scope of evidence

  • The tested evidence is a labeled benchmark and a small pilot, so effectiveness on unlabeled production code has not been shown.Source: §7 · p. 19

Practical use

  • The returned query suite has no model dependency for later scans and can be reviewed, edited, kept, or reverted by an engineer.Runs consume substantial model tokens, but that cost is paid during the maintenance loop rather than every scan.Source: §7 · p. 19

Memory

  • The agent’s only cross-generation memory is a flat, read-only archive; structured memory could let it reuse lessons instead of re-deriving them after repairs.Source: §7 · p. 19

8. Threats to Validity

The authors identify internal, external, construct, and conclusion validity threats that bound what their evidence supports. These include uncertainty about the critic’s contribution, transfer beyond the evaluated setting, the proxy fitness, and run-to-run results.

Internal validity

  • The S2 ablation cannot separate gains from the critic’s signal from gains from the agent’s own corpus reading, and the critic’s stronger model may contribute prior knowledge.The authors have not tested same-model, local-context-only, or category-blind critic variants; SZZ attribution may also misplace the vulnerability-introducing commit in the pilot.Source: §8 · p. 19

External validity

  • Evaluation covers one language and model pair, and a synthetic benchmark; the pilot covers 12 projects and 19 CVEs, so broader transfer is not established.JOERN’s RAM-bound CPG construction and an unstudied token budget also leave scaling and cost-quality trade-offs unresolved.Source: §8 · p. 19

Construct validity

  • The pseudo-labeled fitness is only a proxy for ground-truth F1, while pretrained models and JOERN’s primitives may make OWASP patterns recognizable to the search.The agent lacks the answer key, but the taxonomy and surface patterns are not entirely unfamiliar; the evolved detector’s Youden’s J also uses an approximate TN count unlike the baselines’ J.Source: §8 · pp. 19–20

Conclusion validity

  • Ground-truth F1 varies across stochastic runs, and five-seed ablations bound aggregate effects without showing that mechanisms are inert.A conventionally powered campaign would need roughly ten times as many seeds; ablation tests are uncorrected for multiplicity, and F1 is the sole primary metric.Source: §8 · p. 20

9. Conclusion

SESAA evolves queries without ground-truth labels and produces competitive benchmark results, while pilot queries built from clean code detect some later vulnerabilities. The authors conclude that the critic provides an imperfect signal and that the findings remain bounded by the stated limitations.

Conclusion

  • 0.742 mean F1 across 8 independent runs on the OWASP Benchmark for Python exceeds five widely used tools’ defaults and is in the same range as label-tuned CodeQL.The agent never receives the labels used to tune CodeQL.Source: §9 · p. 20
  • 10 of 19 vulnerabilities were caught at introduction by queries evolved on clean code alone.The queries were developed from code available before the vulnerabilities were introduced.Source: §9 · p. 20
  • The critic’s pseudo-labels provide a real but imperfect signal: tickets are enriched for false positives and repairs track them above chance, though some repairs remove real vulnerabilities.Removing the critic or either ticket type lowers mean F1 in the direction predicted by the design.Source: §9 · p. 20

Generative AI Disclosure

The authors disclose generative AI assistance with implementation, analysis scripts, manuscript drafting, editing, and literature research.

Generative AI Disclosure

Aknowledgments

The work was conducted as part of the Amazon Nova AI Challenge, which provided funding and AWS compute credits; the authors state that key research decisions were independent.

Aknowledgments

  • The Amazon Nova AI Challenge provided funding to participating university teams.Source: Aknowledgments · p. 21
  • The challenge also provided AWS compute credits to participating university teams.Source: Aknowledgments · p. 21
  • The authors independently made the study-design, model-selection, analysis, and publication decisions.Source: Aknowledgments · p. 21

A. Agent Tools

Each generation runs in a constrained ReAct loop with tools for reasoning and action, while keeping withheld labels and the critic implementation inaccessible. A side-effect-free reasoning checkpoint records the intended mutation before the agent acts.

A. Agent Tools

  • The ReAct loop runs in a sandboxed Docker container, with /repo mutable, /corpus read-only, and withheld labels and critic implementation inaccessible.The agent has six tools, or seven when the FP-ticket mechanism is enabled.Source: A. Agent Tools · p. 28
  • The loop has no other side-effecting capabilities beyond its specified tools.Source: A. Agent Tools · p. 28
  • Before acting, the agent records which false negatives to target and what mutation to attempt in a reasoning checkpoint that changes no state.Source: A. Agent Tools · p. 28

bash(command)

The appendices document SESAA’s agent workflow, scoring guidance, detector-writing constraints, and prompt templates. They describe how the agent inspects a corpus, implements and tests Joern detectors, and uses critic feedback and LLM-based precision estimates to revise candidates across generations.

bash(command)

  • The tools separate corpus analysis from committing: analysis returns per-query findings and critic feedback, while submission commits one candidate and ends the generation.The agent can delete a query that prevents analysis from compiling and inspect archived candidates and critic evaluations across generations.Source: bash(command) · p. 28

B. Prompts

  • The appendix reproduces four prompts defining agent behavior, critic feedback, and the critic’s precision-judge prompt, with template variables filled at runtime.Source: B. Prompts · p. 28

B.1. System Prompt

  • The system prompt directs the agent to measure detector quality, refine queries using critic feedback, and submit only when ready.Analysis is repeatable and does not commit; the prompt also describes a ticket mechanism for challenging suspected false positives or tightening queries confirmed as false positives.Source: §B.1 · pp. 29–30

B.2. Generation Prompt

  • Fitness guidance combines LLM-judged precision per query group with density-independent coverage of distinct vulnerability classes, penalizing noisy or narrow detectors.The judge samples findings from each group, while covering more distinct classes raises the recall proxy regardless of how many files a detector flags.Source: §B.2 · pp. 30–31
  • The generation prompt instructs the agent to inspect code patterns, identify sources and sinks, and write taint-tracking queries using reachableByFlows.It also requires evaluating the detector suite with analyse_corpus().Source: §B.2 · pp. 31–32
  • The prompt has the agent explore corpus files, implement Joern detectors, and test them, with each run committing a new generation that seeds the next.It instructs the agent to identify vulnerabilities in the corpus before designing detectors using Joern syntax.Source: §B.2 · p. 32

B.3. Critic Feedback Template

  • After each evaluation, a structured critic summary is injected into the agent’s context to guide the next mutation.The template reports estimated precision, coverage, and critic fitness.Source: §B.3 · p. 33

B.4. Critic Precision Judge

  • The LLM precision judge samples findings from each detector group, estimates each group’s precision from the fraction judged exploitable, and averages across groups.Sampling varies with group size, while unchanged findings are cached to keep scoring deterministic across runs.Source: §B.4 · p. 33

Related papers

Loading 2610.30184v1…