CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

• Direct LLM analysis faces high inference costs and limited repository context, while checker synthesis requires analyzer integration, coordinated multi-file changes, compilation, defect detection, and false-positive control.Source: §1 · p. 2
• Existing benchmarks emphasize patch generation or vulnerability detection and provide limited interaction, while omitting checker-specific coverage, false-positive control, and analyzer compatibility.Source: §1 · pp. 2–3
• 32.30% mean Pass@1 across 21 model–harness configurations, with the best configuration reaching 45.33%, shows checker synthesis remains challenging.CheckerBench evaluates whether generated checkers work across vulnerable and fixed revisions.Source: §1 · p. 3
• CheckerBench provides 300 tasks from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems, while CheckerLab independently verifies checkers under a shared protocol.Verification covers vulnerable–fixed diagnostic behavior, patch localization, residual reports, and computational cost.Source: §1 · p. 3

2.1 Static Analysis and Checker Synthesis

• Checker-synthesis systems target specific analyzers or domains, motivating evaluation of general-purpose agents across analyzer environments.Examples include QLCoder’s tool-assisted CodeQL queries and KNighter’s Clang Static Analyzer checkers from Linux kernel patches.Source: §2.1 · pp. 3–4

2.2 Coding Agents for Software Engineering

• Coding agents support repository navigation, editing, tool execution, and feedback, but commonly produce patches or bug findings rather than validated reusable checkers.Checker synthesis additionally requires analyzer integration and validation on vulnerable and fixed revisions.Source: §2.2 · p. 4

2.3 Benchmarks for Repository-Level Code Agents

• Repository and vulnerability benchmarks score task completion, patches, or detection; CheckerBench instead verifies reusable checkers’ builds, vulnerable–fixed diagnostics, patch localization, and false positives.Source: §2.3 · p. 4

3 The CHECKERBENCH Benchmark

• Each task combines a repository’s vulnerable and fixed revisions and patch with its language, analyzer, pinned environment, and synthesis instruction.The shared skill library is available across runs, and agents must preserve both revisions.Source: §3 · p. 4

3.1 Benchmark Construction

• Construction links CVEs to fixing commits, reconstructs pinned analyzer environments, and packages reproducible compile and scan commands for stable differential analysis.CSA tasks use revision-compatible build snapshots and compilation databases; CodeQL tasks use pinned analysis environments.Source: §3.1 · p. 5, Figure 5 · p. 6
• Tasks expose patches, affected functions, build metadata, commands, and a minimal checker scaffold; a frozen verifier independently checks rebuilds, diagnostic contrast, patch relevance, and false-positive control.Source: §3.1 · p. 5
• Calibration permits up to three attempts with human diagnosis after failures, retaining only verifier-passing traces of at least 40 interaction steps; 300 accepted calibration trajectories are separate from main-evaluation repeats.Source: §3.1 · p. 5, §3.2 · p. 6
• Quality control checks artifact integrity and diagnostic semantics, removes duplicate revision pairs, and keeps related tasks together in a split.Reference checkers, calibration traces, and verifier details remain hidden from evaluation agents.Source: §3.1 · p. 6

3.2 Benchmark Statistics

• 300 tasks span 167 repositories, five language ecosystems, two analyzer environments, and 85 CWEs, with distributions shown by language, repository, and CWE.The tasks use CSA for C/C++ and CodeQL for the other four languages.Source: Figure 4 · p. 5, §3.2 · p. 6

Evaluation workflow

• Agents work in isolated task workspaces, use compilation and analysis feedback, then submit frozen checker source and recorded trajectories for independent verification.The verifier rebuilds checkers in a clean environment, checks vulnerable–fixed diagnostic contrast and patch relevance, and assesses false positives without checker edits.Source: §4 · pp. 6–7
• A run passes only after normal completion and satisfaction of all evaluation gates, including process quality, vulnerability semantics, and false-positive control.The frozen checker is evaluated independently; agent-reported success does not determine the verdict.Source: §4 · p. 7

Metrics

• Pass@1 measures tasks passing full evaluation after one independent synthesis run, while Build and Diff. SR separately measure compilation and differential verification.The default false-positive cutoff is 25%; interaction-efficiency metrics average turns, tool calls, and skill calls over successful trajectories only.Source: §4 · pp. 7–8
• Figure 6(c) plots Pass@1 against average turns on successful runs, with harnesses encoded by color and models by shape.Efficiency averages include subagent activity and are computed over successful trajectories.Source: Figure 6 · p. 8, §4 · p. 8

5.1 Experimental Setup

• Experiments evaluate seven models with three agent harnesses over the same 300 tasks, using three independent repeats and a 50-minute active-generation budget.Runs use isolated sandboxes and compilation and analysis feedback; frozen checkers receive independent evaluation, with Claude Opus 5 judging false-positive triage, process quality, and vulnerability semantics.Source: §5.1 · p. 8

5.3 Comparison with Baselines

• 80.3% valid-checker rate for the skill-guided OpenCode method versus 67.2% for KNighter on 61 Linux-kernel tasks.Removing the skill library lowers the method’s rate to 29.5%.Source: §5.3 · p. 9, Table 3 · p. 9

5.5 Time-Budget Sensitivity

• 7.24% Pass@1 at 10 minutes, 26.05% at 30 minutes, and 32.30% at 50 minutes under retrospective cutoffs of existing trajectories.The final 10 minutes add 2.25 percentage points; performance under separately imposed shorter deadlines remains untested.Source: §5.5 · pp. 10–11

5.6 FP-Threshold Sensitivity

• 32.05% aggregate acceptance at a 10% false-positive threshold, 32.30% at 25%, and 32.83% at 50%, a net increase of 0.78 points.The largest per-configuration increase is 1.67 points for GLM-5.2 with Claude Code.Source: §5.6 · p. 11

5.7 Analysis of Checker Synthesis Failures

• Among 4,360 selected failed runs, leading failure categories are budget exhaustion (25.32%), inadequate synthesis process (22.96%), and inadequate semantic modeling (21.15%).Excess false positives account for 15.96% of failures; examples include implementations that replace planned path-sensitive analysis with AST matching that omits locking and control-flow conditions.Source: §5.7 · p. 11

5.8 Data Contamination and Prior Exposure

• Identifier-substitution paraphrases reduce exact CVE recall from 10.00% to 1.00% for Claude Opus 4.8 and from 8.00% to 1.00% for DeepSeek-V4-Pro, while Pass@1 drops only 2–3 points.The authors interpret this as suggesting that recognizing familiar vulnerabilities yields only limited gains under this intervention.Source: §5.8 · p. 11

6 Conclusion

• CheckerBench provides 300 executable tasks across five language ecosystems for evaluating long-horizon static-analysis checker synthesis.CheckerLab assesses diagnostic behavior on vulnerable and fixed revisions, patch localization, and false positives.Source: §6 · pp. 11–12
• 45.33% Pass@1 is the best configuration across seven models and three harnesses, leaving substantial room for improvement in the evaluated repositories and analyzer configurations.CheckerLab independently rebuilds submitted checkers and evaluates vulnerable–fixed diagnostic behavior, patch localization, and false positives.Source: §6 · pp. 11–12

A Extended Related Work

• The related-work audit assesses repository context, executable environments, harness support, long-horizon interaction, checker synthesis, semantic verification, and false-positive evaluation.Partial credit applies when a system supports only subsets of the specified capabilities or verification criteria.Source: A Extended Related Work · p. 18
• Related evaluations cover software construction, long-horizon progress, and binary reasoning, whereas CheckerBench evaluates reusable analyzer checkers with independent vulnerable–fixed testing.The checker-specific evaluation includes patch localization and false-positive control.Source: A Extended Related Work · p. 18

B.1 Source Collection and Provenance

• Candidates are retained when a CVE links to a fixing commit, retrievable parent revision, and patch exposing a reusable defect property, with provenance recorded for each instance.The source repositories span C/C++, Java, Python, JavaScript, and Go; candidates with unresolved provenance, unavailable source, incompatible terms, or no reusable static-analysis property are discarded.Source: §B.1 · pp. 18–19

B.2 Build Recovery and Executable Scope

• For CSA tasks, compilation databases filter changed files to analyzable translation units, and both revisions are reconstructed in isolated worktrees before differential scans.Pairs are rejected if either revision remains unreconstructable, affected units cannot be analyzed after bounded recovery, or stable differential analysis is unsupported.Source: §B.2 · p. 19

B.3 Task Packaging and Shared References

• Randomly named task workspaces expose the patch, affected functions, build and scan metadata, and a minimal checker template while withholding CVE and CWE labels.The shared library supplies harness-specific skills and read-only analyzer references, but not task-specific defect logic; agents must infer the mechanism and refine their checker using compiler and analyzer feedback.Source: §B.3 · p. 19

B.4 Human-Guided Pilot Rerolls

• Pilot tasks receive up to three isolated rollouts, with human reviewers diagnosing failures and revising prompt guidance while every checker is independently verified.Passing trajectories require at least 40 interaction steps for curation, and tasks with no passing checker after the third attempt are excluded.Source: §B.4 · pp. 19–20

C Quality-Control and Contamination Audits

• CSA admission requires a recomputed vulnerable–fixed contrast with a disappearing report mapped to a patch-modified function, plus broader mainline scanning for false-positive evidence.Artifact isolation keeps reference states, checker implementations, calibration trajectories, verifier thresholds, and adjudication records outside the agent-visible snapshot; deduplication and split assignment group connected tasks together.Source: C Quality-Control and Contamination… · p. 20

C.1 Temporal Coverage and Prior-Exposure Audit

• The temporal ledger distinguishes CVE publication from dated public events and patch or checker evidence; its unresolved first-public audit means publication dates alone cannot certify a task as previously unseen.For 48 tasks, the CVE record explicitly dates a public event before its own publication; the earliest documented event establishes availability by that date, not the absence of earlier sources.Source: §C.1 · p. 21
• CVE-2026-10722’s fix was merged a week before the CVE publication, illustrating why a CVE-only filter can overstate novelty evidence for models released between those dates.The authors retain both dates and require review of the earlier source, without inferring whether a model trained on it.Source: §C.1 · p. 21

C.2 Time-Window Evaluation Protocol

• Temporal comparisons use pinned model releases and audited task subsets, with windows merged to at least 20 distinct CVEs unless a smaller remainder is reported without a fitted trend.No common post-release subset exists for all seven models, and score declines are interpreted alongside task-composition differences rather than as proof of memorization.Source: §C.2 · p. 22

C.3 Paired Prior-Exposure Sensitivity Protocol

• A positive P–D effect together with reduced recognition supports sensitivity to identifying cues, but ordinary paraphrase effects and residual difficulty changes remain alternative explanations.The analysis treats equivalence as supported only when the entire 95% interval lies within the prespecified ±5-percentage-point margin.Source: §C.3 · p. 24

D.1 Repository-Level Task Counts

• Of 300 tasks across 167 repositories, 159 C/C++ tasks use CSA and 141 CodeQL tasks cover Java (30), Python (51), JavaScript (30), and Go (30).Table 6 lists repository-level task counts, target languages, frameworks, and shares.Source: §D.1 · p. 24
• Repository counts use normalized repository identifiers, while a task’s target language may differ from its repository’s dominant implementation language.Dataset prefixes are removed and identifiers compared case-insensitively; python-imaging/Pillow is merged with python-pillow/Pillow, while ImageMagick/ImageMagick and ImageMagick/ImageMagick6 remain separate.Source: §D.1 · p. 24
• 151 repositories contribute one task each, while the Linux kernel repository contributes 107 tasks (35.7%).Seven repositories contribute two tasks each, four contribute three, and four contribute four; shares use all 300 tasks as the denominator and are rounded to one decimal place.Source: §D.1 · p. 24
• JavaScript CodeQL entries include mde/ejs, moodle/moodle, and panva/jose, each listed with 1 task (0.3%).Source: §D.1 · p. 26
• Go CodeQL entries include mholt/archiver, ollama/ollama, and prometheus/client_golang, each listed with 1 task (0.3%).Source: §D.1 · p. 26
• C/C++ CSA entries include michaelforney/samurai, redis/redis, and uclouvain/openjpeg, each listed with 1 task (0.3%).Source: §D.1 · p. 26
• Java CodeQL entries include netplex/json-smart-v2, pytorch/serve, and xuxueli/xxl-job, each listed with 1 task (0.3%).Source: §D.1 · pp. 26–27

D.2 Programming Languages and Static Analysis Frameworks

• 159 CSA and 141 CodeQL tasks make up the 300-task inventory; Table 7 lists repository counts by target language and framework.Source: §D.2 · p. 27, Table 7 · p. 27

D.3 Complete CWE Label Distribution

• The 85-CWE inventory retains 82 weakness entries and three upstream categories; 286 tasks have explicit labels, while 14 are unlabeled.Source: §D.3 · p. 27, Table 8 · p. 27

D.4 Calibration Horizon and Difficulty Distributions

• Calibration trajectories have a median 60-step horizon and maximum 123; the minimum eligible horizon is 40 steps, and difficulty ranges from 0.414 to 0.846.Source: §D.4 · p. 29, Table 9 · p. 29

D.5 Pilot Trajectory Statistics

• Figure 11 summarizes one admitted pilot trajectory per selected task, including 13 whose checkers passed admission despite runs ending at the spending limit.Source: §D.5 · p. 30, Figure 11 · p. 30

D.6 CVE Publication-Time Distribution

• CVE-year counts use the 300 task records, so separate tasks sharing a CVE count separately; temporal-performance uncertainty groups those tasks together.Source: §D.6 · p. 31, Table 12 · p. 31

E.1 System Configurations and Reproducibility; E.2 Task Admission and Independent Repeats

• The evaluation uses 21 harness–model pairs, three repeats across the same 300 tasks, and 18,900 total runs; each repeat allows 50 minutes for iterative checker development.Source: §E.1 · p. 31, §E.2 · p. 32

E.3 Source and Evidence Freezing

• The ledger freezes each submitted source with its digest and trajectory, and later verification consumes that exact candidate rather than replacing it.Source: §E.3 · p. 32

E.4 Backend Execution; E.5 Frozen Mainline Scan and Single-Judge Decisions

• CSA checkers are rebuilt as analyzer plugins and CodeQL queries are independently compiled and executed; mainline warnings and quality judgments assess frozen candidates without follow-up edits.Source: §E.4 · p. 32, §E.5 · p. 32

E.8 Statistical Reporting

• FPpass pools judged alerts from passed checkers, while checker-weighted precision gives each alert-bearing checker equal weight; zero-alert checkers enter neither denominator.The reported metrics are equal-weighted means of repeat-level proportions, and judge labels are not human ground truth.Source: §E.6 · p. 33, §E.8 · p. 34

E.7 Failure Accounting, Recovery, and Aggregation

• Each harness–model Pass@1 averages three independent repeat rates equally; retries remain attached to their logical run, and unresolved infrastructure outcomes are reported separately from model failures.Efficiency summaries average passed-trajectory interaction turns, tool calls, and skill calls across repeats; incomplete repeat groups remain reported as incomplete coverage.Source: §E.7 · p. 34, §E.8 · p. 34

E.9 Human Validation of the LLM Judge

• 0.85 pooled Cohen’s κ was reported across 600 judge–human pairs, with 556 matching labels and 0.9267 exact agreement.Three annotators labeled all items independently, and majority vote defined the human reference.Source: §E.9 · pp. 35–36

F.5 Mainline False Positives and Final Verdict

• The main comparison requires a sampled false-positive rate of at most 25%, and only normally completed generations satisfying every required gate receive a final pass.The cutoff sensitivity analysis varies only that threshold while retaining saved warning samples, judge decisions, and other acceptance requirements.Source: §F.5 · pp. 37–38

G Representative CSA Checker Patterns

• SimpleStreamChecker tracks each resource as open or closed, coordinating acquisition, release, leak reporting, and ownership escape across callbacks.Its compact lifecycle state machine illustrates the coordination required for finite-state resource tracking.Source: G Representative CSA Checker Patterns · p. 39
• NullabilityChecker and MallocChecker illustrate complex designs that combine multiple analysis events and state traits, requiring choices about state keys, report suppression, and path evidence.These examples are intended to expose repository-level design decisions, not serve as templates for direct copying.Source: G Representative CSA Checker Patterns · p. 39
• Across the selected patterns, checkers construct path-sensitive reports from symbolic values and analysis nodes, while CheckerBench evaluates rebuilt checkers by vulnerable–fixed diagnostic contrast rather than reference-code matching.Reports may mark symbols and source ranges and include a visitor for path notes.Source: G Representative CSA Checker Patterns · p. 39

KNighter Comparison Details

• The comparison uses 61 Linux-kernel bug-fixing tasks and evaluates the OpenCode-based method, KNighter, and the method without skills with DeepSeek-V4-Pro under equal 50-minute active task budgets.The budget includes model responses, tool execution, and checker repair; KNighter’s internal coordination delays and subsequent independent verification are excluded, and each checker is frozen when generation ends.Source: KNighter Comparison Details · p. 39
• Both systems receive the same patch, code context, repository revisions, pinned CSA environment, read-only documentation, and reusable skills; all three configurations use an independent verifier and common acceptance checks.The verifier rebuilds frozen checkers and checks build success, vulnerable–fixed diagnostic contrast, and patch relevance; the no-skills variant removes skill packages and strategy references but retains factual API documentation, tools, task inputs, and execution budget.Source: KNighter Comparison Details · p. 39

Ablation outcomes

• Full has 146/300 artifact passes and 119/300 full passes; all 21 deadline-stopped historical artifacts failed an artifact requirement, including three with no submitted source.No candidate was regenerated or edited during completion.Source: §H.2 · p. 40

Ablation setup

• The skills ablation removes skill packages and strategy references, the feedback ablation hides build and analyzer outputs, and one-shot generation uses one model call without tools or skills.The new variants use one generation per task; the one-shot protocol also has no feedback or repair call.Source: §H.2 · p. 40

Comparison caveats

• Historical Full and new ablations differ in execution date, client context limit, and workspace/tool isolation, so the comparisons do not isolate individual component effects.The one-shot result also reflects its strict output protocol: 237/300 requests produced invalid or incomplete structured submissions, and 53 more failed compilation.Source: §H.2 · p. 40

False-positive audit

• False-discovery proportions over saved alerts are 2/78 (2.56%) for Full, 1/16 for w/o Skills, and 16/64 for w/o Execution Feedback.One-shot has no saved alerts, so its N/A entry is not a zero proportion; these counts are not repository-wide false-positive rates.Source: §H.2 · p. 40

H.3 Time-Budget Sensitivity

• Cutoffs apply to existing trajectories using active generation time; unfinished runs count as unsuccessful, and agents are not rerun with shorter deadlines.Queues, maintenance pauses, external verification, and judging are excluded from the generation clock.Source: §H.3 · p. 41
• 40-to-50-minute Pass@1 gains vary by model, while conditional metrics describe changing completed-run subsets rather than within-trajectory quality improvements.The time-budget tables report these retrospective completion curves.Source: §H.3 · p. 41, Table 18 · p. 41, Table 19 · p. 42

H.4 Language, Repository, and CWE Performance

• Language and analyzer effects cannot be separated because C/C++ tasks use CSA, while Python, Go, Java, and JavaScript tasks use CodeQL.The language-specific table retains all tasks in each language’s denominator.Source: §H.4 · p. 44, Table 21 · p. 44
• Pass@1 covers all 300 tasks, whereas average turns cover passed trajectories; these different task sets do not establish that more turns cause success.The reported turn counts include subagents.Source: §H.4 · p. 44
• CWE breakdowns count a task once per distinct label, so 14 multiply labeled tasks raise the total row count to 314 rather than 300.The main figure selects eight explicit CWE labels with at least ten tasks based on counts alone; groups also differ in language, analyzer, and repository composition.Source: §H.4 · p. 45