SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities

1 Introduction

• Fixed task sets can waste effort on tasks that are too easy, too difficult, or redundant, while leaving capabilities underexplored and making coverage costly to expand.The passage also notes that fixed evaluations may become less reliable as agents improve and benchmark artifacts enter training corpora.Source: §1 · p. 1
• SecProbe uses observed performance in an IRT-based loop to estimate agent ability and task characteristics, then select tasks that provide additional information.The framework synthesizes new tasks when suitable ones are missing, conditioning construction on a target CWE class and software context.Source: §1 · p. 2
• Comparable agent ability estimates required up to 29.5% fewer tasks than baselines, while the strongest evaluated configuration achieved a 28.33% success rate.The evaluation used 353 tasks spanning six programming languages and 151 CWE types, with nine frontier models and two agent harnesses.Source: §1 · p. 2
• On-demand synthesis supplies executable repository-scale vulnerability-repair tasks for target CWE classes and software contexts, including exploit-based tests and reference patches.This extends adaptive assessment beyond the tasks already available in a fixed bank.Source: §1 · p. 2

2.1 Code Security

• Executable security benchmarks and task generators expand the supply of security tasks; SecProbe connects observed outcomes to task selection and on-demand synthesis.Its synthesis is conditioned on target CWE classes and software contexts.Source: §2.1 · pp. 2–3

2.2 LLMs in Synthetic Data

• SecProbe extends synthetic-data generation to executable vulnerable repositories, coordinating specialized agents to construct and validate tasks using evaluation outcomes.The tasks include associated exploitation samples.Source: §2.2 · p. 3

2.3 Efficient Benchmarking

• SecProbe combines adaptive task selection with on-demand synthesis, using IRT estimates to identify information gaps and direct generation toward them.This allows both the evaluation schedule and task pool to adapt to observed agent performance.Source: §2.3 · p. 3

3.1 Preliminaries

• Agents must identify hidden vulnerabilities and patch them while preserving intended functionality; evaluation primarily tests whether patches block exploits and pass unit tests.Reference patches provide supplementary evidence of whether repairs address the underlying vulnerabilities.Source: §3.1 · pp. 3–4, §3.5 · p. 6

3.2 Overview

• An IRT feedback loop estimates agent ability and task characteristics from outcomes, then selects existing tasks or synthesizes new ones when suitable tasks are unavailable.Assessment needs guide task generation as well as evaluation effort.Source: §3.2 · pp. 4–5

3.3 Specialist Agents

• Five specialist agents research a target CWE, design a repository task and tests, implement the instance, and independently review it for revision or passage.The Security Reviewer’s decision is advisory; deterministic checks determine final admission to the task pool.Source: §3.3 · p. 5

3.4 Validation and Release

• Deterministic validation checks artifact completeness, tests, API parity, scale, and hidden-grader behavior; invalid tasks are repaired for at most ten rounds or discarded.Repairs cannot weaken tests or alter the reference implementation or grading harness to induce a pass.Source: Validation and repair loop · p. 5, §3.4 · p. 5

3.5 Difficulty Control

• Difficulty controls vary repository scale, architecture, vulnerability complexity, and evaluation tests, allowing task scale and security complexity to vary within a CWE class.The controls specify code size and structure, injected vulnerabilities and exploit paths, and public and hidden tests.Source: §3.5 · p. 6

3.6 Adaptive Evaluation

• SecProbe targets ability regions where agents are concentrated but available tasks provide little information, setting target difficulty at the largest information gap.It chooses target discrimination from an upper quantile of estimated values for nearby generated tasks; outcomes update the IRT posterior and generation mappings, and batches shrink as gaps contract.Source: §3.6 · pp. 6–7

4.2 Quality Analysis of Generated Tasks

• The 353-task pool spans six programming languages and 151 CWE types; repositories have medians of 1,009 source lines and 28 implementation files, and tasks expose a median of six distinct attack paths.The median affected-file footprint was four files (range 1–13); 346 tasks (98.0%) affected multiple files, and 436 of 1,285 vulnerability instances listed multiple files. The 20 most represented CWE types accounted for 106 tasks (30.0%); XML injection (CWE-91) led with eight tasks (2.3%), while CWE-127, CWE-134, and CWE-176 each had seven. Vulnerable seeds averaged 44.5 grader points before repair.Source: §4.2 · pp. 8–9
• Task specifications formed eight application themes, yet only 11.6% of each task’s ten nearest neighbours shared its CWE, and no task pair exceeded 0.90 cosine similarity.The median nearest-neighbour similarity was 0.74 and the maximum was 0.875. Vulnerabilities within the same task had mean cosine similarity 0.506, similar to 0.503 for same-CWE vulnerabilities across tasks. Across expert-rated dimensions, every mean exceeded 4.0 out of 5; the overall rating was 4.40, with 92% acceptance and agreement of 0.80.Source: §4.2 · pp. 9–11, Figure 6 · p. 10, Table 2 · p. 11

4.3 Evaluation of Coding-Agent Vulnerability Awareness

• GLM-5.3 achieved the highest strict pass rate, 28.33% with TERMINUS-2 (versus 26.91% with MINI-SWE-AGENT), and no evaluated configuration exceeded 30%.Harnesses yielded broadly consistent model orderings, but TERMINUS-2 had higher pass rates for several models while MINI-SWE-AGENT had higher mean grader scores across evaluated backbones.Source: §4.3 · pp. 11–12

4.4 Patch-Level Review

• Patch-level review assessed one submitted patch per task using two independent reference-based judgments, with expert adjudication when ratings differed by more than 20 points.The rubric assessed root-cause repair, coverage of affected functions and paths, and preserved application behavior; this review supplements executable testing by examining repair semantics, including alternative implementations. Table 3 reports disagreement and expert-review rates as well as expert findings of incomplete repairs, including cases that pass executable tests.Source: §4.4 · pp. 12–13, Table 3 · p. 13

4.5 Efficiency of Adaptive Evaluation

• 244 accepted tasks reached the information-gap target, versus 328 for random synthesis and 346 for one-shot synthesis.These are reductions of 25.6% and 29.5%, respectively. For held-out agents, adaptive selection reached the target with 140 tasks versus 181 for random selection and 194 for fixed ordering, with lower error at both common budgets. Against 2PL, Rasch and hierarchical 2PL had rank correlations of 0.96 and 0.98 and retained 16 and 18 of the 20 selected tasks.Source: §4.5 · pp. 13–14, Figure 9 · p. 14, Table 4 · p. 14

4.6 Validating Difficulty Control

• From Easy to Hard, pass rate fell from 44.0% to 13.3% and mean grader score from 78.6 to 55.8; 73.3% of matched triplets followed the complete intended ordering.Repository, vulnerability, and evaluation controls each increased empirical difficulty, with vulnerability complexity producing the largest effect.Source: Table 6 · p. 15, §4.6 · p. 15

4.7 Failure Analysis

• Repair failures commonly involved incomplete coverage across functions or files, faulty diagnosis or localization, and missed application-level rules or state transitions.Agents could leave another path unprotected, edit away from the underlying flaw, overlook authorization or permitted-value requirements, or introduce regressions in required behavior or interfaces.Source: Figure 10 · p. 15, Table 7 · p. 16, §4.7 · p. 16

4.8 Cost

• Repository-scale task generation cost approximately $0.64 per task; a linear projection for 353 tasks was approximately $226.24, excluding agent-evaluation costs.The projection varies with repository complexity, token use, and repair rounds.Source: §4.8 · pp. 16–17
• Approximately $5,000, including contingency, is the estimated API budget for evaluating all 353 tasks once with nine model backbones under MINI-SWE-AGENT and TERMINUS-2.This recurring expense motivates adaptive evaluation to reduce the number of task attempts needed for informative assessment.Source: §4.8 · p. 17

• 28.33% was the strongest evaluated configuration’s success rate, revealing substantial remaining challenges in vulnerability identification and repair.Source: §5 · p. 17
• SecProbe connects IRT-based task selection with on-demand multi-agent synthesis to adapt evaluation and renew assessment as coding agents evolve.Source: §5 · p. 17
• The study constructed 353 tasks across six languages and 151 CWE types; quantitative analysis and expert review supported their quality, and adaptive evaluation reduced tasks needed to reach the assessment target.Source: §5 · p. 17

• SecProbe’s intended use is defensive evaluation of coding agents’ ability to identify and repair vulnerabilities.Source: Broader Impact and Responsible Use · p. 17
• Synthetic repositories and isolated environments limit immediate exposure, but generated vulnerable code and exploit examples retain dual-use potential.Source: Broader Impact and Responsible Use · p. 17
• The synthetic task distribution does not represent all deployed software, so task success is not a guarantee of secure deployment.Source: Broader Impact and Responsible Use · p. 17

• Specialized LLM agents research vulnerability classes, design tasks, construct repositories and reference repairs, develop tests, and review generated artifacts.Source: AI Use Statement · p. 17
• The task-construction agents use GPT-5.6-Luna with medium reasoning effort, and generated tasks undergo executable checks and expert quality assessment.Source: AI Use Statement · p. 17
• AI assistants also helped refine the manuscript’s text, figures, and tables; the authors take responsibility for the final text, claims, and code.Source: AI Use Statement · p. 17

• The paper locates task formulation, the multi-agent workflow, difficulty controls, and adaptive policy in Section 3, with implementation and artifact organization in Appendix A.Source: Reproducibility Statement · p. 17
• Validation, execution, scoring, expert-assessment criteria, and specialist-agent prompts are documented in Sections 3.4, 4.1, 4.2, and Appendix C, respectively.Source: Reproducibility Statement · p. 17
• Stochastic generation and changes to externally hosted models can affect exact outputs despite the procedure’s replication specifications.Source: Reproducibility Statement · p. 17

A Implementation Details

• Each task separates metadata, secure reference, released vulnerable repository, and hidden grader artifacts into four roots.Metadata stores intermediate artifacts; the grader root contains hidden tests and scoring.Source: A Implementation Details · p. 22
• Stage-specific filesystem capabilities restrict path access, preventing test authors from inspecting the secure reference and repository builders from accessing the grader.All filesystem access uses four audited tools, and unauthorized paths are rejected.Source: A Implementation Details · p. 22
• The command tool takes an argument vector rather than a shell string and applies a timeout to each invocation.This prevents generated content from being interpreted as shell syntax.Source: A Implementation Details · p. 22

A.1 Hyperparameter Settings

• Tasks target one of three deployment scopes: library component, single-process application, or multi-service application.The scopes distinguish API-based testing, application-boundary testing, and integration across an explicit local network flow.Source: §A.1 · p. 22
• The difficulty-control vector specifies integer lower bounds for repository scale, architecture, vulnerability complexity, and evaluation depth.The measures cover source files and functions, components and APIs, vulnerabilities and attack paths, and public, hidden, and adversarial tests.Source: §A.1 · p. 22
• Released instances require at least 32 atomic requirements, including 16 security requirements, plus minimum repository, architecture, vulnerability, and evaluation bounds.The release gate checks the requested constraints; repositories have a median of 1,009 source lines, and 64 fall below 1,000.Source: §A.1 · p. 23
• The IRT policy uses a twelve-configuration agent pool, Bayesian two-parameter logistic model, and posterior sampling with convergence checks before fitting the adaptive policy.It begins with 150 tasks, then uses batches of 80, 50, 40, and 33 accepted tasks, stopping when the normalized information gap falls below 0.15.Source: §A.1 · p. 23

A.2 Generator Configuration

• All specialists use one model backbone, and only the Online Security Researcher has network access.The researcher is limited to twelve sources and prefers material updated within 730 days, while retaining older standards where appropriate.Source: §A.2 · p. 23
• Later generation stages read the researcher’s output as frozen, untrusted evidence.The task pool was generated with GPT-5.6-Luna under specified tool-turn, command-timeout, repair-round, and container-oracle budgets.Source: §A.2 · p. 23

B Failure Case Studies

• A path-containment repair still permits an unregistered caller named public-guest to read a restricted-vault file, while the reference rejects the request.Both implementations allow the registered caller and reject the tested traversal attempt.Source: B Failure Case Studies · p. 23, Figure 11 · p. 24
• A nonce-reuse test returns 409 because of a stale version, but a current-version request reusing the consumed nonce succeeds and changes the setpoint from 22 to 99.The reference rejects the reused nonce and preserves 22; a fresh nonce succeeds in both implementations.Source: B Failure Case Studies · p. 23, Figure 12 · p. 24
• Restricted deserialization still commits an authenticated value of 240 because the agent retains the bound |x| ≤ 2^63 instead of the reference bound |x| ≤ 2^31.Blocking executable reconstruction leaves a separate obligation to validate accepted-value domains.Source: B Failure Case Studies · pp. 23, 25, Figure 13 · p. 25
• Across the cases, agents strengthen one security layer but leave independent requirements unenforced, with decisive vulnerable branches remaining in helpers on the repaired execution path.The reproductions isolate checks by satisfying surrounding conditions and inspecting protected data or state as well as status codes.Source: B Failure Case Studies · p. 25

C Prompt Templates

• The five prompt templates assign specialists to gather security evidence, produce a shared blueprint and test strategy, construct task artifacts, and conduct a read-only audit.The prompts describe the Online Security Researcher, Task Architect, Exploit Test Strategist, Repository Engineer, and Security Reviewer.Source: C Prompt Templates · p. 25, Figure 14 · p. 26, Figure 15 · p. 26