PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Static Guarantees: Measuring the Static-Pass Dynamic-Fail Gap in Security-Sensitive and LLM-Generated Python Code

Jessica Pourleyli, Maitreyee Das Urmi, Glaucia Melo

arXiv:2609.10762v1cs.CRcs.AIcs.SE

TL;DR

Static analysis can miss vulnerabilities that remain exploitable at runtime, so the paper defines SPDF and evaluates a three-stage detection-and-verification pipeline. Across 1,355 Python samples, the pipeline yielded runtime evidence for 95 of 654 Bandit–Semgrep-clean files, an inclusive rate of 14.53%, while the authors emphasize that this is an observed pipeline yield rather than total prevalence.

  • Problem

    Static analysis is scalable and inexpensive but cannot directly observe runtime exploitability, leaving vulnerabilities dependent on inputs, context, or exploit chaining insufficiently assessed.

  • Method

    The paper combines Bandit–Semgrep filtering, LLM-driven CWE reasoning, and Docker-isolated autonomous exploit verification on security-oriented Python datasets.

  • Results

    14.53% (95/654) was the inclusive observed pipeline yield among Bandit–Semgrep-clean files, including partial dynamic-verification outcomes.

  • Takeaways & Limitations

    Static-analysis success and runtime security provide distinct hierarchical evidence, so passing the evaluated static gate does not establish runtime security.

  • Takeaways & Limitations

    The reported SPDF rates are lower bounds on demonstrated exploitability because sandbox constraints and exploit-harness generation can produce inconclusive or untriggered outcomes.

Abstract

from arXiv · show

Advances in large language models (LLMs) fuel the quest for scalable methods to assess the security of generated and security-sensitive software. Static analysis is widely adopted as a scalable, reproducible, and inexpensive security gate, but cannot directly observe runtime exploit behaviour. Vulnerabilities dependent on adversarial inputs, execution context, or exploit chaining may evade static checks while remaining exploitable in practice, yet passing static analysis is often treated as evidence of secure behaviour. This paper introduces the Static-Pass Dynamic-Fail (SPDF) phenomenon and a three-stage agentic pipeline combining static scanning, LLM-driven Common Weakness Enumeration (CWE) reasoning, and autonomous exploit verification in isolated Docker containers. We evaluate 1,355 Python samples from SecurityEval, RedCode, and CyberNative datasets. Of the 654 samples producing no findings under the composite Bandit-Semgrep gate, the LLM detection stage identified 394 candidate vulnerabilities across 235 files. Dynamic verification confirmed or partially confirmed exploitability in 95 files, yielding an inclusive pipeline rate of 14.53% (roughly 1 in 7 statically clean samples). This rate represents the proportion of Bandit-Semgrep-clean samples for which the pipeline identified a candidate vulnerability and obtained runtime evidence supporting exploitability. Outcomes varied by dataset: among candidate file--CWE pairs, confirmed exploitability was 33.7% for RedCode, 28.6% for CyberNative, and 5.4% for SecurityEval. Several frequently confirmed classes, including CWE-338 and CWE-916, were flagged by neither Bandit nor Semgrep. These findings indicate that static-analysis success and runtime security are hierarchical layers of software assurance rather than interchangeable measures, and have the potential to reshape how AI-generated and security-sensitive code is evaluated.

I. INTRODUCTION

The paper examines vulnerabilities that pass composite static analysis yet remain dynamically exploitable, introducing SPDF and an agentic pipeline to measure this gap.

  • I. INTRODUCTION: The study introduces Static-Pass Dynamic-Fail (SPDF) for vulnerabilities missed by Bandit–Semgrep but demonstrated through runtime exploit verification.It frames static analysis, vulnerability reasoning, and dynamic verification as progressively stronger but non-interchangeable evidence levels.
  • I. INTRODUCTION: The pipeline combines static scanning, LLM-driven CWE reasoning, and autonomous exploit generation in isolated Docker environments.It first filters samples with Bandit and Semgrep, then analyzes statically clean samples and verifies selected candidates dynamically.
  • I. INTRODUCTION: 1,355 Python samples from SecurityEval, RedCode, and CyberNative guide the evaluation of SPDF prevalence, vulnerability categories, and computational cost.The study asks how often statically clean code remains exploitable, which CWE categories evade static analysis, and what the pipeline costs.
  • I. INTRODUCTION: The contributions include an operational SPDF construct and rate, computational-cost measurements, and a curated collection of verified samples and reproducibility artifacts.The artifacts include exploit evidence and exploit-verification code supporting runtime vulnerability assessment research.
  • I. INTRODUCTION: The findings target security-sensitive benchmark code and LLM-generated vulnerable samples, not code produced by assistants in interactive development workflows.Extending the results to assistant-in-the-loop pipelines remains a hypothesis rather than an established conclusion.

II. RELATED WORK

Prior work evaluates generated-code security with benchmarks, static analysis, fuzzing, and exploit-based validation, but the static-pass dynamic-fail gap remains underexplored.

  • II. RELATED WORK: LLM-generated programs may be functionally plausible while containing known CWE-associated vulnerabilities, separating functional success from secure code generation.This motivates evaluating robustness against vulnerability patterns rather than surface-level task satisfaction alone.
  • II. RELATED WORK: Security-focused benchmarks assess generated code beyond functional correctness, including curated CWE tasks, risky execution scenarios, and repair validation with exploits.SecurityEval, RedCode, and VulnRepairEval represent complementary benchmark and evaluation directions.
  • II. RELATED WORK: Static analysis is scalable and widely used, but its effectiveness depends on rule coverage and vulnerability-pattern representation, leaving potential false negatives.Prior work commonly uses tools such as Bandit and CodeQL for generated-code security evaluation.
  • II. RELATED WORK: Execution-aware approaches use fuzzing, sandboxed execution, and proof-of-concept exploits to test whether vulnerabilities can actually be triggered.These methods strengthen security evaluation beyond pattern-based static checks.
  • II. RELATED WORK: The paper addresses this gap because runtime exploitability can depend on adversarial inputs, execution context, or whether a vulnerable path is triggerable.The specific case of statically clean security-sensitive code later failing dynamic exploit verification remains underexplored.

D. Agentic Security Pipelines and Autonomous Software Evaluation

The paper extends agentic software evaluation toward runtime security by combining tool-mediated reasoning, static scanning, CWE analysis, and Docker-based exploit verification across complementary datasets.

  • D. Agentic Security Pipelines and Autonomous Software Evaluation: The proposed direction focuses on the gap between static validation and runtime exploitability using CWE-guided detection and Docker-based verification.It builds on agentic systems that combine reasoning, tool use, and interaction with software environments.
  • D. Agentic Security Pipelines and Autonomous Software Evaluation: The three-stage pipeline combines deterministic Bandit–Semgrep analysis with LLM-based CWE detection and dynamic exploit verification.The stages progressively filter, analyze, and validate security weaknesses using static and execution-based techniques.
  • D. Agentic Security Pipelines and Autonomous Software Evaluation: The evaluation uses 1,355 Python samples from SecurityEval, RedCode, and CyberNative to cover benchmark-driven, adversarial, and LLM-generated security contexts.Only Python samples were retained for compatibility across the analysis and execution components.
  • D. Agentic Security Pipelines and Autonomous Software Evaluation: Table I summarizes the final dataset composition used in the SPDF evaluation.The supplied caption identifies the table’s scope but does not provide its cell values.

B. Pipeline Overview

The pipeline first establishes a composite static-pass condition, then uses CWE-grounded LLM reasoning to identify candidates for autonomous dynamic exploit verification.

  • B. Pipeline Overview: A sample qualifies as SPDF only after passing the static gate, receiving a candidate CWE, and exhibiting runtime evidence of exploitability.Dynamic verification is applied to candidates identified by the detection agent rather than to every statically clean file.
  • B. Pipeline Overview: Bandit and Semgrep form a stricter composite filter because a file is statically clean only when neither tool reports a finding.Bandit targets Python-specific weaknesses, while Semgrep detects broader rule-based vulnerability patterns.
  • B. Pipeline Overview: The static stage used Bandit v1.8.4.dev21 and Semgrep v1.155.0 with JSON outputs and no additional confidence or severity thresholds.Files with no findings from either tool advanced to the second stage.
  • B. Pipeline Overview: 654 of 1,355 files passed composite Bandit–Semgrep analysis and advanced to the CWE Vulnerability Detection Agent.701 unique files were flagged by at least one tool after merging detections and removing duplicate file-level reports.
  • B. Pipeline Overview: The CWE agent uses ReAct-style source inspection and retrieval-augmented analysis to identify file–CWE candidates grounded in MITRE definitions.Candidates remain hypothesized vulnerabilities until dynamic exploit verification, and the agent is instructed to validate suspected weaknesses against supporting evidence.
  • B. Pipeline Overview: Five independent agent runs per sample and up to ten reasoning or tool-use iterations accommodate variability in LLM-based security judgments.Token consumption varies with analysis complexity because execution was bounded by interaction turns rather than a fixed per-instance token budget.

E. Dynamic Exploit Verification Agent

The Dynamic Exploit Verification Agent uses retrieval-augmented reasoning and tailored exploit harnesses in isolated Docker environments to test candidate vulnerabilities. Verdicts require vulnerability-specific runtime evidence, while shared-model dependence limits independence between detection and verification.

  • E. Dynamic Exploit Verification Agent: The agent retrieves CWE and NVD evidence, then generates vulnerability-specific exploit harnesses tailored to the target code.It varies payloads and execution paths for weaknesses such as command injection and path traversal.
  • E. Dynamic Exploit Verification Agent: Generated harnesses run in isolated Docker containers with disabled networking, 256 MB memory, and configurable timeouts.The pipeline records outputs, execution status, resource use, and errors for verdict assignment.
  • E. Dynamic Exploit Verification Agent: A candidate is verified only when runtime behaviour matches the target CWE, while exceptions alone do not establish exploitation.Command injection requires unintended command execution, whereas path traversal requires unauthorized file access.
  • E. Dynamic Exploit Verification Agent: Each file–CWE pair is tested across five executions, and verification requires at least three CONFIRMED runs.NOT TRIGGERED means insufficient evidence under the performed tests, not proof that the weakness is absent or unexploitable elsewhere.
  • E. Dynamic Exploit Verification Agent: Both LLM stages use gpt-5.4-nano, so shared model tendencies may create correlated errors or self-confirmation.The study accounts for this dependency when interpreting exploitability estimates.

F. SPDF Measurement

SPDFR measures the file-level yield of the complete detection-and-verification pipeline among Bandit–Semgrep-clean samples. The analysis also examines vulnerability categories, computational cost, and dataset or category differences in confirmation outcomes.

  • F. SPDF Measurement: SPDFR is the proportion of Bandit–Semgrep-clean files for which the pipeline subsequently confirms runtime exploitability.Its denominator includes all static-pass files, while dynamic testing covers only candidates surfaced by the CWE Detection Agent.
  • F. SPDF Measurement: The study reports conservative and inclusive SPDFR definitions, with the latter counting both CONFIRMED and PARTIAL outcomes.This distinction separates confirmed exploitability from broader evidence of exploitability.
  • F. SPDF Measurement: The evaluation records latency, token consumption, reasoning iterations, and exploit-generation activity to quantify agentic verification cost.These measures address the feasibility of deploying the pipeline in practical security workflows.
  • F. SPDF Measurement: Dataset confirmation differences are tested with Pearson’s chi-square test at α = 0.05, while vulnerability-category differences use a permutation-based chi-square test.The permutation approach addresses small category sample sizes that violate standard asymptotic assumptions.

H. Reproducibility and Artifact Release

The results section traces samples through the SPDF pipeline, from static filtering and candidate detection to dynamic outcomes. The study reports measurable exploitability yields, dataset variation, and the limits of interpreting those yields as total vulnerability prevalence.

  • H. Reproducibility and Artifact Release: The released pipeline, exploit harnesses, SPDF-positive samples, and verification metadata support reproducibility and future runtime-security research.The reported yields remain a model-dependent lower bound because testing covered only LLM-surfaced candidates under the evaluated configuration and environment.
  • H. Reproducibility and Artifact Release: 1,355 samples produced 701 statically flagged files and 654 static-pass files, after which the detection agent found 394 candidates across 235 files.These stages define the progression summarized by Figure 1 and Table IV.
  • H. Reproducibility and Artifact Release: 14.53% (95/654) of Bandit/Semgrep-clean files achieved the inclusive SPDF pipeline yield, versus 12.69% (83/654) under the conservative definition.The inclusive result covers 140 file–CWE pairs across 95 files, while the conservative result covers 120 confirmed pairs across 83 files.
  • H. Reproducibility and Artifact Release: RedCode, CyberNative, and SecurityEval had confirmation rates of 33.7%, 28.6%, and 5.4%, respectively, with dataset and outcome significantly associated.RedCode contributed 106 of 120 confirmed outcomes and 79.9% of candidate file–CWE pairs.
  • H. Reproducibility and Artifact Release: 120 candidates were CONFIRMED and 20 PARTIAL, so 35.6% (140/394) of candidate file–CWE pairs showed at least some exploitability evidence.208 candidates were not triggered and 46 remained inconclusive under the generated harnesses and execution conditions.
  • H. Reproducibility and Artifact Release: More than one in eight static-pass samples contained confirmed vulnerabilities triggerable through execution-based testing.The authors therefore caution that static-analysis success should not be treated as evidence that software resists exploitation.

B. RQ2: What Categories of Vulnerabilities Are Most Likely

SPDF behaviour varies substantially by vulnerability category: some weaknesses are repeatedly confirmed at runtime despite static cleanliness, while others are rarely reproduced.

  • RedCode had the highest exploit-confirmation rate at 33.7%, followed by CyberNative at 28.6% and SecurityEval at 5.4%.
  • CWE-category association with confirmed exploitability was significant (χ2(22) = 138.06, p < 0.0001).
  • CWE-400 produced the most candidate findings and confirmed exploitations, with 27 confirmed cases.Other frequently confirmed categories included CWE-338 and CWE-916.
  • Resource exhaustion, randomness, and cryptographic weaknesses can evade syntactic inspection because their exploitability depends on runtime behaviour.
  • CWE-338 and CWE-330 reached 100% confirmation, whereas CWE-94, CWE-434, and CWE-20 frequently identified by reasoning did not yield successful demonstrations.
  • CWE-338 produced 15 confirmed exploits and CWE-916 produced 9 despite no Bandit or Semgrep findings anywhere in the collection.
  • Static coverage does not ensure instance-level detection: CWE-400, CWE-330, and CWE-22 appeared in static findings elsewhere but still produced confirmed SPDF cases.

C. RQ3: What Is the Computational Cost of the Agentic Dynamic-Exploitability Pipeline?

The agentic pipeline requires substantial computational resources, making dynamic exploit verification better suited to selective rather than exhaustive use.

  • Dynamic verification evaluated 394 file–CWE pairs through 1,304 Docker executions, averaging approximately 3.3 exploit attempts per candidate.
  • The pipeline consumed approximately 96.3M tokens and 28.7 hours across CWE detection and dynamic verification.The CWE Detection Agent used 24.6M tokens and 10.5 hours, while dynamic verification used 71.7M tokens and 18.2 hours.
  • The most expensive individual evaluation consumed 219,084 tokens and more than 20 minutes.Means exceeded medians, indicating a tail of difficult cases requiring additional reasoning and exploit generation.
  • Dynamic exploit verification is substantially more expensive than static analysis and is therefore better suited to selective verification than exhaustive screening.

V. DISCUSSION

The results frame software security as a hierarchy of evidence rather than a binary property, while showing that aggregate SPDF rates are constrained by dataset composition, model dependence, and severity weighting.

  • The pipeline reduced 1,355 samples to 654 statically clean files, 394 candidate weaknesses, and 120 confirmed exploitations across 83 files.The conservative SPDF rate was 12.69%, or more than one in eight statically clean files.
  • RedCode supplied 80% of candidate file–CWE pairs and 88% of confirmed exploitations, so the aggregate rate is overwhelmingly shaped by one benchmark.SecurityEval had the lowest confirmed rate at 5.4%, compared with 33.7% for RedCode and 28.6% for CyberNative.
  • Both reasoning agents use gpt-5.4-nano, creating potential correlated errors and self-confirmation that five-run majority voting cannot remove.Different model families or human audits would help measure how independent the verification signal is.
  • The SPDF rate treats all confirmed exploitations equally even though frequent confirmed classes are dominated by denial-of-service and weak-randomness weaknesses with differing practical consequences.CWE-94 produced numerous candidate findings but no confirmed exploitations.
  • Static analysis remains an efficient first-line filter, but stronger assessments combine it with execution-based verification rather than relying on either alone.Static analysis eliminated 701 files here at negligible cost.
  • Whole vulnerability categories can fall outside rule-based coverage, especially semantic properties, randomness quality, cryptographic strength, authorization logic, and resource consumption.
  • Dynamic verification confirmed 120 weaknesses while 208 candidates were not triggered under the generated harnesses and available execution conditions, supporting selective layered deployment.The paper notes a total cost of roughly 96.3 million tokens and 29 hours.

VI. THREATS TO VALIDITY

The study’s runtime-exploitability findings are bounded by environmental, model, aggregation, and dataset constraints, so reported rates should be interpreted as pipeline-specific evidence rather than universal prevalence estimates.

  • A. Construct Validity: The reported SPDF rates are lower bounds on demonstrated exploitability because sandbox limits and exploit-harness effectiveness can produce NOT_TRIGGERED or INCONCLUSIVE outcomes despite underlying vulnerabilities.The study reports 46 INCONCLUSIVE outcomes and notes that dynamic verification requires an effective generated exploit harness.
  • B. Internal Validity: Using the same gpt-5.4-nano model for detection and verification may make some confirmation signals self-reinforcing.The authors identify model choice and shared-model confirmation as factors that may influence observed rates.
  • B. Internal Validity: Docker execution and resource limits improve safety and reproducibility but may suppress exploits requiring system privileges or external conditions.
  • C. External Validity: The evidence is limited to Python samples from three datasets, and results may differ for production systems, other languages, or alternative scanner versions and rulesets.
  • Future Work: Future evaluations should use detector and verifier models from different families and weight confirmed instances by severity to assess independence and practical risk.
Loading 2609.10762v1…