Ulysses: A Security Harness For Synthesizing Verifiable Web Exploits

1. Introduction

• A finding is only a hypothesis: verification requires tracing the complete attack path, including application state, input transformations, authentication, and payload construction.A Proof of Exploitability demonstrates an attacker’s capability against a deployed system.Source: §1 · pp. 1–2
• LLM-generated findings can be plausible yet unverified, adding triage burden when reports lack executable evidence or proof that a flaw is exploitable.The paper cites curl’s report that about 20% of vulnerabilities were incorrectly generated by AI and estimates that roughly half of sampled May 2026 advisories lacked Proof of Exploit.Source: §1 · pp. 1–2
• ULYSSES converts candidate vulnerability findings into effect-verified Proofs of Exploitability rather than relying on a report or an agent’s self-assessment.Its evaluation covers 25 real-world applications and the OWASP Python benchmark across eight effect types.Source: §1 · p. 3

2. Background

• Web vulnerabilities arise when untrusted input reaches a sensitive operation without adequate validation, potentially enabling data access, modification, authentication bypass, or code execution.ULYSSES targets eight observable effect types, including unauthorized file and database access, remote code execution, privilege escalation, and authentication bypass.Source: §2 · pp. 3–4
• The evaluation distinguishes advisory-informed, zero-knowledge, and static-analysis-seeded conditions by what vulnerability information the agent receives.All conditions target disclosed vulnerabilities; zero-knowledge means no prior information about the target, not absence of source code or a running endpoint.Source: §2 · p. 3

3. Motivation

• Tracing a source and sink is only a start: exploit construction must follow intermediate transformations, as the benchmark endpoint’s wrapper and slice show.The attacker-controlled parameter is wrapped and sliced before reaching pickle.loads, requiring a payload that survives those transformations.Source: §3 · pp. 4–5

4. Design

• ULYSSES’s design addresses four challenges: distinguishing exploitable findings, exploring multiple paths, chaining steps with adaptation, and independently verifying success.An exploiting agent cannot reliably verify its own success.Source: §4 · p. 5

4.1. Vulnerability Scanning

• ULYSSES uses static analyzers, LLM-assisted scanners, or CVE descriptions as starting information, separating detection from exploit verification.The inputs may mark sources and sinks, provide attack plans, or identify possible vulnerable locations.Source: §4.1 · p. 5

4.2. Hypothesis Graphs

• The planning agent represents candidate exploits as a graph whose edges carry reasoning and context; it rebuilds the graph each round using accumulated findings and reflection.This supports concurrent candidate paths and gives each node only the context passed along its incoming edges.Source: §4.2 · p. 6

4.3. Effect Oracle

• The effect oracle checks submitted evidence against planted ground truth and confirms an exploit only after a match; isolated listeners prevent the agent from submitting callbacks directly.The oracle is mechanism-blind, while hardened flags must be discovered by exploiting the application.Source: §4.3 · pp. 6–7
• Table 1 maps eight effect types to representative weakness classes, required evidence, and the oracle’s validation checks.The table’s comparison is between the evidence an exploit must produce and what the oracle checks.Source: Table 1 · p. 7, §4.3 · p. 7

5. Approach

• The overall approach repeats planning, exploitation, and reflection on a hypothesis graph until the time budget or round limit is reached.The agent operates against a hosted target while the oracle independently arbitrates evidence.Source: §5 · p. 7, §5.1 · p. 7

5.1. Threat Model

• The agent has source-code access but attacks the hosted target over the network, without direct access to its database, file system, or operating system.The independent oracle knows planted ground truth but not which findings are exploitable; the agent cannot reach its effect listeners or see its seeds.Source: §5.1 · pp. 7–8

5.2. Agent Loop

• Each bounded round generates hypotheses, tests them in parallel, records oracle-validated exploits, and uses reflection to seed the next planner.The loop returns the refined report and effect-verified PoEs.Source: §5.2 · p. 8

5.3. Planning

• The planner explores unique attack scenarios from scanner chains, source code, and prior reflection, testing batches in decreasing confidence order.Reflection steers later rounds toward plans more likely to succeed than earlier ones.Source: §5.3 · p. 8

5.4. Exploitation

• An exploit subagent tests each hypothesis with scripts and HTTP requests, then relies on the effect oracle—not its own judgment—to confirm success.Only confirmed exploits are kept; vulnerable-but-non-exploitable and benign reports are filtered out.Source: §5.4 · p. 9

5.5. Proof of Exploit Generation

• A separate PoE node runs only after oracle-confirmed success and generates a standalone Python script to reproduce the exploit on the deployed target.PoE quality is checked by rerunning the script and verifying the expected effect.Source: §5.5 · p. 9

5.6. Reflection

• The reflection agent records validated and uncovered effect types, then feeds what worked and failed into the next planning round.This feedback supports improved attempts and multi-step chains across rounds.Source: §5.6 · p. 9

• The benchmark targets live web-application exploit verification, whereas existing benchmarks may use memory-safety bugs, build-only libraries, or synthetic code and validate by crashes or labels.Its instances require a deployed, running application and vulnerable-commit source code.Source: §6 · p. 9
• The benchmark packages each vulnerability at its vulnerable release with seeded state for observing an effect, and scores submissions by executing the generated PoE.Instances are drawn from NVD and GitHub Security Advisories and receive only the CVE identifier, summary, and any required attacker credentials.Source: §6 · p. 9
• WEBEXPLOITGYM contains 25 instances across five languages and seven web frameworks, covering eight effect types and CVEs disclosed between 2025 and 2026.Fourteen instances have a public PoE and 11 do not.Source: §6 · p. 10

Effect oracle

• ULYSSES emits a PoE only after the effect oracle confirms success, avoiding reliance on an exploiting agent’s self-report when the effect is unobservable.The authors attribute sophisticated agents’ lower failure rate to exploring more hypotheses before reporting, but describe them as unreliable judges of whether an exploit worked.Source: §7 · p. 11

Architecture

• Hypothesis graphs achieved much higher recall than ReAct agents across three model families, whose results were within 1.6 points of one another.Both architectures had a 300-second budget per route; the paper reports architecture choice mattered more than the model’s intrinsic capabilities.Source: §7 · p. 12, Figure 5 · p. 12

Vulnerability scanner

• The initial vulnerability scanner sets an upper bound on recall regardless of the model or verification loop; Absentia supplied stronger potential candidates than CodeQL.Figure 5’s accompanying text also reports that hypothesis graphs improve recall at every model.Source: §7 · p. 12, Figure 5 · p. 12

Real-world evaluation

• 10 of 25 targets yielded effect-verified PoEs with advisory information, versus 4 of 25 in zero knowledge.Semgrep-seeded analysis identified 3 targets, but noisy findings required reachability triage and produced fewer conversions than advisories identifying the affected component.Source: §7 · p. 12

Case setup

• The rConfig 8.2.12 case concerns authenticated path traversal in the export-download endpoint; its advisory provides no PoE, reproduction steps, or payload.The vulnerable code combines the application-controlled export directory with a user-supplied filename without sanitizing or constraining it to that directory, allowing traversal to target files outside it.Source: §8 · pp. 13–14

What makes this target difficult

• The exploit requires navigating a CSRF-protected login flow and identifying the endpoint and filename parameter, despite a containment service protecting other controllers.The surrounding protection can make it easy to assume incorrectly that this endpoint is also protected.Source: §8 · p. 13

Zero Knowledge Scenario

• In zero knowledge, the planner found vulnerable source code but was misled by the containment service and failed to establish that the endpoint was reachable.Without a localization signal, the exploit agent spent its budget discovering the vulnerability instead of validating and exploiting it.Source: §8 · p. 13

Static Analysis Scenario

• With Semgrep findings, ULYSSES had to assess noisy candidates, locate a reachable endpoint, authenticate, construct a request, and verify its effect; it produced a PoE for the target vulnerability.The paper describes static-analysis findings as a precise starting point that helps resolve the localization barrier and test multiple hypotheses.Source: §8 · p. 14

Advisory-informed Scenario

• With advisory information, ULYSSES knew the weakness class and vulnerable endpoint but not the payload, allowing it to focus on testing reachability and constructing a PoE.The case-study takeaway contrasts this higher-level description with static analysis’s more precise starting point; both informed settings produced and verified an exploit.Source: §8 · p. 15

9.1. Developers and Industry Impact

• A PoE packages endpoint discovery, request construction, authentication, and success criteria into a reproducible test that reduces maintainers’ verification effort.It also supports communication between teams and reuse for triage, patch validation, and regression testing.Source: §9.1 · p. 15

9.2. Future Work

• ULYSSES currently covers vulnerabilities with observable server-side effects and reported Day-N vulnerabilities.Source: §9.2 · p. 15
• Future work could extend effect verification to classes such as memory-safety bugs, while whether it can discover undisclosed vulnerabilities remains open.The zero-knowledge condition withholds the advisory but still targets a disclosed vulnerability.Source: §9.2 · p. 15

Construct Validity

• The effect oracle is mechanism-blind, so another vulnerability producing the same effect could pass; a route-and-vector gate mitigates this in advisory-informed and static-analysis-seeded conditions.The gate checks the target endpoint, injection signature, and authentication precondition; zero-knowledge applies no such gate.Source: §10 · p. 16

Internal Validity

• Post-cutoff disclosure limits memorization of specific advisories or proofs, but not prior exposure to vulnerable source code or recall of older vulnerabilities.WEBEXPLOITGYM uses one run per target, with generated PoEs manually inspected and re-executed; repeated runs are time-consuming and expensive.Source: §10 · p. 16

External Validity

• Testing covers Dockerized web applications and eight observable running-server effects, not non-web applications, memory bugs, or vulnerabilities without such effects.Source: §10 · p. 16

Conclusion Validity

• WEBEXPLOITGYM results count vulnerabilities confirmed to reproduce target CVEs, all generated proofs were manually inspected, and OWASP reports union pass@3.Source: §10 · p. 16

Vulnerability Detection using Static Analysis

• Static analyzers can produce false positives and miss runtime behavior because taint analysis cannot enumerate every program state.The reviewed approaches include rule-based taint analysis and code-property-graph methods.Source: §11 · pp. 16–17

Vulnerability Exploitation using Dynamic Analysis

• DAST confirms exploits through live responses but often uses shallow, handcrafted or rule-based probes that miss multi-step input transformations.ULYSSES retains effect-based confirmation while adding source-level reasoning to construct multi-step exploits.Source: §11 · p. 17

LLMs for Detecting Vulnerable Code

• LLM-assisted detectors improve candidate detection but do not establish exploitability; ULYSSES consumes their findings and verifies candidates through exploits.IRIS uses fixed CodeQL query templates, while QLCoder retains a high false-positive rate even with a strong backbone model.Source: §11 · p. 17

Exploitation Benchmarks

• WEBEXPLOITGYM extends benchmark coverage to running web applications and eight observable effects, beyond benchmarks centered on memory-safety bugs.Those other benchmarks validate primarily through crash reproduction or differential testing against patched versions.Source: §11 · p. 17

Proof of Vulnerabilities Generation

• ULYSSES verifies exploits by observable effects on a live HTTP service, unlike proof-generation approaches centered on code-level tests, crashes, or patches.It requires neither a bug oracle nor a patch.Source: §11 · pp. 17–18

Agent Harness for Security

• ULYSSES is open source, accepts findings from any detector, and runs on any backbone model, unlike vendor-bound security-review agents.Source: §11 · p. 18

12. Conclusion

• ULYSSES produced verified exploits for 10 applications and, through coordinated disclosure, reported the first working proof for four advisories.Source: §12 · p. 18
• Providing vulnerability location more than doubles successful exploits, from 4 to 10 of 25 applications, while authenticated and multi-step vulnerabilities remain challenging.The authors identify localization and model capabilities as areas requiring progress toward autonomous exploitation.Source: §12 · p. 18

Ethics and Privacy Statement

• Evaluation used disclosed N-day vulnerabilities on open-source applications and the synthetic OWASP Benchmark, with targets deployed locally rather than production systems.Source: Ethics and Privacy Statement · pp. 18–19
• The authors disclose generated exploits to affected maintainers and caution that the open-source red-teaming agent could be repurposed for malicious use.Source: Ethics and Privacy Statement · p. 19

A. Appendix

• Table 3 lists the vulnerability instances included in the benchmark.Source: Table 3 · p. 23