Trajectory-Level Security Debt in LLM Coding Agents

1 Introduction

• As agents reuse their earlier output, security-relevant design choices can persist through later changes, while code reuse can spread insecure choices across repositories and dependencies.Prior work reports vulnerable Copilot suggestions in roughly 40% of tested CWE scenarios.Source: §1 · p. 1
• Recorded intermediate states can be scanned before submission, whereas final-artifact-only evaluation leaves the evolution of security findings unobserved.The study also asks when findings enter trajectories and whether they can be repaired while preserving tested behavior.Source: §1 · p. 1
• SDLI charges scanner risk when an agent reaches a new historical-best functional progress level, retaining earlier charges even if code is later removed.Its risk reading is instantiated with four SAST tools.Source: §1 · p. 1
• 830 passing SWE-bench runs, 80 mostly working ProgramBench runs, and 13 public MirrorCode trajectories provide three populations for studying security findings alongside functional success.The large populations support endpoint analysis; MirrorCode offers a trajectory case study.Source: §1 · p. 1

2.1 Long-Horizon Agent Tasks

• Checkpoint monitoring offers a way to inspect security findings while implementations evolve across long-horizon agent tasks.The background describes repository-scale benchmarks and workflows extending beyond isolated functions.Source: §2.1 · p. 2, Figure 1 · p. 2

2.2 Why Python and Why BBCS

• The study focuses on Python and combines Bandit, Bearer, CodeQL, and Semgrep as BBCS, while noting that prior empirical evaluation does not validate precision on these artifacts.Source: §2.2 · p. 2

2.3 Related Work

• SDLI couples scanner readings to progress across recorded states, complementing iterative-refinement studies, commit-level attribution methods, and exploit-based evaluations.The paper states that its class-level measurement does not provide commit-level attribution and that exploit-based validation would strengthen score interpretation.Source: §2.3 · p. 2

2.4 Research Questions

• The research questions ask whether correct trajectories accumulate debt, whether same-task runs differ, how debt enters or leaves trajectories, and whether flagged debt can be repaired without losing progress.Source: §2.4 · p. 2

3 The SDLI Metric

• With constant risk 1 and observed progress 0, 0.8, 0.4, 0.8, SDLI is 0.8; the implementation uses a historical-maximum progress envelope, and recovering previously achieved progress adds no charge.Source: §3 · p. 3
• SDLI is cumulative progress-weighted scanner exposure, not current risk; risk appearing after progress stops contributes zero, and removing findings does not refund prior charges.A lower-risk path lowers SDLI only when the reduction occurs at checkpoints that add progress; the paper recommends retaining the risk trace and final reading too.Source: §3 · p. 3
• At each checkpoint, the risk reading counts CWE classes flagged by at least the chosen number of tools once, weighted by the most severe rating reported for each class.Severity weights are 3.0 for high or critical, 1.0 for medium, and 0.25 for low.Source: §3 · p. 3
• BBCS-union requires one tool, whereas BBCS-agree2 requires at least two tools to report the same normalized class; agreement does not establish independent errors or validated precision.The class-level measure also discards the number of affected sites.Source: §3 · p. 3

4 Benchmarks and Measurement Setup

• The study uses 830 functionally correct SWE-bench runs, 712 ProgramBench final workspaces, and 13 public MirrorCode trajectories.ProgramBench's primary population is 80 runs passing at least 90% of official tests; three MirrorCode runs reach recorded progress 1.0.Source: §4 · p. 3
• SWE-bench and ProgramBench use endpoint scores with a zero accounting baseline, while MirrorCode uses recorded test progress carried into a historical-best envelope.MirrorCode provides 463 code snapshots, but changing test coverage limits comparison with a fixed independently run suite; CodeQL scans selected checkpoints and is interpolated between them.Source: §4 · p. 3
• The endpoint score is not necessarily a lower bound on full SDLI: risk appearing only after progress can yield positive endpoint risk but zero SDLI, while later removal can leave positive SDLI and zero endpoint risk.Only the findings observed across checkpoints necessarily contain the final set under the same scan configuration.Source: §4 · p. 3
• The scanner covers Python, and scan scope, rules, weights, and checkpoint availability constrain measurement and exact reproducibility.CodeQL takes roughly two to five minutes per snapshot in the observed runs; checkpoint-frequency sensitivity and total monitoring overhead were not measured, and some remote registry configurations were not pinned to immutable revisions.Source: §4 · p. 4

5.1 Short-Horizon Patches on SWE-bench

• 3.9% of passing SWE-bench runs had two-tool class agreement, while 14.9% were positive under the recall-favoring union reading.The reported endpoint scans cover issue-fixing tasks in existing projects; the paper does not establish whether agents introduced the flagged code.Source: §5.1 · p. 4, Table 2 · p. 5
• Same-task SWE-bench runs varied: eight of nine hatch-422 runs had agree2 SDLI of 3.0, while one sibling scored zero.The reported variation on vault-cli-111 appears in the union channel, where one run scored zero and another accumulated 3.25.Source: §5.1 · p. 4, Table 3 · p. 5

5.2 Whole-Program Synthesis on ProgramBench

• 21 of 80 mostly working ProgramBench runs (26.2%) had non-zero agree2 endpoint SDLI, versus 66 of 80 (82.5%) under union.After excluding CWE-22, CWE-77, and CWE-939, agree2 remained positive in 5 of 80 runs (6.2%); this is a sensitivity summary, not a validated vulnerability estimate.Source: §5.2 · p. 4, Table 2 · p. 5
• Agree2 prevalence fell from 34.6% across 712 ProgramBench finals to 26.2% above 90% pass rate and 4.2% among near-solved runs.The union reading showed no such gradient (81.0%, 82.5%, 83.3%); changing task and model composition prevents a causal interpretation.Source: §5.2 · p. 4, Figure 2 · p. 5

5.3 Long-Horizon Traces on MirrorCode

• One of three MirrorCode runs reaching recorded progress 1.0 had agree2 SDLI 3.000 and union SDLI 7.014, the benchmark’s highest per-run debt.This was a 7,810-line reimplementation; two of the three runs had zero scores under both channels.Source: §5.3 · p. 5

5.2 Whole-Program Synthesis on ProgramBench

• The results do not support a controlled comparison of task horizons, and the ProgramBench cohorts differ in task type, extraction coverage, models, and passing criterion.The 26.2% rate is not confirmed insecurity; excluding whole CWE classes can also discard genuine defects.Source: §5.2 · p. 4, §5.3 · p. 6

6 A Long-Horizon Case Study

• Replaying the recorded writes and edits reconstructs the gowsdl file after each action, but the official pass ratio remains available only at the end.The replayed final file was byte-for-byte identical to the published workspace; the last valid reading was carried through temporary syntax errors.Source: §6 · p. 6
• The run's program downloads remote XML and parses it, while the reported classes include TLS verification bypass, XML parsing, path handling, and a subprocess call.Three scanners reported overlapping classes, but the rule labels do not establish exploitability.Source: §6 · p. 6
• At action 63, one 1,296-line implementation write introduced every CWE class later reported, and the readings then stayed constant through 53 functional-refinement actions.The agent's comparison harness passed all 11 bundled services, and the official evaluation reached 90.5%.Source: §6 · p. 6
• The agent recreated the reference tool's optional TLS bypass and permissive XML parsing, so preserving its interface can conflict with removing scanner findings.Whether the bypass is a vulnerability depends on the intended trust boundary and deployment.Source: §6 · p. 6
• Across 463 scanned MirrorCode checkpoints, agree2 readings never decreased; six of seven runs carrying agree2 debt showed the pattern at their first scanned checkpoint.These class-level readings do not establish that findings persisted at the same sites, and the traces do not show that sudden entry is universal.Source: §6 · p. 6

7 Is the Debt Repairable Once Pointed Out?

• The repair rewound to the first finding-bearing state at action 63 and tested four feedback arms varying exact versus vague feedback and single attempts versus retries.Candidates were rescanned, compiled, compared on 11 bundled services, and checked by replaying 18 later edits.Source: §7 · pp. 6–7
• The precise-feedback single attempt used defusedxml, an HTTP/HTTPS scheme allowlist, and a changed TLS-context construction while retaining the -i certificate-verification bypass.Thus the changed TLS idiom did not remove that capability.Source: §7 · p. 7
• One vague single attempt changed only the XML parser, while the vague retry arm made additional TLS changes on its second round.One run with one repair model cannot establish general success rates or the advantage of intervening early.Source: §7 · p. 7
• In this case, 16 of 21 initial findings belonged to advisory-heavy classes whose reports may flag legitimate CLI operations and require contextual review.The authors neither treat every such finding as harmless nor assume behavior-preserving repair is impossible.Source: §7 · p. 7
• The precise-feedback repair reduced the measured union reading to 3.25, while an adjusted reading retaining three advisory classes was 7.0; neither score validates exploitability.The generated Go output remained byte-identical on 11 local services, and all 18 later edits replayed successfully.Source: §7 · p. 7, Figure 3 · p. 7

8.1 What Checkpoint Monitoring Adds

• In the gowsdl replay, findings appeared at action 63 and persisted through 53 refinement actions, so checkpoint monitoring offered earlier visibility and localization, not additional detection coverage.The large-population measurements are endpoint readings, and the study did not quantify transient findings, repair cost, or whether earlier feedback improves repair success.Source: §8.1 · p. 8

8.2 Adoption Paths for SDLI

• Proposed SDLI uses include an IDE risk gauge, a stop condition for autonomous loops, and a post-training reward signal, but none is established as an effective intervention.A stop policy must also check current risk because SDLI ignores increases after progress plateaus; scanner-only rewards may favor detector evasion.Source: §8.2 · p. 8

8.3 Future Work

• Future studies should cover more tasks and repair models, extend language coverage, and evaluate human review of advisory findings at their point of entry.Replaying more ProgramBench runs supplies risk traces, but full SDLI also needs reliable intermediate progress.Source: §8.3 · p. 8

8.4 Limitations

• Population findings are SAST signals, not agent-attributed vulnerabilities: the study lacks manual or exploit-based validation and did not identify whether findings predated agent changes.Transcript extraction may omit code needed to assess a flow.Source: §8.4 · p. 8
• The Python-focused oracle, endpoint-heavy measurements, heterogeneous checkpoints, and repeated tasks and models limit coverage and comparability across the selected artifacts.The high-scoring gowsdl repair used one model and checked compatibility on 11 bundled services, not hidden tests; fewer scanner matches do not prove fewer exploitable defects.Source: §8.4 · p. 8

• Endpoint measurements show that working solutions can carry SAST findings, while MirrorCode and a reconstructed ProgramBench case expose persistence over recorded states.Source: §9 · p. 8
• Same-task variation motivates studying lower-risk alternatives but does not establish that agents can be steered toward them.Source: §9 · p. 8
• The repair case preserves tested local behavior while reducing scanner readings, but equivalent code can evade pattern matching.Source: §9 · p. 8
• Validating findings, attributing them to changes, and comparing checkpoint intervention with final-state scanning remain necessary for stronger operational meaning.Source: §9 · p. 8

• Public artifacts include SDLI scripts, per-trajectory BBCS scans, the case study replay, repair experiment prompts and candidate files, and intermediate artifacts archived on Zenodo.Source: Reproducibility · p. 8
• The configurations specify Bandit 1.8.0, Bearer 1.49.0, and CodeQL 2.20.0.Source: Reproducibility · p. 8
• Semgrep is recorded as 1.101.0 for SWE-bench and MirrorCode and 1.170.0 for ProgramBench.Source: Reproducibility · p. 8