Source-linked AI summary
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Nicholas Roberts, Frederic Sala, Aws Albarghouthi
TL;DR
Existing agentic coding benchmarks provide limited evidence about how design decisions affect future extensions because they emphasize single-shot correctness. SlopCodeBench evaluates agents as they repeatedly extend their own solutions under changing specifications, measuring structural erosion and verbosity alongside correctness. Across trajectories, code quality degrades: agents pass checkpoints but increasingly produce eroded and redundant code, while prompting improves initial quality without slowing degradation.
Problem
Single-shot benchmarks focus on correctness rather than whether agent-generated code remains extensible under iterative change.
Method
SlopCodeBench evaluates 36 problems across 196 checkpoints as agents extend carried-forward workspaces under changing specifications, measuring structural erosion and verbosity.
Results
Across trajectories, structural erosion rises in 77% and verbosity in 75.5%; agent code is 2.3× more verbose and 2.0× more eroded than code from 473 open-source repositories.
Takeaways & Limitations
Prompting can improve initial code quality, but it does not stop iterative degradation.
Takeaways & Limitations
When an agent fails or crashes mid-problem, remaining checkpoints receive correctness zero, while missing quality measurements are excluded rather than imputed.
Abstract
from arXiv · showhide
Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark of 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that agents pass checkpoints while producing code that erodes and bloats with each turn.
1 Introduction
SlopCodeBench addresses the limited ability of single-shot benchmarks to measure whether agent-generated code remains extensible under changing requirements. It evaluates iterative extensions while tracking correctness, verbosity, and structural erosion.
- Single-shot coding benchmarks measure correctness for a complete specification, not whether code remains extensible under future change.
- SlopCodeBench evaluates 36 iterative software-development problems across 196 checkpoints, with evolving specifications and unconstrained internal structure.
- 14.8% was the state-of-the-art checkpoint pass rate, and no problem was fully solved end-to-end.
- Structural erosion rose in 77% of trajectories and verbosity in 75.5%.
- Agent code was 2.3× more verbose and 2.0× more eroded than code from 473 open-source repositories.
2 SlopCodeBench
SlopCodeBench carries each agent’s workspace across evolving checkpoints while withholding internal interfaces, tests, and implementation-language constraints. It measures correctness alongside two independent code-quality failure modes: structural erosion and verbosity.
- Design Principles: SCBench specifies observable CLI or API behavior without prescribing internal interfaces, forcing agents to make architectural decisions.
- Design Principles: Agents see specification prose and examples but never the visible test suite or its feedback.
- Design Principles: Problems are black-box and language-agnostic, with outputs evaluated through CLI or API interfaces and normalized where needed.The paper evaluates the Python track because of cost.
- Task Formulation: Each problem is an ordered sequence of checkpoints in which the agent updates its previous workspace from the current specification.The initial workspace is empty, and later checkpoints build on prior code without prior conversational context.
- Measuring Code Quality: Structural erosion measures complexity concentrated in already-complex functions, while verbosity measures duplicated and rule-flagged redundant code patterns.Erosion uses complexity mass based on cyclomatic complexity and source size; verbosity uses targeted AST-based patterns with deduplicated line counts.
- Evaluating Solutions: Evaluation includes core, error, functionality, and regression tests, with regression tests carrying prior-checkpoint requirements forward.
3 Iterative Evaluation of Coding Agents
Across iterative checkpoints, coding agents struggle to preserve correctness and code quality as their own solutions evolve. Quality degradation is widespread, exceeds that observed in human repository histories, and persists despite prompting that improves initial quality.
- Solve rates and cost: No configuration passes every checkpoint for any problem, while each checkpoint’s cost rises without corresponding correctness improvements.GPT 5.5 achieves the highest strict solve rate at 14.8%, and no agent fully solves any of the 36 problems.
- Solve rates and cost: Core pass rate falls from 64.6% to 35.5% as the problem progresses, while functionality pass rate changes by less than six points.The gap between core and isolated pass rates widens from 2.5× to 5.4×; error pass rate drops from 80.1% to 62.2%.
- Solve rates and cost: Mean cost per checkpoint grows 2.2× from the start to the end, while mean relative lines changed fall from 97.4% early to 29.5% late.Total recorded token accounting is 13.18B tokens across 196 checkpoints.
- Trajectory degradation: Structural erosion increases in 77% of trajectories and verbosity in 75.5%, with highly complex functions becoming more numerous and more complex.Functions with at least 10 cyclomatic complexity rise from 3.6 to 23.7, while mean maximum cyclomatic complexity rises from 27.5 to 69.0.
- Comparison with repository histories: Agent checkpoints are 2.3× more verbose and 2.0× more eroded than human repository histories, with agents accumulating both issues faster.Agent verbosity and erosion grow roughly 7× and 5× faster than human medians, respectively.
- Prompting effects: Anti-slop and plan-first prompting reduce initial quality problems but raise checkpoint cost by 12.1% on average and have little impact on iterative degradation.Anti-slop lowers verbosity and erosion across several models, while strict performance drops by 2.4 percentage points on average for anti-slop and 3.6 points for plan-first.
4 Related Work
Prior coding benchmarks emphasize single-shot correctness or constrained iterative settings, leaving code-quality degradation under realistic chained extensions insufficiently measured.
- Single-shot benchmarks evaluate correctness against fixed specifications, so pass rates can miss defect categories that tests do not exercise.
- Prior iterative benchmarks often sever accumulation through clean states, isolated modifications, or unchained long-horizon framing.
- Repository-history benchmarks preserve evolution but study maintenance or narrow feature additions in existing codebases and inherit contamination risk.
- Research on code smells reports higher per-line cyclomatic complexity in LLM code, while repeated modification can produce structural drift and defects missed by tests.
5 Conclusion
SlopCodeBench evaluates agents forced to make autonomous design decisions across evolving specifications, finding low end-to-end success and consistent degradation that prompting does not stop.
- No agent solves any of the 36 problems end-to-end, and the best solve rate is 14.8% of 196 checkpoints.
- When forced to iterate, agents consistently degrade their codebases across trajectories.
- Agent checkpoints show higher erosion and verbosity than sampled Python repository histories, while accumulating both metrics faster.
- Prompting interventions improve initial quality issues but cannot stop the eroding behavior.
A Benchmark Details
The benchmark specifies observable CLI behavior while progressively expanding a Python code-search task from exact and regex matching to language-aware structural patterns.
- Checkpoint 1: Checkpoint 1 requires an executable that scans Python files and emits one JSON object per match as JSON Lines.
- Checkpoint 1: Rules initially support exactly exact and vanilla regex matching, with optional Python language selection and restricted regex flags.
- Checkpoint 1: The initial interface accepts a root directory, a JSON rules file, and an optional encoding whose default is utf-8.
- Checkpoint 2: Checkpoint 2 extends language filtering to Python, JavaScript, and C++, applying omitted-language rules to all three.
- Checkpoint 3: Checkpoint 3 adds code-like patterns with metavariables, including optional variables, repeated-name consistency, and literal-dollar escaping.
- Checkpoint 3: Pattern-match output adds captures and occurrence ranges, with deterministic ordering by end position and lexicographic capture keys.
B Quality and Erosion Analysis
Quality analysis illustrates structural erosion through concentrated complexity and evaluates whether erosion metrics predict later performance or depend on metric choices.
- Opus 4.7’s forge implementation grows 7× in cyclomatic complexity, from CC 13 to CC 92, across eight checkpoints.The increase is monotonic across every checkpoint.
- The same implementation expands from 38 to 240 lines as argument handling shifts from argparse to duplicated manual branches.
- Across nine erosion variants, predictive correlation with next-checkpoint pass rate stays near zero, while correlation with next-checkpoint cost remains positive.
- The erosion-family conclusion does not depend on the exact high-CC threshold or size term, although LOC and maximum CC remain stronger raw cost predictors.
C.1 Environment and Invocation
The evaluation standardizes model execution through compatible harness versions, shared container tooling, and repeated checkpoint resets. The forge example illustrates how later extensions can replace compact parsing with duplicated branching.
- For older models, the evaluation uses the earliest compatible publicly available harness version, or the nearest later compatible version when necessary.
- The container installs required languages and a shared tooling baseline, excluding commands that failed on only one harness.
- Installed packages, shell history, and agent session data reset between checkpoints, while experiments evaluate only the Python track.
- Claude Code and Codex are invoked directly in headless mode, with model-specific versions listed in Table 7.
- Opus 4.7’s forge solution replaces argparse with repeated manual branches, illustrating structural accumulation during extension.The later implementation repeats paired --X val and --X=val branches across seven flags before nested dispatch.
C.3 just-solve (Baseline)
The baseline asks agents to solve each specification completely while continuing from their prior code at later checkpoints. Comparison prompts additionally target code quality, planning, and avoidance of common verbose or heavily nested patterns.
- just-solve (Baseline): The minimal baseline injects each checkpoint specification verbatim and marks only later checkpoints as continuations.Agents receive a Jinja system-prompt template; the continuation flag is false initially and true thereafter.
- just-solve (Baseline): The just-solve prompt instructs agents to implement a program that 100% solves the specification, with virtual-environment and dependency requirements.
- just-solve (Baseline): All prompt variants retain the continuation workflow, including reuse of the virtual environment and updating requirements when extensions need dependencies.
- just-solve (Baseline): The anti-slop prompt explicitly discourages verbose patterns, defensive over-engineering, unnecessary abstractions, heavy nesting, if/else ladders, and excessive helper methods.
- just-solve (Baseline): The plan-first prompt requires planning before coding, writing a simple solution, checking correctness and edge cases, and refactoring for quality.
D.1 Human Panel construction
The human comparison uses a stratified temporal panel of 473 maintained Python repositories and tracks quality across sampled commit histories. Human code generally degrades less often and more slowly than agent trajectories, with modest post-ChatGPT-era shifts.
- Human Panel construction: 473 maintained Python repositories span web frameworks, scientific computing, infrastructure, machine-learning libraries, and command-line utilities.The sample is stratified into four GitHub star tiers and contains 13667 checkpoints from up to 30 source-touching commits per repository.
- Human Panel construction: Pre-ChatGPT median verbosity is 0.155 versus 0.166 post-ChatGPT, a small shift relative to the agent mean of 0.44.
- Human Panel construction: Among 321 repositories spanning both eras, median within-repository shifts are +0.002 for verbosity and +0.014 for erosion.55% become more verbose and 59% more eroded post-2024, but the signal is described as dwarfed by the agent gap.
- Human Panel construction: 68% of repositories with at least five sampled commits end with higher verbosity, with median first-to-last growth of 36%.
- Human Panel construction: Human per-checkpoint slopes are 0.0022 for verbosity and 0.0053 for erosion, versus 0.0144 and 0.0264 for agent trajectories.
- Human Panel construction: Human repositories include prominent verbosity outliers such as huggingface/transformers and fastapi/fastapi, while mature projects such as pallets/flask show low verbosity.