Source-linked AI summary
How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making
Shubhra Mittal
TL;DR
Production agents remain unreliable on long, multi-step workflows despite strong benchmark performance. This study measures horizon-resolved success across controlled tasks and models, finding geometric decay and a step-count effect rather than relief from shorter context.
Problem
Production agents remain unreliable on long workflows even as benchmark success rises, and the study asks how success changes with task horizon and whether step count or context length drives degradation.
Source: §1 · pp. 1–2Method
The study evaluates four oracle-verified tool-using task families across nine models and compares context regimes designed to separate context length from step count.
Source: §3.1 · p. 4, §3.2 · p. 4Results
28 of 36 model-task cells favor geometric decay; all nine models collapse to a small fraction of starting agentic-task performance by horizon 16, and compressed context steepens rather than eases decay.
Source: §4.1 · p. 6, §4.2 · p. 6, §4.5 · p. 8Takeaways & Limitations
Teams should measure per-step reliability on production-relevant tasks and report success by horizon rather than relying on aggregate pass rates.
Source: §6 · p. 10The tasks are synthetic, so ecological validity for fully open-ended real-world workflows remains unresolved; API limits also make the reported horizon-16 agentic collapse a lower bound on severity at longer horizons.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Production deployments of large language model (LLM) agents remain unreliable on long, multi-step workflows even as benchmark success rates climb steadily. We argue this gap is largely an artifact of task horizon: benchmarks are dominated by short-to-medium horizons where success remains high, while production workloads demand an order of magnitude more dependent steps. We measure the effect directly, characterizing the shape of agent degradation and disentangling its cause across a large controlled study spanning nine models, six open models from 1.2B to 671B parameters, and three deployed proprietary systems; four task families, including a genuinely agentic tool-use loop; five horizons; and three context regimes. Task success follows a geometric law governed by a single per-step reliability parameter, which rises with model scale but saturates well below 1 even for the strongest models, guaranteeing eventual collapse at sufficiently long horizons. The effect is sharpest on the agentic task, where every model tested, including widely deployed systems, falls from near-perfect success to near zero within sixteen steps of (n=10,664 analyzed trajectories. Degradation is driven by step count rather than context length: bounding the context window steepens decay rather than easing it (logit slope -0.69 vs. -0.44), p=3x10-6), contradicting a lost-in-the-middle explanation and warning against a common production shortcut. Projecting measured reliability onto representative benchmark horizons quantifies a substantial gap between benchmark and production conditions, from 0.42 at GAIA-length horizons to 0.24 at hundred-step production horizons. For teams responsible for agent orchestration and reliability at scale, these results argue for horizon-aware evaluation and reliability budgeting in place of aggregate pass-rate metrics. Code, prompts, seeds, and raw trajectories are released.
1. Introduction
Production agents can pass short benchmark tasks yet remain unreliable on long, dependent workflows. This study tests whether success declines geometrically with task horizon and whether step count or context length drives that decline.
1. Introduction
- Production agents remain unreliable on long, open-ended workflows even as benchmark scores and the horizons of tasks agents can complete rise.The paper frames this as a gap between benchmark performance and the demands of live deployments.Source: §1 · p. 1
- A task horizon is its number of dependent steps; if each step succeeds independently with probability r, success is approximately r^H and declines geometrically as H grows.The paper argues that benchmarks mostly sample short-to-medium horizons, while production workflows often require an order of magnitude more steps.Source: §1 · p. 2
- The study measures success against horizon and separates step count from context length across controlled tasks, finding geometric decay in 28 of 36 model-task cells.The study includes an agentic tool-use loop and reports that collapse on this task is severe enough to affect how reliability is budgeted.Source: §1 · p. 2
2. Related Work
Prior agent benchmarks and evaluation work have largely emphasized aggregate success, while related context-length studies examine different degradation mechanisms. This paper positions horizon-resolved measurement as a way to study reliability and distinguish step-count effects from context effects.
2. Related Work
- Most agent-benchmark studies report aggregate single-attempt success on fixed task sets and rarely resolve performance against horizon.This study instead fixes the models and measures decay across horizons in a controlled environment.Source: §2 · p. 2
- Evaluation-rigor work calls for reporting more than a single pass rate, including variance, compute budgets, and the effects of inference-time compute.The paper notes that, to its knowledge, this literature does not make horizon-resolved success its primary axis.Source: §2 · p. 3
- Long-context and compositional-reasoning studies examine degradation with input length or task complexity, while this study tests context length against step count directly.Prior work includes long-input effects [16], compositional complexity [4], and single-to-multiturn drops [11]; the study’s reported opposite result is that bounding context worsens performance rather than improving it.Source: §2 · p. 3
- Unlike long-context retrieval probes, the compressed regime removes information generated earlier by the agent, testing loss of its own reasoning scaffold.The paper distinguishes this from whether a model can locate a fact in a long document.Source: §2 · p. 3
- The agentic task uses an unmodified ReAct loop so measured degradation reflects a control loop practitioners deploy rather than a bespoke structure.The paper identifies ReAct-style interleaved reasoning and acting [1] and verbal self-reflection [2] as dominant current control loops.Source: §2 · p. 3
3. Method
The study uses oracle-verified tool-using tasks, paired context regimes, and a nine-model evaluation to measure performance across horizons. It applies paired comparisons and model selection to distinguish degradation patterns and releases the materials needed to reproduce its analyses.
3.1 Task families
- Four synthetic tool-using task families pair each agent loop with a ground-truth simulator, enabling exact success verification without LLM-judge noise.Ledger tests arithmetic working memory, Refchain long-range retrieval, Cipher procedural execution, and ToolQA requires self-directed inspection calls along a hidden chain.Source: §3.1 · p. 4
3.2 Context regimes: disentangling step count from context length
- At fixed horizons, the natural, compressed, and padded regimes hold operation counts equal while varying context length or turn structure.The same task instance repeats across regimes, enabling paired comparisons.Source: §3.2 · p. 4
3.3 Models, design, and metrics
- The evaluation covers nine instruction-tuned models from six vendor families, including six open models from 1.2B to 671B parameters and three deployed proprietary systems.The combined streaming and agentic study analyzed 10,664 trajectories; agentic temperature was fixed and seeds recorded.Source: §3.3 · pp. 4–5
3.4 Statistical procedure
- Regime comparisons use paired differences; per-cell success intervals use Wilson scores, and AIC selects among geometric, threshold, and linear decay shapes.The paired task instances increase statistical power, while Wilson intervals avoid normal approximations near success rates of 0 or 1.Source: §3.4 · p. 5
3.5 Data and code availability
- Code, task generators, prompts, seeds, the raw dataset, and analysis scripts are publicly available to reproduce the reported figures and numbers.The repository includes an evidence trail mapping reported numbers to their sources in the released data.Source: §3.5 · p. 5
4. Results and Discussion
Across the tested tasks, success generally declines with horizon, often geometrically, while model and task properties shape the rate of decay. Context-window compression does not ease the decline, and projections show lower success at production-length horizons.
4.1 Degradation follows a geometric law
- The geometric form best fits 28 of 36 model-task cells, while threshold or cliff shapes fit the remainder; non-trivial tasks have per-step reliability below 1.Success declines monotonically with horizon wherever a task is non-trivial for the model, and task structure influences the decay rate.Source: §4.1 · pp. 5–6, Figure 1 · p. 5
4.2 The agentic task: deployed models collapse within sixteen steps
- On the agentic toolqa task, every model collapses to a small fraction of its starting performance by horizon 16, despite eight of nine models starting near ceiling.Qwen2.5-72B falls from perfect success at horizons 2 and 4 to roughly one in eight trials at horizon 16.Source: §4.2 · pp. 6–7, Table 2 · p. 6
4.3 Per-step reliability rises with scale but never reaches one
- Per-step reliability rises with parameter count (Pearson r=+0.36 vs. log10(params)), but no tested model reaches perfect per-step reliability on a non-trivial task.Deployed proprietary models cluster alongside the strongest open models rather than exceeding them by a wide margin.Source: §4.3 · p. 7
4.4 Degradation accelerates within a trajectory
- Mean per-step accuracy falls from 0.58 in the first third of trajectories to 0.44 in the last third, and agents rarely recover after their first error.This within-trajectory decline means the observed hazard rises rather than remaining constant as a pure geometric model assumes.Source: §4.4 · p. 7
4.5 The driver is step count, not context length
- Compressed context steepens decay: the logit slope per horizon doubling is -0.69 for compressed versus -0.44 for natural (p=3x10-6 vs. natural).Padded is -0.40 (p=0.51), statistically indistinguishable from natural; single-prompt delivery decays at essentially the natural rate from a lower starting point.Source: §4.5 · p. 8, Figure 2 · p. 8
4.6 Mechanism
- Format and tool-call drift affects 21% of trajectories and rises with horizon, adding an interface-level failure mode alongside reasoning failures.The paper argues that step-level verification can catch failures earlier and more cheaply than waiting for end-to-end task failure.Source: §4.6 · pp. 8–9
4.7 The benchmark gap, quantified
- 0.42 at GAIA-length horizons (8 steps) declines to 0.24 at hundred-step production horizons in projections using mean measured per-step reliability r=0.61.The projections also give 0.36 at WebArena-length horizons (15 steps), 0.33 at tau-bench-length horizons (20 steps), and 0.30 at SWE-bench/OSWorld-length horizons (30 steps).Source: §4.7 · p. 9
5. Limitations and Threats to Validity
The study’s controlled, synthetic tasks support exact oracle verification, but leave transfer to fully open-ended real-world workflows unresolved.
5. Limitations and Threats to Validity
- The synthetic tasks enable exact oracle verification and a clean, controllable horizon, but broader validity for fully open-ended real-world workflows remains unresolved.The agentic toolqa family addresses the concern that the design is not genuinely agentic, but does not settle broader ecological validity.Source: §5 · p. 9
- The agentic API budget ended before the longest horizon, making the reported collapse at H=16 a lower bound on severity rather than a ceiling.Three small models were also excluded because hosted routes failed to adhere to the structured response protocol.Source: §5 · p. 9
- The nine-model sample cannot establish that every current or future model follows the same geometric form, and reliability was measured at a fixed, moderate decoding temperature.Temperature and other decoding-parameter sensitivity remains for future study.Source: §5 · p. 9
6. Conclusion
The paper finds that agent success decays geometrically with horizon, with per-step reliability remaining below perfect even at scale. It recommends horizon-aware evaluation and conservative reliability budgeting for production systems.
6. Conclusion
- Task success follows a geometric form governed by per-step reliability, which rises with model scale but saturates below perfect reliability on non-trivial tasks.The paper describes long-horizon collapse as a structural consequence of compounding.Source: §6 · p. 9
- The effect is sharpest on the agentic tool-use loop, where every tested model collapses within a modest number of self-directed steps.Bounding context steepens rather than eases decay; the paper attributes degradation to step count rather than context length.Source: §6 · p. 9
- The authors recommend measuring per-step reliability on production-relevant tasks, budgeting conservatively, and reporting benchmark success by horizon rather than as one aggregate.Code, task generators, prompts, seeds, raw trajectories, and analysis scripts are released for independent tracing and reproduction.Source: §6 · p. 10
- For production teams, the achievable horizon should be treated as a function of measured per-step reliability rather than benchmark scores.The paper notes that the hazard accelerates over a trajectory.Source: §6 · p. 10