Source-linked AI summary
When Agents Implement Systems: A Case Study in Defects, Detection, and Evaluation Rigor
Phanindra Reddy Madduru
TL;DR
The paper asks how reliably LLM coding agents handle systems-level requirements during end-to-end implementation, where existing evaluations provide limited evidence. It presents an observational case study of one agent building a specified multi-component data system, catalogs five defects, and benchmarks a prescribed retrieval-filtering trade-off. Filtered retrieval reached its construction-based ceiling quickly, whereas unfiltered search recovered all required evidence only 69% of the time at budget 10; the study also documents an unverified performance fix.
Problem
Existing evaluations provide limited evidence about agent behavior when implementing multi-component systems under systems-level requirements such as schema consistency, asynchronous orchestration, and retrieval filtering.
Method
The paper observationally analyzes one LLM agent’s end-to-end implementation against a fixed specification, catalogs five defects, and evaluates candidate filtering versus unfiltered search using HotpotQA gold entity labels.
Results
Unfiltered search reached only 69% exact match at k = 10, while filtered retrieval reached 100% by k = 5; the gap was significant at every tested budget.
Takeaways & Limitations
The case study supports execution-based evaluation and independent re-measurement of agent fixes rather than relying solely on static checks or self-reported engineering claims.
Takeaways & Limitations
The study is a single-session, single-agent case study, and its retrieval ablation isolates filtering by substituting gold entity labels for query understanding.
Abstract
from arXiv · showhide
As LLM coding agents increasingly perform end-to-end engineering work, we lack empirical characterization of how they behave on systems-level requirements: schema design, async orchestration, configuration correctness, and retrieval-filtering trade-offs. We present a case study of one such agent implementing a multi-component data system against a detailed pre-existing specification. Storage technologies, schema, entity-resolution algorithm, and retrieval-filtering strategy were fixed in advance; the agent autonomy was in the implementation, in diagnosing and fixing defects it introduced, and in interaction-design choices left open. Over a single session, we catalog five such defects, categorized by constraint violated and detection method. We further evaluate, on the public HotpotQA benchmark, the one retrieval trade-off specified in that architecture: restricting candidates to a graph-identified entity set before ranking versus unfiltered search. We substitute the benchmark gold evidence labels for entity identification, since we lacked LLM access to run that stage, and report standard recall rather than the benchmark own accuracy metrics. Across retrieval budgets from 1 to 10 and 100 questions against a pooled corpus of 2994 paragraphs, filtered recall reaches its ceiling by a budget of 3, expected once candidates are restricted to the gold paragraphs themselves, while unfiltered search recovers all required evidence only 69 percent of the time even at a budget of 10, a gap that holds at every budget tested, with sign test p less than 0.0001. We close with a discussion of where the agent autonomy succeeded versus required correction, including one instance where a claimed performance fix was never re-measured on the regression that motivated it.
1 Introduction
The paper studies whether an LLM coding agent can reliably implement a specified multi-component system while introducing, diagnosing, and fixing systems-level defects. It catalogs these defects and evaluates one prescribed retrieval-filtering trade-off rather than proposing a new retrieval method.
- Existing evaluations largely test bounded issue resolution or tool-assisted task success, not end-to-end systems implementation against a detailed specification.
- The case study shifts evaluation toward schema consistency, idempotent storage, configuration resolution, and retrieval-filtering precision under systems-engineering constraints.
- A single LLM coding agent implemented a multi-component data system, and five self-introduced defects were later diagnosed and fixed.
- The paper evaluates graph-identified candidate filtering versus unfiltered semantic search on a public external benchmark.
- Graph-guided retrieval filtering is established; the paper contributes a defect catalog, a reproducible trade-off evaluation, and analysis of an unverified performance fix.
2 Case Study System and Methodology
The target is a four-part data pipeline whose technologies and core design choices were fixed in advance, leaving implementation and interaction details to the agent. The study reconstructs defects observationally from session artifacts and observable symptoms.
- The system combines a graph store, metadata-filtered vector index, asynchronous ingestion pipeline, and force-directed graph-rendering client.
- The ingestion pipeline performs extract, resolve, embed, and link stages through a task queue and must remain idempotent under repeated execution.
- The design document fixed storage technologies, schema, entity-resolution algorithm, and retrieval-filtering strategy, while implementation remained autonomous.
- The observational methodology logs each defect’s code diff, symptom, diagnostic path, and fix using artifacts produced during development or later use.
3 Observed Defects, Categorized by Systems Constraint
Five defects spanned four systems constraints, with correctness failures appearing in both storage and rendering. The cases show that some failures escaped existing tests and required runtime or visual probing.
- Five defects spanned four systems constraints, including two correctness defects at storage and rendering layers.
- A traversal inconsistency returned edge endpoints absent from the downstream node set, causing a runtime rendering crash.
- The consistency defect escaped type checks and existing tests because they asserted edges without testing the joint node-edge invariant.
- The same consistency pattern was reintroduced at a second call site before the invariant was made explicit and tested directly.
- A zoom-dependent canvas-label formula produced screen-filling text after its floor was re-multiplied by the rendering transform.
- The rendering defect was caught only through automated screenshot comparison during unrelated interaction testing, not through static checks or assertions.
4 A Specified Retrieval Trade-off: External Benchmark Evaluation
The paper externally benchmarks filtered versus unfiltered retrieval on HotpotQA, isolating whether restricting candidates to graph-identified entities improves evidence recovery. Filtered retrieval reaches its design-implied ceiling quickly, while unfiltered search remains substantially incomplete even at larger budgets.
- Retrieval setup: The benchmark compares graph-identified candidate filtering before ranking with full-corpus embedding-similarity ranking.The filtered condition narrows search using structural information, while unfiltered retrieval represents standard RAG-style ranking.
- Benchmark design: HotpotQA distractor paragraphs stress the trade-off because they are lexically similar to the gold evidence.The evaluation reuses raw paragraphs and gold supporting-fact labels rather than reproducing HotpotQA’s answer or supporting-fact leaderboard metrics.
- Benchmark design: 2,994 paragraphs from 300 validation questions form the shared corpus, with the first 100 questions used for evaluation.Each paragraph’s Wikipedia article title serves as its entity identifier, avoiding an additional LLM extraction stage.
- Evaluation measures: Recall@k is reported for k ∈{1, 3, 5, 10}, alongside exact recovery of both gold paragraphs and a paired exact sign test.The evaluation measures recovery of HotpotQA’s annotated supporting-fact paragraphs rather than the benchmark’s native answer metrics.
- Results: 69% exact match at k = 10 is the unfiltered ceiling, while filtered retrieval reaches 100% by k = 5 and differs significantly at every tested budget.Every sign test rejects the null of no paired difference at p < 10^-4; filtered recall@1 is 0.500 because ranking still matters within the two-paragraph pool.
5 Discussion
The discussion contrasts cases where the agent made structural corrections with failures requiring human detection or stronger evaluation. It also bounds the conclusions to a single agent session and an isolated retrieval decision.
- Where autonomy succeeded: In two of five defects, the agent inferred root causes from single symptoms and applied structural fixes rather than local patches.These fixes centralized configuration-path resolution and generalized endpoint backfill after the same pattern appeared at another call site.
- Where correction was required: Interactive testing by a human exposed an uncatalogued visualization failure that locally correct operations produced through repeated refetching and rerendering.The failure was emergent rather than attributable to any single incorrect operation.
- Evaluation rigor: 242-second latency on an 18-chunk document motivated a concurrency fix, but the agent validated it only on a smaller document.The discussion characterizes this as conflating architectural soundness with verification on the regression that motivated the fix.
- Scope and limitations: The case study is limited to one session and one agent, while the retrieval ablation substitutes gold entity labels and isolates filtering from entity-identification quality.The paper makes no claim that the defect frequencies generalize, including to smaller, more efficient models.
6 Conclusion
The conclusion argues that systems-level agent evaluation must include execution-based probing and independent remeasurement, not only static checks or self-reported fixes. The retrieval comparison is preserved for reproducibility.
- Evaluation implications: Several cataloged defects lacked static or type-level signatures and were caught only through empirical probing.This supports execution-based evaluation of agentic systems work alongside static analysis.
- Evaluation implications: The unverified-fix case and isolated retrieval benchmark expose a shared gap: the development loop did not force regression remeasurement or external validation.The paper retains the ablation script and results with its source so the retrieval comparison can be independently rerun.