Source-linked AI summary
Evaluating Agentic Code Repair Capabilities in Distributed Systems
Yibo Yan, Huijuan Wang, Junzhou He, Yizhuo Liang, Shaoyu Wang, Huanchen Sun, Seo Jin Park
TL;DR
Distributed-system code repair lacks a dedicated benchmark and controlled evidence on debugging context. DDBench addresses both gaps, finding that context produces asymmetric gains: weaker models improve pass rates while stronger models reduce costs, though poorly targeted context can mislead.
Problem
LLM evaluation lacks both a source-level code-repair benchmark for distributed-system bugs and controlled evidence on how debugging context affects agent success.
Method
DDBENCH evaluates 60 historical bugs from 13 open-source distributed systems across three difficulty tiers and matched symptom-only versus context-augmented conditions.
Results
Debugging context yields asymmetric gains: weaker models improve pass rates, whereas stronger models reduce costs, including 69% lower token consumption for Opus 4.6.
Takeaways & Limitations
DDBENCH exposes distributed reasoning differences and shows that well-curated debugging context can improve performance and efficiency while reordering model rankings.
Takeaways & Limitations
The study uses only mini-swe-agent and lacks a controlled cross-agent-scaffold replication separating agent design from model capability.
Abstract
from arXiv · showhide
LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified. Distributed-system debugging, however, remains an under-explored regime: bugs span processes, nodes, and protocol interactions, with root causes rarely recoverable from source alone and brute-force exploration intractable across non-deterministic interleavings. This leaves two gaps in LLM and agent evaluation: no code-repair benchmark targets distributed-system bugs, and no controlled study isolates how much externally provided debugging context changes agent success on them. We introduce DDBench, a code-repair benchmark of 60 historical bugs mined from 13 open-source distributed systems, partitioned into three difficulty tiers. DDBench evaluates every case under two matched conditions: a symptom-only condition where the agent receives only the bug symptom and repository, and a context-augmented condition where it additionally receives a bounded debugging context (logs, traces, runtime state, and targeted code-investigation notes), isolating the effect of debugging context from model capability. The evaluation of ten LLMs on DDBench reveals several findings. First, distributed debugging exercises a reasoning dimension that single-process benchmarks do not surface: models' pass rates span 61 pp, and pairwise bootstrap separates 9 of 15 top-tier model pairs at p < 0.05 on DDBench's hardest case-set. Second, bounded debugging context lifts aggregate pass rate by +18.1 pp, and the lift is asymmetric: weaker models gain pass rate, while stronger models gain efficiency. Third, debugging context requires careful curation, as even faithful debugging context can sometimes mislead LLMs.
1 Introduction
DDBENCH addresses the gap in evaluating agentic distributed-system debugging, where bugs span processes, nodes, and event sequences that single-process benchmarks poorly capture. Its two-condition design measures both repair capability and the effect of externally provided debugging context.
- Motivation: Distributed-system debugging is harder than single-process repair because root causes span processes, nodes, and network interactions across intractable event sequences.Existing single-process repair benchmarks therefore poorly capture distributed debugging capability.
- Benchmark: DDBENCH contains 60 historical bugs from 13 open-source distributed systems across three difficulty tiers.The tiers contain 31, 15, and 14 cases, respectively.
- Method: DDBENCH evaluates each case under symptom-only and context-augmented conditions, making debugging context a controlled experimental variable.The symptom-only condition provides the symptom and codebase, while the context-augmented condition additionally supplies debugging information.
- Findings: 9.7% →48.4% is GLM 4.7’s pass-rate gain with context, while Opus 4.6 reduces token consumption by 69% and steps-to-completion by 41%.Weaker models primarily gain pass rate; stronger models see marginal pass-rate gains but substantial cost reductions.
- Findings: 61 pp and 9 of 15 model pairs distinguish models on DDBENCH’s hardest case set.The evaluation covers 10 proprietary and open-weight LLMs, indicating that distributed debugging exercises a reasoning dimension not surfaced by single-process benchmarks.
2 DDBENCH
DDBench is a 60-case benchmark for repairing historical bugs in 13 open-source distributed systems, organized by reasoning difficulty. It evaluates each case under matched symptom-only and context-augmented conditions to isolate the effect of bounded debugging context.
- Benchmark design: DDBench contains 60 historical bugs from 13 open-source distributed systems, partitioned into three disjoint tiers ranked from highest to lowest difficulty.Each case includes a faulty-commit worktree, symptom description, and hidden PASS/FAIL oracle.
- Debugging context: Debug-context is a bounded, per-case artifact containing realistically obtainable debugging information, including runtime signals, logs, traces, and targeted code investigation.The bundle is treated as a first-class experimental variable and can be repopulated with outputs from alternative debugging tools.
- Evaluation conditions: Each case is evaluated twice: once with only the repository and symptom description, and once with the same agent, tools, and an added debug-context bundle.The paired design compares diagnosis and repair with versus without externally provided debugging information.
- Curation: DDBench combines agent-based mining and artifact synthesis with human validation, sanitization, and acceptance, admitting cases requiring cross-process or interleaving-aware reasoning.Researchers manually review surviving cases and re-check symptom and debug-context sanitization before final admission.
3 Evaluation
DDBench distinguishes models that cluster on single-process repair benchmarks by exposing cross-process reasoning differences. Bounded debugging context substantially improves repair success while shifting benefits from pass-rate gains for weaker models to efficiency gains for stronger models, with effectiveness varying by search-space size and occasional misleading cases.
- Model differentiation: 61 pp separated the 10 models on tier-1 symptom-only DDBench, compared with a 3.8 pp gap between Claude Opus 4.6 and GPT 5.4 on SWE-bench Verified.DDBench’s cross-process reasoning demand exposes capability differences that single-process benchmarks compress.
- Debugging context: +18.1 pp raised aggregate tier-1 pass rate from 32.6% to 50.6% with context; nine models improved, one tied, and none regressed.Across 310 paired runs, context produced 71 FAIL→PASS flips versus 15 PASS→FAIL flips.
- Debugging context: +38.7 pp was GLM 4.7’s pass-rate gain, while Claude Opus 4.6’s token consumption fell 69% without pass-rate improvement.The results divide by baseline capability: weaker models primarily gain success rate, stronger models mainly gain efficiency, and mid-range models gain both.
- Variation by bug and tier: +42.5 pp was the largest tier-1 pass-rate lift, occurring for concurrency hazards where traces and race-detector reports narrow interleaving search.Across tiers, the lift decreased from +14.7 pp on tier-2 to +7.1 pp on tier-3, consistent with tier difficulty proxying diagnostic search-space size.
- Limitations: Debug-context can mislead when the abstraction distance between the supplied observation and the problematic code location is large.When abstraction distance is small, context reduces the search space; the supplied case study examined ZooKeeper-4846, Raft-268, and Dragonfly-6687.
- Agent behavior: 25–45% reductions occurred in per-case steps, first-edit time, tokens, and exploratory reads, while edit count remained unchanged.The narrowing affected diagnostic search rather than the fix itself, and savings persisted regardless of outcome.
4 Limitations and Future Work
The study’s limitations concern its single-agent evaluation scaffold, atomic treatment of debugging context, and initially curated benchmark scale. Future work will replicate across agent scaffolds, ablate context contents, and expand cases across systems and bug families.
- Agent scaffold: The evaluation used only mini-swe-agent, although DDBench supports other repository-editing agents through addons.Released addons already support Claude Code and Codex.
- Debug-context ablation: The symptom-only versus context-augmented comparison treats logs, traces, goroutine dumps, and code-investigation notes as one indivisible bundle.This prevents identifying which component drives gains or how gains scale with the number and type of supplied files.
- Benchmark scale: DDBench currently contains 60 cases selected through a multi-stage quality process rather than a scale ceiling.The released mining protocol is intended to support expansion across additional systems and bug families.
5 Conclusion · A Related Work
DDBENCH makes distributed-system debugging measurable through matched repair conditions that separate model reasoning from the effects of debugging context. It complements existing single-process repair, operational diagnosis, runtime-signal, and distributed-testing research by providing a code-repair evaluation harness for historical distributed bugs.
- 5 Conclusion: Distributed debugging isolates reasoning about cross-process interactions and undeterministic timings, while debugging context measures how additional information changes agent behavior.These are presented as two distinct questions about coding agents.
- 5 Conclusion: DDBENCH evaluates 60 historical bugs under matched symptom-only and context-augmented conditions, exposing cross-process reasoning and debugging-context effects.Its conclusions identify model reasoning across processes and non-deterministic schedules, alongside tooling that can improve performance and reduce cost.
- 5 Conclusion: 61 pp cross-model spread on tier-1 demonstrates that DDBENCH distinguishes distributed-debugging performance across models.The benchmark also reports an asymmetric context lift that can reorder rankings at fixed budget.
- A Related Work: SWE-bench-style benchmarks package historical issue–PR pairs as faulty worktrees with hidden test oracles and score agents by whether patches pass.Their source material is described as imposing structural limits because it comes almost exclusively from single-process tasks.
- A Related Work: IT-operations benchmarks use logs, metrics, traces, and alerts to diagnose distributed deployment issues, but do not evaluate repair of actual distributed systems.They establish cross-service reasoning as a distinct capability axis and often inject controlled faults.
- A Related Work: Most LLM debugging and program-repair work targets single-process benchmarks, shaping agents around read–edit–test loops, localized retrieval, and repository-wide search.A separate strand augments debugging with runtime signals, though the supplied passage ends before specifying its scope.
- A Related Work: Distributed-system failure studies show that difficult bugs involve protocol invariants, cross-node concurrency, and partial failures, motivating specialized testing and model-checking infrastructure.This research provides the failure-mode taxonomy targeted by DDBENCH.
- A Related Work: Runtime-signal datasets provide production and benchmark logs, traces, and telemetry, but require DDBENCH’s debug-context channel to support scored code repair.They lack a paired faulty commit, sanitized symptom description, or reproduction oracle for judging whether an agent’s patch compiles and passes.
B Dataset Details · B.1 Per-Project Coverage Breakdown · B.2 Licenses
DDBench covers 60 cases from 13 open-source distributed systems, organized by implementation language, system family, and difficulty tier. Its permissive licensing and pinned faulty worktrees support reproducible benchmark packaging and redistribution.
- B.1 Per-Project Coverage Breakdown: DDBench spans 13 open-source distributed systems across five languages and six system families.The cases are partitioned into three difficulty tiers: tier-1, tier-2, and tier-3.
- B.1 Per-Project Coverage Breakdown: 60 cases are distributed across the 13 mined projects, with each project annotated by implementation language, system family, and per-tier case count.Table 4 provides the full per-project breakdown.
- B.1 Per-Project Coverage Breakdown: 14 cases come from Dragonfly, while Ra, RabbitMQ, and RocksDB each contribute 1 case.The skew reflects project issue-tracker activity and the frequency of confirmed bugs meeting the benchmark’s cross-process reasoning criteria.
- B.2 Licenses: All 13 projects use permissive terms—Apache 2.0, MPL 2.0, or BSL 1.1—enabling redistribution of packaged benchmark cases.Each case can be distributed as a pinned faulty worktree with a containerized reproduce oracle.
- B.2 Licenses: Table 5 records the license and upstream repository URL for every project in DDBench.The table covers all 13 open-source systems included in the benchmark.
- B.2 Licenses: Each faulty worktree is a pinned commit of the original repository with no edits applied on top.No released case ships modified upstream source; dual-licensed projects rely on Apache 2.0 to relicense derivative artifacts.
C Agent-Assisted Mining Methodology … C.6 Sanitization Prompt
DDBench uses a semi-agentic, specification-driven pipeline to mine, package, verify, and sanitize distributed-system bug cases. Agents provide breadth and debugging artifacts, while automated checks and human review enforce reproducibility, oracle soundness, and answer-leakage prevention.
- C.1 Why Semi-Agentic?: Manual curation does not scale across 13 systems and thousands of issues, while automated scraping cannot reliably identify semantically cross-process, interleaving-aware bugs.The pipeline therefore combines agentic semantic analysis with human judgment rather than relying exclusively on either approach.
- C.1 Why Semi-Agentic?: Agents handle broad, expertise-heavy tasks such as issue mining, symptom drafting, reproducer hardening, and debug-context generation; humans retain acceptance, quality gating, and sanitization.Each agent stage uses a narrow skill specification and is followed by review.
- C.2 Agentic Mining Protocol: The mining protocol proceeds through candidate admission, case-artifact synthesis, hermetic containerization, and pre-merge validation under machine-readable specifications released with DDBENCH.Candidates must satisfy a dual-reasoning contract, and cases failing checks are repaired and re-verified or rejected.
- C.3 Case Containerization Verification Prompt: Hermetic verification runs setup and reproduction inside pinned Docker environments, classifying cases by build and oracle outcomes with explicit timeouts and status categories.The verification prompt runs cases sequentially, enforces 30 minute setup and 5 minute reproduction limits, and reports non-VERIFIED diagnostics.
- C.4 Oracle Verification: Oracle soundness requires FAIL on the buggy commit and PASS after applying the reference solution, with iterative hardening of determinism, timing, and repetition.The reference solution is Markdown rather than a gold patch, allowing any patch that restores the violated invariant and documenting the diagnosis chain.
- C.5 Debug-Context Generation Prompt: Debug-context generation gathers reproducible runtime outputs and bounded code-investigation notes from the buggy container while prohibiting access to solution, benchmark, and injected-test artifacts.Evidence must stop at identifying an area worth further investigation and is checked for reproducibility, injected tests, raw-output fidelity, and fix disclosure.
- C.6 Sanitization Prompt: Sanitization separates externally observable symptoms, runtime-obtainable evidence, and source-dependent fix information into three tiers, then requires zero remaining violations after lexical and semantic rescans.Tier 1 stays in symptom.md, Tier 2 moves to evidence/, and Tier 3 is redacted; human sanitization complements the automated leakage pass.
D Benchmark Features … D.3 DDBENCH
DDBENCH targets distributed debugging through small but reasoning-intensive fixes, a tightly bounded debugging-context channel, and a benchmark design combining executable repair with cross-process reasoning. Its context bundles reflect realistic short triage sessions while distinguishing DDBENCH from single-process repair and diagnosis-only operations benchmarks.
- D.1 Difficulty: Reasoning, Not Patch Size: Median upstream tier-1 fixes add plus remove 8 lines, indicating that DDBENCH difficulty lies in reaching surgical fixes rather than patch size.Typical underlying issues include misplaced ordering, missing fences, or violated invariants.
- D.2 Bounded Debug-Context Channel: 321 tokens is the median debug-context bundle size across 60 cases, with a maximum of 1,403 tokens and at most three markdown files per case.The bundle distribution is heavy-tailed but tightly bounded.
- D.2 Bounded Debug-Context Channel: 3.0 KB is the median bundle size, rising to a maximum of 10.4 KB across the 60 cases.Bundle size is measured alongside estimated tokens and file count.
- D.2 Bounded Debug-Context Channel: 55.0% of cases include code investigation, 41.7% include test output, and 20.0% include crash output.The remaining cases use structured runtime artifacts, and most bundles combine one or two categories.
- D.2 Bounded Debug-Context Channel: Most debug-context bundles combine one or two evidence categories, mirroring artifacts a developer might capture during a short triage session.Categories are not mutually exclusive, so their shares do not sum to 100%.
- D.3 DDBENCH: DDBENCH uniquely combines an executable repair oracle, cross-process bugs, and a controllable runtime-evidence channel.Existing code-repair benchmarks provide executable oracles but restrict workloads to single-process systems, whereas IT-operations benchmarks diagnose from telemetry rather than repair source.
- D.3 DDBENCH: DDBENCH compares benchmarks across cross-process reasoning, code-fix production, controllable debug-context exposure, and language coverage.The comparison includes SWE-bench, Multi-SWE-Bench, DebugBench, and AIOpsLab.
E Benchmark Infrastructure … G.1 Pairwise Model Differentiation on Distributed System Debugging
DDBENCH uses hermetic, resettable containers and oracle-based scoring to evaluate agents under symptom-only and evidence-augmented debugging prompts. Its evaluation shows that distributed-system debugging differentiates models more sharply than SWE-bench Verified.
- E Benchmark Infrastructure: DDBENCH evaluates each case through setup, agent launch, reproduction, and reset operations coordinated by bench.sh.The workflow resolves declarative case metadata and layered Docker images, then runs the agent and correctness oracle against an isolated worktree.
- E Benchmark Infrastructure: Cases share per-project clones and images while retaining isolated worktrees pinned to their respective buggy commits.This design reduces repeated clone and build costs without merging case state across commits.
- E Benchmark Infrastructure: Agents operate in containers with their working directories pinned to case worktrees and selectable runtime addons for Claude Code, Aider, Codex, or mini-swe-agent.The agent command supports controlled launches under chosen information conditions.
- G Benchmark Results: Oracle-based scoring recompiles agent edits and executes each case’s reproduce.sh script in a fresh container against the built worktree.The per-case correctness oracle is isolated from the agent’s execution environment.
- F DDBENCH Prompt: Symptom-only prompts require agents to inspect SYMPTOM.md, explore code, form a root-cause hypothesis, implement a targeted fix, and verify ./REBUILD.sh.They prohibit searching for the specific issue, pull request, or known fix.
- F DDBENCH Prompt: Evidence-augmented prompts add developer-gathered debugging evidence that agents must review early while treating it critically as potentially incomplete.Evidence may include test outputs, logs, stack traces, crash reports, or code-level observations.
- G.1 Pairwise Model Differentiation on Distributed System Debugging: 9 of 15 model pairs were distinguished by DDBENCH at p < 0.05, versus 2 of 15 pairs on SWE-bench Verified.For example, DDBENCH separated Opus 4.6 from Sonnet 4.6 at p < 0.01, whereas SWE-bench Verified yielded p = 0.18.
- G Benchmark Results: Eight of ten models improved under evidence augmentation, while Claude Sonnet 4.6 and GPT OSS 120B tied and none regressed on tier-2.Table 8 reports per-model pass rates on 15 tier-2 cases and defines Δ as percentage-point improvement.
G.2 Pass Rate on Tier-2 and Tier-3
Context augmentation improves pass rates for most models on tier-2 and tier-3, but gains narrow near the tier-3 ceiling. Stronger models often convert context into cost savings, while weaker models gain pass rate, sometimes at higher cost.
- Tier-2: 8 of 10 models improved on tier-2, 2 tied, and none regressed under context augmentation.The ties were Claude Sonnet 4.6 at 73.3% and GPT OSS (120B BF16) at 26.7% in both conditions.
- Tier-2: +33.3 pp was GLM 4.7’s largest tier-2 gain, doubling pass rate from 33.3% to 66.7%.GPT 5.4, Gemma 4 (31B), and Kimi K2.5 each gained +26.7 pp.
- Cost and takeaway: Above 40% were the tier-2 cost reductions achieved by GLM 5, GLM 4.7, and Kimi K2.5.The section’s takeaway is that near-ceiling models realize more benefit as cost reduction, whereas weaker models realize it as pass-rate gains, often at higher cost.
- Tier-3: 6 of 10 models improved on tier-3, 2 tied at 85.71%, and 2 regressed by −7.14 pp each.GLM 5.1’s regression reflected a single case flip at n = 14 within binomial noise.
- Tier-3: 92.86% was the context-augmented pass rate for Claude Opus 4.6, Claude Sonnet 4.6, GPT 5.4, and GPT 5.4 mini.Their convergence reflects the expected ceiling effect because tier-3 cases are predominantly easier.
G.3 Efficiency Gain from Debug Context
Debug context reduces agent search effort broadly across difficulty tiers, with especially large token savings and near-constant reductions in exploratory steps. Distributional evidence indicates this is a broad search-space reduction rather than an effect confined to pathological runs.
- Tokens: 0.88M →0.47M and 0.93M →0.47M: Debug context nearly halves median per-case tokens on tier-1 and tier-2.Tier-3 tokens fall 28%, from 0.95M →0.68M; context-augmented runs consume fewer tokens at every quantile on tiers 1 and 2.
- Agent steps: 23–28%: Median agent steps drop across all tiers, with reductions of −10, −13, and −12 steps on tiers 1, 2, and 3.These reductions align with the previously reported mean of −13.9 steps on tier-1.
- Why the CDF matters: Across tiers: CDFs show a distributional efficiency shift, not merely improvements from a handful of pathological runs.The shift persists across tiers and supports debug context acting as a search-space reducer for agent trajectories.
G.4 Per-Model Step-Count Distributions
Per-model distributions show that context augmentation consistently reduces agent steps across the evaluated models, with larger absolute savings for models that explore more under symptom-only conditions. The strongest model still exhibits a long right tail, indicating that difficult cross-process searches remain challenging despite debugging context.
- Cross-model synthesis: −4 to −25 steps: every evaluated model in the subset shifts left under context augmentation, ruling out a cohort-level effect driven only by frontier models.The reported reductions range from −4 steps for GPT 5.4 to −25 steps for Claude Opus 4.6.
- Claude Opus 4.6: 55 to 30 steps: Claude Opus 4.6’s median falls by ≈45% under context augmentation, yet its right tail extends past 180 steps in both conditions.This is the longest right tail in the cohort, showing that even the strongest model encounters cases dominated by cross-process search.
- GLM 4.7: 77 to 54 steps: GLM 4.7 has the highest medians in the subset under both symptom-only and context-augmented conditions, with a right tail past 190 steps in both.Its long exploratory phase on hard cases is consistent with greater opportunity for context to remove search steps.
- Cross-model synthesis: Larger absolute reductions occur for models with higher symptom-only medians, while the GPT 5.4 family has less exploratory slack to remove.Context is interpreted as removing exploratory steps: Claude Opus 4.6 and GLM 4.7 explore extensively, whereas GPT 5.4 models settle quickly.
G.5 Per-case Paired Outcome in Tier-1 under Debug-Context Augmentation
Debug-context augmentation produces predominantly gains for weak models and tie-passes for strong models across tier-1 cases. However, rare regressions occur when runtime symptoms misdirect the agent.
- Per-case paired outcomes: Weak models show predominantly FAIL→PASS gains; GLM 4.7 flips 13 of 31 tier-1 cases.The paired outcomes are reported across all 10 models and 31 cases.
- Per-case paired outcomes: Strong models are dominated by blue tie-passes rather than gains.Models are ordered by symptom-only pass rate, with strong models in the top rows.
- Per-case paired outcomes: Losses are rare and concentrated in cases where the runtime symptom misdirects, including dragonfly-6687.Dragonfly-6687 is the highlighted example analyzed in §3.3.
G.6 Case Studies: When Debug Context Helps, and When It Misleads · NeurIPS Paper Checklist · 1. Claims
The case studies show that debugging context can expose otherwise unrecoverable signals, reduce expensive search, or mislead agents across abstraction gaps. The checklist states that the paper’s claims accurately reflect its contributions, scope, results, assumptions, and limitations.
- G.6 Case Studies: When Debug Context Helps, and When It Misleads: Three tier-1 cases represent context effects where debugging signals are unreachable from code, search is expensive, or abstraction gaps cause misleading guidance.Cases were selected by cross-model consensus.
- G.6 Case Studies: When Debug Context Helps, and When It Misleads: 9/10 models changed from FAIL to PASS on ZooKeeper-4846 when context exposed a post-recovery signal unavailable through code reading alone.The bug involved stale permission entries after snapshot and in-memory state diverged during crash recovery.
- G.6 Case Studies: When Debug Context Helps, and When It Misleads: 4/10 models changed from FAIL to PASS on Raft-268, with 0 regressions, as context compressed an otherwise tractable interleaving search.The bug was a shutdown-path deadlock involving a channel and an already-stopped goroutine.
- G.6 Case Studies: When Debug Context Helps, and When It Misleads: 225 to 36 steps and first-edit step 174 to 9 illustrate how context reduced Raft-268 search effort for Kimi K2.5.Claude Opus also improved from 58 to 30 steps and from first-edit step 51 to 21 while passing under both conditions.
- G.6 Case Studies: When Debug Context Helps, and When It Misleads: 1 win and 1 loss on Dragonfly-6687 show that faithful context can mislead agents toward a downstream deserialization symptom instead of the serialization race.The race occurs when two writers interleave into a shared byte stream.
- NeurIPS Paper Checklist · 1. Claims: The checklist answer is Yes: the abstract and introduction accurately reflect the paper’s dataset, benchmark infrastructure, evaluation, contributions, and scope.The justification says these claims are supported and discussed in later sections.
- NeurIPS Paper Checklist · 1. Claims: Claims should clearly state contributions, assumptions, limitations, and expected generalization, while matching theoretical and experimental results.The checklist warns that [No] or [N/A] answers may be viewed negatively by reviewers.
2. Limitations … 6. Experimental setting/details
The paper reports its limitations, omits theoretical results, and provides detailed experimental and reproducibility information. It releases the code, dataset, and benchmark infrastructure while identifying future work to broaden evaluation and ablate debugging-context content.
- 2. Limitations: The evaluation is limited to a single agent scaffold, lacks content-level ablation, and uses one curated release; proposed remedies include multiple scaffolds, ablations, and broader systems and bug families.These limitations and future directions are discussed in §4.
- 3. Theory assumptions and proofs: The work includes no theoretical results, theorems, or proofs.Therefore, requirements concerning assumptions and complete proofs are not applicable.
- 4. Experimental result reproducibility: The paper discloses the models evaluated, agent scaffold, and evaluation protocol needed to reproduce its main experimental results.These details are provided in §3.
- 5. Open access to data and code: The authors released the codebase, dataset, benchmark infrastructure, and semi-agentic mining tools, with public availability planned upon publication.The materials are submitted alongside the paper to facilitate reproducibility.
- 6. Experimental setting/details: The paper does not train models, but specifies the experimental setting and evaluation protocol, including evaluated models and the agent scaffold.Dedicated subsections in §3 provide these details.
7. Experiment statistical significance · 8. Experiments compute resources · 9. Code of ethics
The paper reports documented uncertainty estimates for headline experimental claims, describes its containerized evaluation infrastructure, and states that no model training was performed. It also affirms conformity with the NeurIPS Code of Ethics, including privacy, integrity, transparency, and proper handling of research assets.
- 7. Experiment statistical significance: Headline experimental claims include explicit uncertainty estimates computed using a documented method.The paper reports error bars, confidence intervals, or statistical significance information for experiments supporting its main claims.
- 7. Experiment statistical significance: 10,000-resample paired case-level bootstraps over 31 tier-1 cases produce two-sided p-values and 95% CIs for pairwise model differences.The SWE-bench Verified comparison instead uses an analogous unpaired bootstrap over per-model Bernoulli draws.
- 7. Experiment statistical significance: Per-model debug-context effects use the same 10,000-resample paired bootstrap procedure.This places context-effect estimates within the paper’s documented statistical-significance methodology.
- 8. Experiments compute resources: The containerized evaluation infrastructure is fully described, providing the compute-resource information needed to reproduce the experiments.The paper states that it did not train any models and therefore has no strong requirement for particular compute workers.
- 9. Code of ethics: The research process, methodology, and results are stated to conform fully to the NeurIPS Code of Ethics.The authors report taking precautions to avoid harm, respect privacy, and maintain integrity and transparency.
- 9. Code of ethics: The study involved neither human subjects nor crowdsourcing, so no IRB approval or equivalent was needed.This is explicitly attributed to the absence of experiments involving human subjects or crowdsourcing.
- 9. Code of ethics: Creators and original owners of assets used in the paper are stated to have been properly credited, with licenses and terms of use respected.The paper answers yes to the asset-crediting and licensing requirement.