Source-linked AI summary
Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair
Mehdi Bahrami, Kosaku Kimura, Satoshi Munakata, Satoshi Nakashima, Yu Ishikawa, Kosuke Maeda, Nao Soma, Kenichi Kobayashi, Keisuke Miyazaki, Keizo Kato, Shigeki Fukuta, Tatsuo Kumano, Nobutaka Imamura, Kevin Musgrave, Shahbaz Abdul Khader, Kwun Ho Ngan, Joe Townsend, Fayas Asharindavida, Matthieu Parizy, Akira Sakai, Yuma Ichikawa, Yang Zhao, Michiaki Takizawa, Taku Fukui, Hiroki Ohtsuji, Wei-Peng Chen, Hiromichi Kobashi
TL;DR
Industrial software-repair agents must handle long, tool-using attempts in an auditable, repeatable way. Kozuchi Agent addresses this with a phase-driven, model-agnostic harness, CI pipeline, and cross-agent test-time selection, resolving 374/500 SWE-bench Verified instances and 41/128 Multi-SWE-bench Java instances with a locally hosted open-weight model.
Problem
Software-repair agents need auditable, repeatable support for long-horizon, tool-using bug-fixing trajectories beyond model selection alone.
Method
Kozuchi Agent uses explicit repair phases, persistent state, deterministic tools, a model-independent action interface, CI orchestration, and cross-agent testing for candidate selection.
Results
374/500 SWE-bench Verified instances and 41/128 Multi-SWE-bench Java instances are resolved with the same locally hosted open-weight agent, ranking first among strict open-weight Java submissions.
Takeaways & Limitations
The results support a narrower claim that the harness design and cross-agent selector transfer from Python to Java unchanged, while remaining errors are mainly semantic or selection-related.
Takeaways & Limitations
Cross-agent testing depends on high-quality agent-generated tests, so weak tests can directly reduce the selector’s ability to discriminate among candidates.
Abstract
from arXiv · showhide
Industrial software-engineering teams increasingly need LLM agents that turn bug reports into correct patches, yet benchmark-scale operation adds long horizons, tool-use discipline, context persistence, heterogeneous clusters, and evaluation reuse. We present Kozuchi Agent, a language-agnostic open-weight repair agent and CI-operated evaluation pipeline. Explicit phases, persistent state, deterministic tools, a model-independent action interface, and cross-agent test-time selection make runs auditable and repeatable. With locally hosted Qwen3.5-27B, no fine-tuning, and TTS@8, Kozuchi resolves 374/500 SWE-bench Verified instances on the official evaluator. Unchanged on Multi-SWE-bench Java, the same 27-billion-parameter agent resolves 41/128 instances (32.03%), ranking first among strict open-weight submissions and fourth of 42 overall; on Python it ranks 12th of 135 and first among open-weight systems. Per-phase behavior remains within +/-5 percentage points across languages. Remaining failures mainly reflect semantic correctness, Java-specific harness issues, and selection errors. Across both tracks, results compare favorably with open/local peers by parameter count. Analysis of candidate diversity, selector regret, and patch reliability shows that the remaining gap is primarily semantic correctness and selection rather than edit formatting or proprietary-model access. Operationally, reusable CI stages reduce operator touch-points from five to one across heterogeneous internal clusters.
1 Introduction
Kozuchi Agent addresses industrial software-repair demands by combining a phase-driven, model-agnostic agent harness with deterministic tools, cross-agent selection, and a reusable CI pipeline. The introduction frames long-horizon execution, tool-grammar drift, heterogeneous clusters, and evaluation cost as central engineering challenges, while limiting industrial claims to internal pipeline replacement.
- Motivation and challenges: Industrial repair requires an agent that turns issue descriptions and repository states into correct, test-passing patches through auditable and repeatable long-horizon workflows.Real fixes involve reproduction, regression testing, root-cause analysis, multi-file editing, and verification across many LLM turns.
- Motivation and challenges: Tool-grammar drift and heterogeneous execution clusters require a shared runtime that absorbs model-specific action syntax and spans inference, testing, and grading environments.The targeted environments include GPU inference and training, VM-based testing, and Docker-based benchmark grading.
- Evaluation infrastructure: 6 of 9 CI stages can be short-circuited per iteration through reuse mode, making Docker-based benchmark reruns fit a few SLURM jobs instead of a research-cluster reservation.The evaluator-grade score remains the gate for external claims and the floor for experimentation.
- System contributions: Kozuchi Agent combines eight explicit repair phases, a model-independent action contract, deterministic sandbox operations, multi-cluster orchestration, reusable artifacts, and cross-agent test-time selection.The workflow is layered on mini-swe-agent and supports auditable trajectories and repeatable CI pipelines.
- System contributions: K=8 candidate inference runs are merged through cross-agent selection, with released artifacts exposing trajectories, selector choices, evaluation outputs, and deeper analyses.The published scripts regenerate empirical values from benchmark harness reports, trajectories, and selection bundles.
- Scope and evidence: The industrial claim is limited to internal evaluation-pipeline replacement involving touch-points, reusable stages, and cluster abstraction, excluding production traffic, developer KPIs, and proprietary repositories.The introduction identifies candidate count and cross-agent selection as the strongest isolated causal evidence.
2 Related Work
Kozuchi Agent builds on established software-engineering agents, runtime tools, and industrial coding systems while focusing on an auditable, CI-operated harness. Its main novelty is cross-agent test-based selection integrated with phased workflows, persistent state, deterministic tools, and industrial-scale evaluation.
- Agentic SWE Systems: SWE-agent established LLM shell observation and command emission, while Kozuchi adds phase contracts, fixed tools, and multi-cluster execution.AGENTLESS, OpenHands, and AutoCodeRover provide complementary software-agent designs and platforms.
- Industry-Facing Coding Agents: Commercial coding agents demonstrate demand for asynchronous issue-to-patch systems; Kozuchi instead contributes inspectable harness and evaluation infrastructure.Its focus includes phase boundaries, deterministic tools, queue-aware CI orchestration, reusable benchmark artifacts, and inspectable selection.
- Tools, Training, and CI: Prior work informs tool-augmented training and execution-based code evaluation, whereas Kozuchi holds the model fixed and injects classic software-engineering tools at runtime.These tools include line tracing, caller discovery, and guarded editing; fine-tuning remains a separate active direction.
- Novelty: Kozuchi’s originality lies in cross-agent selection, where independently generated tests are cross-applied without hidden or production-only feedback.The work integrates this selector with phased workflows, persistent state, runtime tools, and CI orchestration into one auditable industrial-scale harness.
3 Background and Industrial Context
This section situates Kozuchi Agent within repository-level software-repair evaluation and motivates its industrial harness. The harness replaces a manual, heterogeneous-cluster workflow with a CI-driven pipeline built around persistent, deterministic, model-independent repair operations.
- Benchmark and evaluation context: SWE-bench evaluates repository-level patches against hidden issue-revealing and regression tests, extending earlier function-level code-generation and program-repair benchmarks.SWE-bench Verified is described as a 500-instance human-curated subset with cleaner problem statements.
- Agent architecture: Kozuchi Agent builds on mini-swe-agent by adding phase control, persistent shared state, deterministic tools, and model-independent action formatting.These additions support long repairs while preserving the underlying tool-using repair loop.
- Industrial workflow: The CI-driven harness replaces manual server startup, one-off benchmark launches, Docker grading, and spreadsheet score recording across heterogeneous internal clusters.The replacement provides one configurable agent, a fixed runtime contract, and a publishable artifact bundle.
4 Definitions and Design Invariants
Kozuchi formalizes its repair runtime, trajectories, phase transitions, tool calls, and verification as explicit, auditable design objects. Runtime invariants enforce parseable actions, configured control flow, patch isolation, bounded observations, and strategy changes after repeated failures.
- Runtime and auditability: The agent runtime R = ⟨P, S, T, A, F, H⟩ is configuration-declared as phases, shared state, tools, actions, parsing, and handovers.This replaces recovery from ad hoc scripts with an explicit runtime definition.
- Runtime and auditability: Published trajectories record prompts, completions, tool calls, responses, feedback, handovers, and state snapshots, making each instance auditable.Trajectories are the paper’s main audit object.
- Control and tool contracts: 8 phases and 15 normal/fallback edges define the evaluated phase-transition graph.The phase transition system is represented as GΦ = (VΦ, EΦ).
- Control and tool contracts: Exactly one parseable action is required per assistant turn, with four surface formats preserving one logical contract across model families.Parser failures remain explicit in the trajectory.
- Runtime and auditability: Runtime invariants enforce configured phase exits, exclude notes and logs from patches, summarize long observations with elision, and warn after repeated failures or identical actions.Every turn yields one parseable action or a recorded format error.
5 Motivating Empirical Observations
Early prototypes revealed that long-horizon repair runs were operationally fragile: they lost evidence, repeated work, depended on brittle model-specific tools, localized defects poorly, and made recomputation costly. These failures motivated phase boundaries, durable shared state, deterministic tools, configurable action formats, and CI artifact reuse.
- Operational failures: Long repair contexts caused evidence loss and repeated work, while multi-hour benchmark stages made recomputation expensive.These failures exposed the need for durable shared state and CI artifact reuse.
- Tooling limitations: Model-specific tool syntax made backend swaps brittle, and shell-only debugging localized defects poorly.The observations motivated configurable action formats and deterministic tools.
- Design response: The resulting design introduced phase boundaries, durable shared state, deterministic tools, configurable action formats, and CI artifact reuse.These mechanisms directly addressed the recurring failures observed in early prototypes.
6 System Design
Kozuchi Agent uses an explicit, phase-driven runtime with persistent artifacts, deterministic tools, and model-independent actions to make long repair trajectories auditable. Its reusable CI stages and test-time selection support repeatable evaluation across models and clusters.
- Phases: 8 phases and 15 directed edges structure repair trajectories around auditable completion or explicit give-up exits.The configured graph covered all eight declared phases for 495 persisted trajectories, without claiming that every run solves its instance.
- Context Compression and Handover: 495 persisted trajectories retain phase artifacts and handover state in shared storage, making filesystem state durable across context compression and phase changes.Artifacts include tests, traces, root-cause notes, patch diffs, and reports; handover writes concise memos.
- Skills: 14 phase- and tool-gated skills narrow prompts and action spaces by exposing reusable guidance only when relevant.Skills are prompt-guidance procedures rather than learned capabilities, and the policy targets trajectories spanning hundreds of turns.
- Action Formats: One executable action per turn separates semantic requirements from model-specific syntax, enabling multiple model families to share the runtime and tool semantics.The interface preserves a common state machine while action formatting varies by model family.
- Tool Sandbox: 3 deterministic tools—line tracing, caller discovery, and guarded editing—make localization and modifications inspectable while limiting dependence on a brittle tool zoo.Guarded editing couples each change to an expected target line, reducing free-form patch corruption.
- CI Evaluation: Reusable CI stages let operators rerun only the stage under study while holding earlier artifacts fixed, enabling repeated TTS@8 sweeps across local and cluster execution.The stage graph includes generation, grading, synthesis, optional training, benchmarking, selection, and reporting, with reuse short-circuits.
7 Cross-Agent Test-Time Selection
Kozuchi separates candidate generation from cross-agent selection, using archived agent-chosen tests and fixed scoring weights rather than benchmark feedback. With eight runs, the selector achieves 374 official resolutions, approaching the available diversity ceiling while leaving compute under-monetized.
- Selection design: 8 independent runs generate candidate patches and agent-chosen bug-revealing and regression-preservation tests, which the selector cross-applies without choosing tests.This separation avoids using official benchmark tests during selection and supports selection before hidden or production-only feedback is available.
- Selection design: 0.3 and 0.7 are fixed global weights for bug-revealing and regression pass rates, respectively, and were neither learned nor tuned on benchmark outcomes.The asymmetry encodes an engineering prior for merge-gated operation.
- Diversity Ceiling: 376 internal Docker re-grade resolutions, or 374 on the official evaluator, reach about 376/408 = 92.2 % of the oracle union ceiling.At least one run resolves 408 instances, while all runs resolve 234; useful diversity is mainly which instances are solved.
- Diversity Ceiling: k≈2.2 places the cross-agent selector near an oracle, indicating that much of the eight-way compute budget remains in candidate generation that selection does not fully monetize.Pairwise completion overlap is high, limiting the additional value extracted from eight-way generation.
8 Evaluation
Kozuchi resolves 374/500 Python instances (74.80%) on the official evaluator, ranking first among open-weight systems and 12th overall. Selection improves substantially over individual runs, while remaining failures are dominated by semantic correctness rather than patch application.
- Python results: 374/500 instances (74.80%) are resolved on the official Python evaluator, with a Wilson 95% CI of [70.82%, 78.41%].The internal Docker re-grade resolves 376/500 (75.2%); the two-instance difference reflects evaluator differences.
- Peer comparison: 12th overall and highest-ranked open-weight system, Kozuchi significantly outperforms 16 of 17 curated open-weight peers.The next open-weight submission resolves 356/500 (71.2%), while a 480-billion-parameter Qwen3-Coder configuration trails by 26 instances with BH-FDR q=0.011.
- Behavioral analysis: Resolution falls from 81.7% at 1 LOC to 4 LOC to 41.7% at >100 LOC, with LOC churn showing the strongest single feature signal (r=−0.197, p=1.1×10−5).After adjustment for consensus-based hardness proxies, patch-churn and trajectory-effort coefficients are not significant at α=0.05.
- Failure analysis: 115 of 126 unresolved instances (91.3%) are WRONG_FIX, whereas only 1 of 495 produced patches (0.20%) fails at the edit layer.WRONG_FIX patches apply cleanly but do not flip hidden FAIL_TO_PASS tests; PATCH_DID_NOT_APPLY and EMPTY_PATCH are both 0.
- Selection: 374/500 (74.8%) selected instances exceed the 338.6/500 (67.7%) mean individual-run baseline, while the order baseline resolves 362/500 (72.4%).The selector contributes +14 absolute (2.8 percentage points) over first-run-wins and +33 to +42 absolute over individual per-run pass@1.
9 Operational Planning and Lessons Learned
Operational lessons emphasize structured, auditable long-horizon runs, configuration-level handling of model-family variation, and CI simplification. In the inventoried workflow, CI reduces operator touch-points from five to one, while the approximately 70-engineer-minute estimate remains an artifact-derived order-of-magnitude claim.
- Operational lessons: Every Python TTS@8 trajectory visits all eight phases, with rework concentrated in CODE_FIX↔VERIFY_PATCH and phase behavior transferring to Java within ±5 percentage points per phase.Phase boundaries keep long-horizon runs structured and auditable.
- Operational lessons: Model-family variation is cheapest to handle at the action-format boundary, where switching tool-call syntax becomes a configuration change rather than an architectural change.The operational lesson is to isolate model-specific variation at the interface boundary.
- Operational indicators: Five operator touch-points become one CI push in the inventoried workflow.This indicator comes from the published workflow inventory.
- Operational indicators: About 70 engineer-minutes per cycle is an artifact-derived estimate, not a controlled before/after measurement from a developer pilot.The figure is reported as an order-of-magnitude operational claim based on published per-stage inventory and prior SWE-bench runtime and CI-adoption anchors.
10 Conclusion
Kozuchi Agent demonstrates operationally ready, open-weight issue-to-patch evaluation across SWE-bench Python and Multi-SWE-bench Java. The results attribute its performance to harness engineering and isolate candidate count and selector signals as the measured ablation factors.
- Operational readiness evidence: 374/500 SWE-bench Verified instances were resolved using a locally hosted open-weight model, no fine-tuning, eight candidate runs, and cross-agent testing.This corresponds to 74.80 %, with Wilson 95 % CI [70.82 %, 78.41 %] and repository-clustered bootstrap [67.0 %, 79.8 %].
- Operational readiness evidence: 41/128 Multi-SWE-bench Java instances were resolved under the same locally hosted open-weight, no-fine-tuning setup.The passage reports this as 32.0.
- What we learned: Harness engineering centers on phase decomposition, single-command grammar, persistent shared filesystem, deterministic SE tools, cross-agent test selection, cluster indirection, and CI reuse.The paper presents these mechanisms as the main lessons from the results.
- What we learned: Ablations isolate candidate count and selector signals, while controlled per-mechanism ablations are deferred to follow-up work using the published trajectory bundle.The deferred ablations include no-phase, no-formatter, no-Orchestra, and no-tools conditions, without re-running TTS@8.
Contributions of Each Team
The teams divided responsibilities across agent design, evaluation, verification, Java tooling, software-engineering tools, paper writing, and test-time selection.
- The overall design and implementation of Kozuchi Agent, along with SWE-bench evaluations, were led by one team.
- Another team led paper writing, designed the Verifier, prepared Java tools, and evaluated Multi-SWE-bench Java.
- The Software Engineering Tool Suite was developed by a separate team.
- Test-time selection evaluations were conducted by another team.