PaperlayerView exact revision on arXiv

Source-linked AI summary

Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases

Sherman Wong, Zhenting Qi, Zhaodong Wang, Nathan Hu, Samuel Lin, Jun Ge, Erwin Gao, Wenlin Chen, Yilun Du, Minlan Yu, Ying Zhang

arXiv:2512.10398v6cs.CLcs.AIcs.LGcs.SE

TL;DR

Real-world software engineering agents must handle long-context reasoning and retain useful knowledge across tasks and sessions. CCA uses the Confucius SDK’s AX/UX/DX design, memory, and modular extensions; it achieves 59% Resolve@1 on SWE-Bench-Pro, exceeding prior research baselines and commercial results under identical conditions.

  • Problem

    Real-world software engineering requires agents to reason across massive repositories and long workflows while accumulating reusable knowledge across tasks and sessions.

    Source: §1 · p. 2

  • Method

    CCA is built on an SDK that separates Agent, User, and Developer Experience and provides persistent notes and modular extensions for tool use.

    Source: §1 · p. 2

  • Results

    59% Resolve@1 on SWE-Bench-Pro, exceeding prior research baselines and commercial results under identical repositories, model backend, and tool access.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across public benchmarks and real-world settings, CCA’s orchestration, memory, and tool abstractions can matter as much as, or more than, the backbone model.

    Source: §5 · p. 11

    CCA could not be compared with Claude Code on SWE-Bench-Pro, and the mini PyTorch-Bench comparison is not directly comparable because execution environments differ.

    Source: §3.2 · p. 8

Abstract

from arXiv · show

Real-world software engineering tasks require coding agents that can operate on massive repositories, sustain long-horizon sessions, and reliably coordinate complex toolchains at test time. Existing research-grade coding agents offer transparency but struggle when scaled to heavier, production-level workloads, while production-grade systems achieve strong practical performance but provide limited extensibility, interpretability, and controllability. We introduce the Confucius Code Agent (CCA), a software engineering agent that can operate at large-scale codebases. CCA is built on top of the Confucius SDK, an agent development platform structured around three complementary perspectives: Agent Experience (AX), User Experience (UX), and Developer Experience (DX). The SDK supports a unified orchestrator with advanced context management for long-context reasoning, a persistent note-taking system for cross-session continual learning, and a modular extension system for reliable tool use. In addition, we introduce a meta-agent that automates the construction, evaluation, and refinement of agents through a build-test-improve cycle, enabling rapid agent development on new tasks and tool stacks. Instantiated on the Confucius SDK using the meta-agent, CCA demonstrates strong performance on real-world software engineering tasks. On SWE-Bench-Pro, CCA achieves a Resolve@1 of 59%, exceeding prior research baselines as well as commercial results, under identical repositories, model backends, and tool access.

1 Introduction

CCA and the Confucius SDK address long-horizon software engineering by structuring agent, user, and developer experience and providing mechanisms for context management, persistent notes, tool use, and agent development. On SWE-Bench-Pro, CCA reaches 59% Resolve@1, exceeding prior research baselines and commercial results under identical repositories, model backend, and tool access.

1 Introduction

  • Existing coding agents often rely on flat histories, heuristic prompts, or tightly coupled tools that are difficult to scale to long-horizon, multi-file, multi-step workflows.The introduction identifies long-context reasoning across dispersed code and traces, and persistent knowledge across tasks and sessions, as core challenges.Source: §1 · p. 2
  • The SDK treats Agent Experience, User Experience, and Developer Experience as distinct design axes and provides the platform on which CCA is instantiated.The SDK unifies an orchestrator, long-term memory, and modular extensions.Source: §1 · p. 2, Figure 2 · p. 3
  • Hierarchical working memory and context compression retain essential state for long-horizon reasoning, while persistent hierarchical notes capture trajectories and failure modes across sessions.The note-taking system also produces interpretable artifacts for humans.Source: §1 · p. 2
  • Typed modular extensions define tool-use behavior, parsing, prompt shaping, and interaction policies; CCA includes file search, editing, and CLI tools.The separation is described as improving agent control and reasoning stability while supporting developer observability and composability.Source: §1 · p. 2
  • A meta-agent automates a build-test-improve loop to synthesize, evaluate, and refine agent configurations for rapid development and adaptation to new environments.Source: §1 · p. 3
  • 59% Resolve@1 on SWE-Bench-Pro, exceeding prior research baselines and commercial results under identical repositories, model backend, and tool access.The authors also report that agent scaffolding, not just model capability, is a primary determinant of performance.Source: §1 · p. 3

2 Method

The method combines an AX/UX/DX design philosophy, modular SDK abstractions instantiated in CCA, and algorithms for agent reasoning and improvement.

2.1 Design Philosophy: AX, UX, and DX

  • The SDK separates the model’s structured cognitive workspace (AX), human-facing transparency and interaction (UX), and developer observability and control (DX).It distinguishes agent inputs from human-readable traces and gives developers visibility into both.Source: §2.1 · p. 3

2.2.1 Context Management

  • To manage long sessions with multi-file refactors and nested tool calls, the SDK combines hierarchical working memory with adaptive context compression.A file-system tree stores scoped, tagged Markdown artifacts; when context nears configurable thresholds, an Architect agent replaces older turns with a structured summary while preserving recent raw messages. The authors say these mechanisms retain important state and support long-context coding without changing the orchestrator or extensions.Source: §2.2.1 · pp. 4–5

2.2.2 Note-Taking

  • A dedicated note-taking agent distills session trajectories into persistent Markdown notes, including failures and resolutions, so future sessions can retrieve known fixes.The notes are organized in a searchable, editable tree and can be created without affecting the primary agent’s online latency.Source: §2.2.2 · p. 5

2.2.3 Extensions

  • Composable extensions handle output parsing, prompt shaping, tool execution, and result routing, separating agent capabilities from the orchestrator.CCA pairs the orchestrator with coding-oriented extensions; the described ablations vary extensions and configurations while holding the loop fixed.Source: §2.2.3 · p. 6

2.3.1 The Confucius Orchestrator

  • The shared orchestrator maps native tool calls or XML tags to actions, routes them to extensions, and bounds execution with an iteration cap and agent- or extension-driven continuation.The same loop is reused across Confucius SDK agents, with CCA configured using coding-oriented extensions.Source: §2.3.1 · p. 6

2.3.2 Meta-agent: A Build-Test-Improve Loop

  • The Meta-agent automates agent design through a build-test-improve loop, synthesizing orchestrator configuration, tool wiring, and prompts, then refining them against production-grade tests.CCA itself was developed through this loop until performance stabilized.Source: §2.3.2 · pp. 6–7

3 Experiments

Across SWE-Bench-Pro, CCA outperforms scaffold baselines under matched environments, and ablations examine learned tool use, context management, and persistent notes. Evaluations also test robustness across multi-file tasks and compare CCA with open-source scaffolds on SWE-Bench-Verified.

3.1 Setup

  • CCA is evaluated against SWE-Agent and Live-SWE-Agent on SWE-Bench-Pro’s 731-task public split, and against open-source agents on the 500-task SWE-Bench-Verified benchmark.The setup uses Claude 4 Sonnet, Claude 4.5 Sonnet, Claude 4.5 Opus, and GPT-5.2 backbones, with repository and tool environments held identical for the Live-SWE-Agent comparison.Source: §3.1 · p. 7

3.2 Main Results on SWE-Bench-Pro

  • 59.0% Resolve@1: CCA with GPT-5.2 sets leading SWE-Bench-Pro performance; CCA with Claude 4.5 Sonnet reaches 52.7%, above Live-SWE-Agent’s 45.8%.The comparison uses identical environments, and the reported gains are attributed to agent scaffolds; Claude Code is not directly comparable on SWE-Bench-Pro because it lacks a compatible programmatic tool interface.Source: §3.2 · pp. 7–8, Table 1 · p. 7

3.3 Meta-Agent Learned Tool-Use

  • Disabling Meta-agent-derived tool-use features causes a large Resolve@1 decline on a 100-example SWE-Bench-Pro subset, even with context management held constant.The refined file-edit prompt gives the closest match, a minimal unified diff, and instructions to repair the search string rather than switch tools or attempt workarounds.Source: §3.3 · p. 8, Table 2 · p. 8

3.4.1 Context Management

  • On the evaluated subset, hierarchical memory and context compression raise Claude 4 Sonnet Resolve@1 from 42.0 to 48.6, a +6.6 gain.The comparison includes only tasks where both variants produced executable solutions; the authors report a smaller improvement for Claude 4.5, with both context-managed variants outperforming the simple-tool configuration.Source: §3.4.1 · p. 9

3.4.2 Endless-Read Robustness

  • CCA maintains stable Resolve Rate across SWE-Bench-Pro task groups by number of edited files, with only moderate regression as more files are modified.The authors suggest cumulative localization uncertainty and compounding diffs as likely sources of degradation.Source: §3.4.2 · p. 9, Table 3 · p. 10

3.5 Evaluations on Long-term Memory

  • Across 151 tasks rerun with notes from the first pass, iteration turns fall from 64 to 61, token cost from 104k to 93k, and resolve rate rises from 53% to 54.4%.The evaluation skips first-run cases where no meaningful insight can be distilled; the reported token cost excludes system prompt tokens.Source: §3.5 · p. 10, Table 4 · p. 10

3.6 Comparison with Open-Sourced Scaffolds on SWE-Bench-Verified

  • On SWE-Bench-Verified, CCA with Claude 4 Sonnet achieves a 74.6% Resolve Rate, exceeding OpenHands under the same backbone and a mini-SWE-Agent variant using Claude 4.5 Sonnet.The benchmark is reported as sensitive to Claude’s internal thinking budget.Source: §3.6 · p. 10, Table 5 · p. 11

4 Related Work on Coding Agents

Prior coding-agent work ranges from tool-supported repository interaction to test-time adaptation, population-based refinement, fixed pipelines, and community platforms.

4 Related Work on Coding Agents

  • SWE-Agent established iterative repository issue resolution using an LLM with file-editing, command-execution, and testing tools.It is presented as a foundational system for this paradigm.Source: §4 · p. 11
  • Later systems explore test-time self-evolution and population-based refinement, while Agentless uses a fixed three-stage pipeline for localization, patch generation, and test-case generation.Live-SWE-Agent adapts strategies and sometimes tools or prompts mid-run; Satori-SWE evolves multiple agent instances or candidates.Source: §4 · p. 11
  • OpenHands provides a community toolkit with unified file-I/O and code-execution APIs and a ReAct-style planner over popular base models.Source: §4 · p. 11

5 Conclusion

The conclusion presents CCA as an agent for large codebases built on the Confucius SDK, with scaffolding for long-horizon work and adaptation to tools and workflows.

5 Conclusion

  • CCA is a coding agent for large-scale codebases built on the Confucius SDK.Source: §5 · p. 11
  • The SDK separates AX, UX, and DX and combines modular tools, structured memory, and interpretable traces for multi-step reasoning.Source: §5 · p. 11
  • Hierarchical memory, context compression, persistent notes, extensions, and the meta-agent support long-horizon stability and adaptation to new tools and workflows.Source: §5 · p. 11
  • The authors frame the modular SDK as a foundation for research into long-context reasoning, long-term memory, test-time adaptation, and reinforcement learning with structured traces.Source: §5 · p. 11

A.1 Benchmarking Coding Agents

The appendix describes benchmarks for realistic coding tasks and the growing need for agent support across large codebases.

A.1 Benchmarking Coding Agents

  • SWE-Bench evaluates real GitHub issues in repository context, requiring patches to pass project tests; Pro targets long-horizon enterprise issues that may span dozens of files.The benchmark family also includes multilingual and multimodal variants, while SWE-fficiency targets runtime optimization.Source: §A.1 · p. 14

A.2 Large-scale Software Engineering

  • Monorepos centralize billions of lines of code, creating needs for automated code discovery, understanding, and consistent changes at scale.The passage connects this setting to context management that retrieves relevant project fragments rather than feeding an entire codebase to an LLM.Source: §A.2 · p. 14
  • ECO uses a code-generating model to suggest performance improvements in large distributed software while aiming to preserve correctness and reduce runtime and resource use.Source: §A.2 · p. 14

A.3 Training LLMs for Software Engineering

The appendix surveys execution-based task environments, synthetic task generation, and reinforcement learning as approaches to training software-engineering agents.

A.3 Training LLMs for Software Engineering

  • SWE-Gym packages Python repositories, dependencies, tests, and issue descriptions into executable environments for validating proposed patches.Source: §A.3 · p. 14
  • SWE-Smith generates bug-fix tasks by perturbing code or simulating faults, producing about 50,000 instances across 128 GitHub projects.Training on this synthetic data significantly improves benchmark performance, according to the passage.Source: §A.3 · p. 14
  • SWE-RL uses repository evolution data such as commits, diffs, and issue resolutions as implicit demonstrations for reinforcement-learning fine-tuning.The passage reports improved reasoning and success rates on long-horizon tasks, including recovery of intermediate steps such as running tests or checking files.Source: §A.3 · pp. 14–15

B Context Summarization Ablations

With the main orchestrator and summary format, length, and cadence held comparable, the ablation varies only the summarizer. Sonnet-based summaries resolved more of the 50 selected instances than Haiku-based summaries, while examples show that Sonnet preserved more actionable progress details.

B Context Summarization Ablations

  • The ablation selected 50 long-context instances where summarization occurred and varied the summarizer while keeping other summary conditions comparable.The main orchestrator was Claude 4 Sonnet; the summarizers were Claude 3.5 Haiku and Claude 4 Sonnet.Source: B Context Summarization Ablations · p. 15
  • 26/50 instances were resolved with Sonnet-based summaries, compared with 22/50 using Haiku-based summaries.The runs used the same main orchestrator and comparable summary format, length, and cadence.Source: B Context Summarization Ablations · p. 15
  • Sonnet’s example summary retained implementation phases and the next planned task, while Haiku’s summary omitted important keywords and progress detail.The examples describe Sonnet tracking Phase 1 completion, Phase 2 progress, and the next step of updating the Run function.Source: B Context Summarization Ablations · p. 17

C.1 Implementation Details

The Meta-agent turns a developer’s natural-language specification into a configured agent with selected extensions and memory policies. It can then test the agent on regression tasks and iteratively refine its prompts, extensions, or tools, alongside a suite of developer interfaces.

C.1 Implementation Details

  • The Meta-agent converts a natural-language agent specification into configuration and prompts, wiring in selected extensions and memory policies after clarifying requirements.The specification can include repository scope, latency or safety constraints, extensions, and evaluation tasks.Source: §C.1 · p. 17
  • It runs candidate agents on regression tasks, inspects outputs and traces, and proposes prompt, extension, or tool-wrapper changes before rerunning tests.The build-test-improve loop can also assist with designing and debugging tools for the extension layer.Source: §C.1 · p. 17

C.2 Development Cycle

  • Developer-cycle tools include onboarding with coding-agent templates and multi-turn requirement clarification, plus interfaces for prompt tuning, evaluation, and agent management.The listed tools include a Playground, Eval UI, and centralized agent management.Source: §C.2 · p. 18
  • The Trace UI visualizes call stacks, tool interactions, and memory flows to support inspection of agent execution.Figure 5 also identifies latency metrics, token usage, and tool invocations in the trace view.Source: §C.2 · p. 18, Figure 5 · p. 18

D Thinking Budget Scaling

The paper examines how Claude 4 Sonnet’s thinking budget relates to Resolve Rate on a SWE-Bench-Verified subset. Returns diminish beyond 16k tokens, but the model’s internal reasoning length cannot be measured precisely with the available inference interface.

D Thinking Budget Scaling

  • Claude 4 Sonnet was evaluated at 8k, 16k, and 32k thinking-budget settings on a subset of SWE-Bench-Verified.The settings varied while the rest of the setup remained identical.Source: D Thinking Budget Scaling · p. 19
  • Beyond 16k thinking tokens, Resolve Rate shows diminishing returns on the evaluated subset.The study compares 8k, 16k, and 32k-token settings; the table reports Resolve Rate as a function of budget.Source: Table 6 · p. 19, D Thinking Budget Scaling · p. 19
  • The thinking-budget parameter does not precisely control internal reasoning length, and Claude returns only a summarized reasoning trace, preventing an exact scaling curve against true thinking length.Source: D Thinking Budget Scaling · p. 19

E Example Execution Trace

The section presents examples of CCA in practice, covering repository-level diagnosis and testing, reusable notes, and a PyTorch issue benchmark with a controlled comparison to Claude Code.

E Example Execution Trace

  • The task required allowing archiveDataType = null for owned archives while preserving non-owned behavior, prompting changes to the type, facade signatures, and EntityRestClient call.CCA identified the requirement after exploring the repository and locating the relevant methods and types.Source: E Example Execution Trace · pp. 20–21
  • After changing three files, all three reproduction tests passed, and CCA reported that the fix preserved non-owned archive behavior while allowing owned-archive requests without archiveDataType.The preceding reproduction had failed all three checks: null support, nullable method signatures, and passing null from EntityRestClient.Source: E Example Execution Trace · pp. 20–21

F Example Notes for Long-term Memory

  • The note hierarchy distinguishes project-specific knowledge from genuinely reusable shared insights and uses a README to summarize notes and explain navigation.The example presents notes for an OpenLibrary project alongside shared Python and string-manipulation notes.Source: F Example Notes for Long-term Memory · p. 21
  • The OpenLibrary note records context-dependent wildcard handling: escape asterisks for exact and alternate-name matches, preserve them for surname matching, and retain the original name when creating records.The note explains that unescaped asterisks can match unintended author names and that query order proceeds from exact to alternate-name to surname matching.Source: F Example Notes for Long-term Memory · p. 22
  • A second note captures the empty-string edge case in honorific removal and recommends returning the original value when removing a prefix leaves no name.For example, removing an honorific from a name consisting only of “Mr.” could leave an empty string and cause downstream problems.Source: F Example Notes for Long-term Memory · p. 23
  • Long-term notes record edge cases such as removing extensions or URL protocols and stripping punctuation, whitespace, prefixes, or suffixes to save tokens and reduce missed-case failures.The passage says this concrete edge case saves tokens in Run 2 and helps avoid failures when edge cases are missed across consecutive runs.Source: F Example Notes for Long-term Memory · pp. 23–24

G Case Studies: Comparison with Claude Code

  • The controlled comparison used identical Claude Sonnet 4.5 models, equivalent codebases and tools, and expert judges to compare solutions to real PyTorch issues.Both frameworks had file-manipulation and bash tools plus NVIDIA A100 80GB GPU resources; only the agent framework varied.Source: G Case Studies: Comparison with Claude… · p. 24

CUDA Memory Checkpoint Assertion failure with Expandable Segments

  • CCA removed two allocator assertions, while Claude Code added seven lines to null a split block’s next pointer; PyTorch’s eventual fix matched CCA’s approach.The issue concerned restoring a smaller CUDA graph checkpoint after a larger one with expandable segments enabled. The authors present this as one of three representative cases.Source: CUDA Memory Checkpoint Assertion… · p. 26, §G.2 · p. 26

Excessive Memory Allocation and Deallocation during Llama-2 Training

The examples contrast CCA’s minimal fix for unintended CUDA memory reclamation with CC’s broader memory-management change, and show both agents correcting a GPU test tolerance before CC also addressed a warning.

Excessive Memory Allocation and Deallocation during Llama-2 Training

  • CCA disabled memory reclamation when expandable segments were enabled with a six-line guard; CC used a 63-line pressure-based solution with emergency reclamation above 95%.Both fixes addressed the immediate issue, but CCA’s fix targeted the problematic behavior while CC took a broader approach.Source: Excessive Memory Allocation and… · p. 27

Neural Network Precision Regression during Weight Manipulation

  • Both agents corrected the GPU test by changing its tolerance to atol=1e-4 and rtol=1e-4, allowing it to pass.The failure showed relative differences of approximately 0.003 against an expected tolerance of 1.3e-06.Source: Neural Network Precision Regression… · p. 27
  • After confirming the test passed, CC also changed RNNBase to call flatten_parameters() after manual weight assignment, addressing a non-contiguous-memory warning.The passage describes this additional 20-line change as potentially over-engineered.Source: Neural Network Precision Regression… · p. 27

G.3 Architectural Analysis

The analysis contrasts CCA’s context-preserving exploration with CC’s stateless subagent delegation, which can lead to more complex solutions on well-scoped debugging tasks. It also identifies RL training as a future direction that could build on CCA’s trajectories and feedback signals.

G.3 Architectural Analysis

  • Both agents reproduced the issue, explored the allocator and expandable-segment logic, and identified a terminal-block invariant violation as the cause.CCA explored within its original context, while CC delegated investigations to stateless subagents; Figure 6 presents simplified traces for the issue.Source: §G.3 · p. 27, Figure 6 · p. 28
  • The trace describes CCA recognizing the invariant violation and implementing a fix in a few lines.The passage identifies the violation as stale `remaining->next` values after block splits, where `next` must be `nullptr`.Source: §G.3 · p. 28
  • For well-scoped debugging, delegation’s benefits may be outweighed by context loss and inter-agent misalignment, contributing to more complex solutions than necessary.The analysis says CC’s main agent trusted an exhaustive-analysis subagent’s recommendation despite independently preferring a simpler solution.Source: §G.3 · p. 28

H Future Work

  • CCA’s trajectory-friendly traces and the meta-agent’s fine-grained feedback could support RL rewards and curricula spanning tools, environments, and task complexity.The proposed reward signals include note-taking quality, tool-use robustness, recovery behaviors, and multi-step exploration efficiency.Source: H Future Work · pp. 28–29
Loading 2512.10398v6…