PaperlayerView exact revision on arXiv

Source-linked AI summary

From Model Scaling to System Scaling: Scaling the Harness in Agentic AI

Shangding Gu

arXiv:2605.26112v1cs.AIcs.LG

TL;DR

Agentic-AI evaluation often centers on models and one-shot success, although agent behavior also depends on the surrounding system. This paper frames harness scaling as a system-level research agenda and concludes that different harnesses can produce qualitatively different agents.

  • Problem

    Agentic-AI evaluation often focuses on model capability and one-shot success, while the surrounding execution system also shapes agent behavior.

    Source: §1 · p. 3, §7 · p. 12

  • Method

    The paper frames harness scaling through interacting reasoning, memory, context, skill-routing, orchestration, and verification-governance components.

    Source: §3 · p. 5

  • Results

    The authors conclude that comparable models projected onto different harnesses produce qualitatively different agents.

    Source: §7 · p. 11

  • Takeaways & Limitations

    Future benchmarks should evaluate memory, context, skill routing, orchestration, and verification-governance as first-class objects rather than measure only one-shot success.

    Source: §7 · p. 12

    Stronger models may reduce system failures, but the authors argue they do not remove the need for explicit information, permission, and failure-tracing mechanisms.

    Source: §6 · p. 11

Abstract

from arXiv · show

This paper studies the next major bottleneck in agentic AI as system scaling, not only model scaling: the design of auditable, persistent, modular, and verifiable architectures around foundation models. We refer to this shift as scaling the harness: treating the structured execution layer around a foundation model as a first-class object of design, evaluation, and optimization. Although recent large language models enable agents to use tools, retrieve information, maintain memory, and execute long-horizon workflows, evaluation remains largely model-centric, often reducing agents to final-task success while treating memory, retrieval, tool use, orchestration, verification, and governance as secondary implementation details. This framing is increasingly inadequate because agent performance emerges from the interaction among the foundation model, memory substrate, context constructor, skill-routing layer, orchestration loop, and verification-and-governance layer. Together, these components form the agent harness, which translates model capability into long-horizon agent behavior. We study scaling the harness through three core bottlenecks: context governance, trustworthy memory, and dynamic skill routing, together with the orchestration and governance mechanisms that coordinate and constrain them. We further outline a research agenda for harness-level benchmarks that go beyond one-shot task success to measure trajectory quality, memory hygiene, context efficiency, communication fidelity, verification cost, and safe evolution over time. To make the discussion concrete, we develop CheetahClaws: https://github.com/SafeRL-Lab/cheetahclaws, a Python-native reference harness, and compare it with Claude Code and OpenClaw. Our main claim is that future progress in agentic AI will depend as much on system design as on stronger foundation models.

1 Introduction

Agentic AI performance depends on the harness around a foundation model, not only on the model itself. This section frames system scaling as improving that structured execution layer and outlines an evaluation agenda for its long-horizon behavior.

System-scaling framing

  • Agentic AI should be evaluated as a system-scaling problem: beyond a sufficient model capability threshold, long-horizon performance increasingly depends on the harness around the model.The harness encompasses context construction, memory, skill routing, orchestration, verification and governance, and adaptation over time.Source: §1 · pp. 3–4

Motivation

  • Existing production harnesses couple models to tools, subagents, and persistent project memory, illustrating that practical agent capability arises from model–harness interaction.The introduction names Claude Code, OpenClaw, and research-side SWE-agent [40] as examples.Source: §1 · p. 3
  • Evidence cited in the introduction shows that harness choices affect measured performance: interface redesign can improve SWE-bench accuracy with a fixed model, while benchmark rankings can change when costs and prompting factors are controlled.The field-level analysis reports that many results become non-Pareto-optimal after controlling for capability, costs, prompting strategy, and demonstrations [17]; SWE-agent interface design is cited as an example [40].Source: §1 · p. 3
  • Context length alone does not ensure useful information access, and multi-agent breadth can come with coordination failures that single-agent metrics miss.The introduction also notes that models may favor evidence at the beginning or end of context [21], and that attention can dilute over long inputs [11].Source: §1 · p. 3
  • Long-horizon reliability requires governed context and high-quality memory, while collaboration and adaptation require communication protocols, uncertainty sharing, and auditable evolution.The paper identifies retrieval, compression, ordering, freshness, trust, and avoiding stale or contaminated memory as context and memory challenges; it also flags drift and over-generalization and communication protocols [12].Source: §1 · p. 3

Harness-level framework

  • The proposed framework separates base-model reasoning from memory, context construction, skill routing, orchestration, and verification-and-governance as first-class harness design factors.It treats the harness as an object of design and analysis, rather than reducing an agent to a model plus prompt.Source: §1 · pp. 3–4

Evaluation agenda and reference harness

  • Future agent benchmarks should measure trajectory quality, memory hygiene, context efficiency, verification cost, safe evolution, and robustness under repeated use.The paper develops CheetahClaws, a Python-native reference harness, and compares it with Claude Code and OpenClaw.Source: §1 · p. 4

2 Related Work

Related work spans tool-using coding agents, context and memory systems, reusable skills and multi-agent coordination, and system-oriented benchmarks, while harness-level study and evaluation remain limited. This paper reframes these developments through system scaling and compares Claude Code, OpenClaw, and CheetahClaws.

Agentic coding systems and harness engineering

  • Coding agents build on tool-using policies such as ReAct [44], while production runtimes and SWE-agent [40] illustrate the importance of harness and interface design.SWE-agent shows that carefully designed tool schemas can substantially change benchmark accuracy with a fixed backbone model.Source: §2 · p. 4

Context, memory, and retrieval

  • Retrieval-augmented generation [19], MemGPT [29], and Voyager [35] motivate treating retrieval, memory, and context as a governance problem amid privacy drift [11] and retrieval uncertainty [12].These systems address external memory, hierarchical memory management, and a growing skill library, respectively.Source: §2 · p. 4

Skills and multi-agent coordination

  • Reusable skills [35] and multi-agent frameworks [13] [20] motivate joint skill routing and delegation, with heterogeneous-subagent routing identified as the next systems bottleneck.Related work also examines orchestrator–subagent gains, population diversity [42] [45], negotiation [22], and the agentic web [41].Source: §2 · p. 4

Benchmarks, governance, and agent evolution

  • Executable multi-step benchmarks [23] [46] [24] broaden agent evaluation, but single-episode success leaves memory quality, context efficiency, communication fidelity, and safe evolution largely unmeasured.The passage also identifies surveys of LLM-based agents and catalogues of agentic safety threats.Source: §2 · p. 4
  • The paper reframes prior developments through system scaling and compares Claude Code, OpenClaw, and Python-native reference harness CheetahClaws.This adopts the harness itself as a controllable, reproducible object of study.Source: §2 · pp. 4–5

3 System Scaling: A Framework for Agentic AI

System scaling treats the harness around a foundation model as a first-class design target: its interacting components govern how information, computation, authority, and verification are allocated over time. The paper distinguishes short-horizon prompts, reusable skills, and longitudinal memory as complementary temporal levers.

3 System Scaling: A Framework for Agentic AI

  • System scaling improves memory, context construction, skill routing, orchestration, and governance around the reasoning substrate, rather than only scaling the model’s reasoning quality.The harness comprises six interacting components: R, M, C, S, O, and G; orchestration coordinates context drawn from memory, model reasoning, skill dispatch, and verification before actions or memory write-backs.Source: §3 · p. 5, Figure 1 · p. 6
  • The six-factor decomposition is conceptual, not a quantitative model: its factors are not strictly orthogonal, though each offers an intervention that can be changed or measured with the same foundation model.Reasoning and context are hardest to separate because either can compensate for the other; memory and governance are easiest to isolate because writes and audit trails can exist independently of an inference step.Source: §3 · p. 5

3.1 Agent Harnesses as System Infrastructure

  • Claude Code [3], OpenClaw [34], and CheetahClaws share core harness problems but realize them differently according to deployment priorities, not simply model differences.Table 1 presents illustrative design patterns rather than a ranking: vendor-scale systems prioritize reliable use, personal-assistant systems multi-channel management, and research harnesses transparency and reproducibility.Source: §3.1 · pp. 6–7, Table 1 · p. 6
  • CheetahClaws makes memory trust explicit with per-entry confidence and recency fields used in retrieval ranking and conflict resolution, unlike the other two systems’ implicit trust from access patterns.All three systems consolidate session content into persistent memory, using subagent extraction, background daemons, or dedicated consolidation routines.Source: §3.1 · p. 7

3.2 Prompt, Skill, and Memory as Temporal Layers

  • Prompt controls what to do now, skill controls how to perform a class of tasks, and memory controls what persists over time; the three levers are complementary.Prompting is flexible and cheap but does not itself provide persistence or reliable long-horizon structure; skills are reusable execution patterns that require routing and composition, while memory can drift, over-generalize, or become polluted.Source: §3.2 · p. 7

4 Three Bottlenecks in System Scaling

System scaling depends on governing context, making memory trustworthy at runtime, and routing skills together with verification; increasing capacity alone does not ensure reliable agent behavior.

4.1 Context Governance

  • Effective context is governed—not merely enlarged—by selecting the minimum sufficient, relevant, compact, traceable, freshly validated material under a token budget.Without governance, low-value tokens dilute or bury relevant evidence, and long context can degrade performance; selection policies should record provenance for audit.Source: §4.1 · p. 8

4.2 Trustworthy Memory

  • Trustworthy memory depends on scoped precision, durability, and live verifiability; retrieval makes that trust usable but does not create it.Because stale notes may still rank highly and prompt confident destructive actions, retrieval should penalize staleness and treat memories as hypotheses to re-check; Claude Code combines persistent CLAUDE.md context with live repository tools [4, 15, 2], retaining prior verification while bounding drift.Source: §4.2 · pp. 8–9

4.3 Dynamic Skill Routing and Verification

  • Effective skill use requires scoped capabilities, selective routing, composable post-conditions, and verifiable outputs, since specialized skills can produce plausible but unchecked claims.The proposed direction is adaptive routing using subtask estimates, confidence-aware escalation, composition, and optimization for verified outputs; skill scaling without verification can make progress faster but less reliable.Source: §4.3 · p. 9

5 Toward System-Level Evaluation and Agent Evolution

System-level evaluation should measure how agents act—not only whether they succeed—and whether they remain useful, grounded, safe, and auditable over repeated use. The section proposes evaluating process and longitudinal behavior alongside outcomes, with explicit standards for agent evolution.

5.1 From Outcome Metrics to Process Metrics

  • The proposed process protocol extends rather than replaces SWE-bench, AgentBench [23], WebArena [46], and Terminal-Bench [24] by measuring system factors that outcome scores can hide.Those benchmarks already evaluate executable repository tasks, diverse interactive environments, realistic browser tasks, and environment-grounded terminal tasks, respectively.Source: §5.1 · p. 9
  • Outcome scores can conflate model capability with harness design and miss differences in cost, risk, and auditability, so evaluation should pair task success with process metrics.Process measures include context and computation used, trajectory, retrieval and verification, and risk incurred.Source: §5.1 · p. 9

5.2 From Single Episodes to Longitudinal Evaluation

  • A multi-agent system with Claude Opus 4 as lead and Claude Sonnet 4 subagents outperformed single-agent Claude Opus 4 by 90.2% on Anthropic’s internal research evaluation.Multi-agent architectures were especially effective on breadth-first tasks that explore independent directions in parallel.Source: §5.2 · pp. 9–10
  • On a BrowseComp-based analysis, token usage alone accounted for 80% of performance variance; adding tool-call count and model choice raised explained variance to 95%.These results suggest that computation allocation and tool use strongly shape agent performance.Source: §5.2 · p. 10
  • Multi-agent collaboration requires reliable shared state, uncertainty communication, contradiction detection, de-duplication, conflict resolution, and verified handoffs—not just task decomposition.System-design issues, inter-agent misalignment, and inadequate verification can cause failures independently of underlying model limitations.Source: §5.2 · p. 10
  • Repeated-use benchmarks should measure memory precision and hygiene, minimal-context efficiency, communication fidelity, drift, recovery after stale memory or wrong routing, and safety under tool access.Agents accumulate conventions, preferences, failures, and procedures across sessions, but this can also cause contamination, staleness, over-generalization, and privacy leakage.Source: §5.2 · p. 10

5.3 Standards for Safe Agent Evolution

  • Persistent behavior raises risks of hard-to-detect surviving behaviors [14], proxy-objective reward hacking, and agentic threats including memory poisoning, identity spoofing, tool misuse, and goal manipulation.The section argues that an agent evolution standard must make these failure modes visible, measurable, and bounded.Source: §5.3 · p. 10
  • An agent evolution standard should specify what persists, what updates, what is measured, and what is auditable, distinguishing state components and update policies while tracking regression, drift, recurring failures, and inspectable traces.Updates interacting with tool permissions or governance boundaries may require review, replay, or stronger verification; behavioral evaluation alone is insufficient for persistent risks [14]. Reward-hacking risks and auditable traces for memory writes, routing changes, permissions, and collaboration failures are also identified.Source: §5.3 · pp. 10–11

6 Discussion: Alternative Views and Limitations

The discussion argues that stronger models and end-to-end training do not remove the need for explicit system mechanisms, while system-level evaluation, though costlier, exposes operational factors simple benchmarks can miss.

Objection 1: Stronger models will eventually solve system problems

  • Stronger models may reduce system failures, but explicit mechanisms must govern exposed information, authorized actions, and failure tracing for agents that act on the world.Source: §6 · p. 11

Objection 2: End-to-end training will replace modular systems

  • End-to-end training may improve coordination, but deployment still requires modular boundaries for auditability, permission control, rollback, and provenance.Even an end-to-end policy must act through permission, verification, and audit surfaces.Source: §6 · p. 11

Objection 3: System-level evaluation is too expensive or environment-specific

  • Although system-level evaluation costs more and trace metrics are harder to standardize, it should complement simple benchmarks by exposing operational cost, latency, tool risk, memory drift, and verification overhead.Source: §6 · p. 11

7 Conclusion

The conclusion argues that agentic AI must scale the harness alongside foundation models: the execution system shapes long-horizon behavior, and future evaluation should assess its components rather than only one-shot success.

7 Conclusion

  • System scaling requires improving context construction, trustworthy memory, skill routing, verification, tool governance, communication, and evolution over time.The paper argues that the harness is a primary source of practical capability: comparable models projected onto different harnesses, including Claude Code, OpenClaw, and CheetahClaws, produce qualitatively different agents.Source: §7 · p. 11
  • Stronger foundation models remain essential, but model capability alone is insufficient to analyze long-horizon agents.A mature science of agentic AI must study what the full execution system remembers, retrieves, exposes, permits, verifies, and leaves auditable.Source: §7 · pp. 11–12
  • Future benchmarks should evaluate memory, context, skill routing, orchestration, and verification-and-governance as first-class objects, not only one-shot success.The conclusion identifies scaling the harness alongside model scaling as the next major bottleneck of agentic AI.Source: §7 · p. 12

Related papers

Loading 2605.26112v1…