PaperlayerView exact revision on arXiv

Source-linked AI summary

From Model Scaling to System Scaling: Scaling the Harness in Agentic AI

Shangding Gu

arXiv:2605.26112v1cs.AIcs.LG

TL;DR

Agentic AI evaluation often treats the model as primary, despite behavior depending on memory, context, routing, orchestration, verification, and governance. This paper frames progress as scaling the harness, develops a systems framework and reference harness, and concludes that harness architecture is a primary source of practical capability alongside model scaling.

  • Problem

    Agentic AI lacks a sufficiently system-centered evaluation framework for behavior shaped by context, memory, tools, routing, orchestration, verification, and governance.

  • Method

    The paper defines a six-component harness framework and develops CheetahClaws to concretize the framework alongside comparisons with existing harnesses.

  • Results

    Different harnesses projected onto comparable models produce qualitatively different agents, making the harness a primary source of practical capability.

  • Takeaways & Limitations

    Future agentic AI benchmarks should evaluate memory, context, skill routing, orchestration, verification, governance, and long-horizon execution alongside one-shot success.

  • Takeaways & Limitations

    Deployed agents still require modular boundaries because auditability, permission control, rollback, and provenance remain necessary in environments involving private resources and external services.

Abstract

from arXiv · show

This paper studies the next major bottleneck in agentic AI as system scaling, not only model scaling: the design of auditable, persistent, modular, and verifiable architectures around foundation models. We refer to this shift as scaling the harness: treating the structured execution layer around a foundation model as a first-class object of design, evaluation, and optimization. Although recent large language models enable agents to use tools, retrieve information, maintain memory, and execute long-horizon workflows, evaluation remains largely model-centric, often reducing agents to final-task success while treating memory, retrieval, tool use, orchestration, verification, and governance as secondary implementation details. This framing is increasingly inadequate because agent performance emerges from the interaction among the foundation model, memory substrate, context constructor, skill-routing layer, orchestration loop, and verification-and-governance layer. Together, these components form the agent harness, which translates model capability into long-horizon agent behavior. We study scaling the harness through three core bottlenecks: context governance, trustworthy memory, and dynamic skill routing, together with the orchestration and governance mechanisms that coordinate and constrain them. We further outline a research agenda for harness-level benchmarks that go beyond one-shot task success to measure trajectory quality, memory hygiene, context efficiency, communication fidelity, verification cost, and safe evolution over time. To make the discussion concrete, we develop CheetahClaws: https://github.com/SafeRL-Lab/cheetahclaws, a Python-native reference harness, and compare it with Claude Code and OpenClaw. Our main claim is that future progress in agentic AI will depend as much on system design as on stronger foundation models.

1 Introduction … 4 Three Bottlenecks in System Scaling

The paper frames agentic AI as a system-scaling problem centered on the agent harness, then organizes its framework around infrastructure, temporal layers, and three system bottlenecks.

  • 2 Related Work: Related Work situates the paper’s argument within prior research on agentic AI and system scaling.
  • 3.1 Agent Harnesses as System Infrastructure: Agent harnesses are presented as system infrastructure within a framework for system scaling in agentic AI.
  • 3.2 Prompt, Skill, and Memory as Temporal Layers: Prompt, skill, and memory are treated as temporal layers in the system-scaling framework.
  • 4.1 Context Governance: Context governance is identified as the first bottleneck in system scaling.
  • 4.2 Trustworthy Memory: Trustworthy memory is identified as a bottleneck requiring dedicated treatment in system scaling.
  • 4.3 Dynamic Skill Routing and Verification: Dynamic skill routing and verification form a third bottleneck in system scaling.

5 Toward System-Level Evaluation and Agent Evolution

This section addresses system-level evaluation and agent evolution through process metrics, longitudinal evaluation, and standards for safe agent evolution.

  • 5.1 From Outcome Metrics to Process Metrics: The section examines a shift from outcome metrics to process metrics.
  • 5.2 From Single Episodes to Longitudinal Evaluation: It considers moving from single-episode assessment to longitudinal evaluation.
  • 5.3 Standards for Safe Agent Evolution: It proposes standards for safe agent evolution.

6 Discussion: Alternative Views and Limitations · 7 Conclusion · 1 Introduction

Agentic AI should be understood as a system-scaling problem: behavior emerges from how foundation models are embedded in architectures that manage context, memory, tools, orchestration, verification, governance, and adaptation. The paper therefore treats the agent harness as a first-class object of design and evaluation, proposing process-level benchmarks and CheetahClaws as a concrete reference harness.

  • 1 Introduction: Agent behavior depends on the surrounding system, including context construction, memory retrieval, tool invocation, and subagent coordination, not the foundation model alone.
  • 1 Introduction: The paper frames agentic AI as a system-scaling problem spanning memory, context construction, skill routing, orchestration, and verification-and-governance.
  • 1 Introduction: Production harnesses such as Claude Code and OpenClaw, alongside research systems such as SWE-agent, demonstrate that harness design can materially shape practical agent capability.
  • 1 Introduction: Benchmark results can conflate model capability with costs, prompting strategy, demonstrations, and interface design, making some reported scores non-Pareto-optimal under controlled comparisons.
  • 1 Introduction: Long-horizon performance requires reusable skills, persistent memory, disciplined context construction, and verification-aware execution beyond local prompt engineering.
  • 1 Introduction: The proposed harness-level framework separates base-model reasoning from memory, context construction, skill routing, orchestration, and verification-and-governance as system factors.
  • 1 Introduction: The evaluation agenda emphasizes trajectory quality, memory hygiene, context efficiency, verification cost, safe evolution, and robustness under repeated use, using CheetahClaws alongside Claude Code and OpenClaw.

2 Related Work

Related work spans tool-using language models, persistent memory, reusable skills, multi-agent coordination, and executable benchmarks. This paper reframes these developments through system scaling and compares Claude Code, OpenClaw, and CheetahClaws.

  • Agentic coding systems and harness engineering: Modern coding agents build on reasoning–acting, tool invocation, and self-correction, packaging these primitives into runtimes with tools, subagents, hooks, and persistent project memory.The passage identifies ReAct, self-taught tool invocation, verbal self-correction, Claude Code, and Codex-style harness engineering as relevant precedents.
  • Context, memory, and retrieval: Retrieval, hierarchical memory management, and growing skill libraries treat external memory and reusable capabilities as system components for agent behavior.The cited work includes retrieval-augmented generation, MemGPT, and Voyager.
  • Skills and multi-agent coordination: Reusable skills and multi-agent frameworks support durable callable behavior and agent-to-agent communication, including orchestrator-plus-subagent configurations.The passage names AutoGen, MetaGPT, and CAMEL as multi-agent frameworks and describes skills as behavior offloaded from prompts.
  • Benchmarks, governance, and agent evolution: Executable multi-step benchmarks evaluate agents as systems, but single-episode success still dominates while memory quality, context efficiency, communication fidelity, and safe evolution remain largely unmeasured.The passage also situates this gap alongside surveys of LLM-based agents and catalogues of agentic safety threats.
  • Contribution: This work reframes prior developments through system scaling and concretizes the engineering perspective by comparing Claude Code, OpenClaw, and CheetahClaws.CheetahClaws is identified as a Python-native reference harness.

3 System Scaling: A Framework for Agentic AI

System scaling treats the harness surrounding a foundation model as the primary object of design: a structured system that allocates information, computation, authority, and verification across six interacting components. This framework distinguishes short-horizon prompting, reusable skills, and longitudinal memory while emphasizing that long-horizon behavior depends on their orchestration, governance, and deployment priorities.

  • System scaling framework: The harness comprises reasoning, memory, context construction, skill routing, orchestration, and governance components that mediate between user intent, model outputs, and the external environment.Model scaling primarily improves reasoning, whereas system scaling targets the surrounding system factors.
  • System scaling framework: Orchestration wraps the control loop: context draws from memory, skill routing dispatches tools or subagents, and governance verifies intermediate reasoning and external actions before memory updates.This interaction makes the harness more than a simple model interface or isolated prompt-conditioned foundation model.
  • System scaling framework: Each component is a distinct intervention point, but the decomposition is conceptual rather than a quantitative model because its factors are neither strictly orthogonal nor jointly measurable as an equation.The paper uses the axes to distinguish engineering and research levers affecting long-horizon behavior.
  • Deployment-dependent design: Comparable harnesses can implement context governance, memory trust, skill routing, and auditability differently because deployment priorities shape their system designs more than the foundation model alone.Illustrative priorities include reliable use, multi-channel personal assistance, and transparency and reproducibility.
  • Temporal axes of system scaling: Prompt, skill, and memory operate at different temporal scales: prompt controls the immediate task, skill provides reusable execution patterns, and memory preserves information across turns or sessions.Prompting is flexible but brittle for long-horizon structure; skills improve reuse but create routing demands; memory can suffer drift, over-generalization, and pollution.

4 Three Bottlenecks in System Scaling

System scaling is constrained by three coupled bottlenecks—context construction, memory, and skill routing—whose effectiveness depends on verification and governance. Each bottleneck requires a system move that governs selection, trust, or allocation rather than relying on raw capacity alone.

  • Section framing: The three factors are presented as the system components where model scaling alone has been least sufficient, with verification and governance tightly coupled to their operation.The section organizes each component around four subaxes, its dominant failure mode, and the system move that addresses it.
  • Context construction: Context governance must construct the minimum sufficient context by balancing relevance, compactness, traceability, and freshness.Larger context windows increase capacity but do not ensure that the right evidence is attended to or that added tokens improve performance.
  • Memory: Trustworthy memory treats stored information as a verifiable hypothesis, because stale-but-confident items can remain retrievable while their targets silently drift.Retrieval should combine staleness penalties and confidence-gated risk with relevance, then re-check content against the live environment while retaining prior verification.
  • Skill routing: Effective skill use requires scoped capabilities, selective routing, composability, and explicit verification of every skill output.The principal threat is confident-but-unchecked specialized output, so routing should become adaptive through online subtask estimates, confidence-aware escalation, and mixture-style composition.

5 Toward System-Level Evaluation and Agent Evolution

System-scaled agents require evaluation that jointly measures outcomes and the processes producing them, including context, computation, retrieval, verification, risk, and long-horizon behavior. They also need explicit evolution standards governing persistence, updates, measurement, and auditability over time.

  • System-level evaluation: Single-score benchmarks can conflate foundation-model capability with harness design, especially when small system choices accumulate across long-horizon and multi-agent tasks.Relevant choices include file-inspection order, retained facts, test timing, and recovery from failed actions.
  • System-level evaluation: Evaluation should report task outcomes jointly with process metrics covering context and computation use, trajectory structure, retrieval and verification, and incurred risk.This extends existing benchmarks rather than replacing their executable, interactive, and browser-based task evaluations.
  • System-level evaluation: 80% of performance variance in a BrowseComp-based analysis was explained by token usage alone, rising to 95% after adding tool-call count and model choice.The result suggests that multi-agent evaluation must expose compute allocation and system configuration rather than report success alone.
  • System-level evaluation: Next-generation benchmarks should measure memory retrieval precision and hygiene, context efficiency, communication fidelity, trajectory or session drift, verification-aware recovery, and safety under tool access.These dimensions address failures from persistent state, including contamination, staleness, over-generalization, and privacy leakage.
  • Agent evolution: An agent evolution standard should specify what persists, what updates, what is measured, and what is auditable, separating state components and tracking regression, drift, recurring failures, and reward hacking.Inspectability should include memory writes, routing changes, tool permissions, and collaboration failures; otherwise learning agents may become opaque accumulations of prompts, notes, and heuristics.

6 Discussion: Alternative Views and Limitations

The system-scaling claim addresses three objections: stronger models may internalize system functions, end-to-end training may replace modularity, and system-level evaluation may be too expensive or environment-specific. The discussion argues that deployed agents still require explicit controls and evaluation of operational factors beyond endpoint accuracy.

  • Objection 1: Stronger models will eventually solve system problems: Stronger models may improve agent behavior, but deployed failures also involve stale memory, over-broad tool permissions, and missing provenance.The passage distinguishes failures of next-token prediction from system-level failures.
  • Objection 2: End-to-end training will replace modular systems: End-to-end training may improve coordination, but deployed agents still require modular boundaries for auditability, permission control, rollback, and provenance.These requirements arise when agents operate over private files, credentials, tools, repositories, browsers, and external services.
  • Objection 3: System-level evaluation is too expensive or environment-specific: System-level evaluation costs more and is harder to standardize, yet deployment depends on cost, latency, tool risk, memory drift, and verification overhead.Evaluation protocols should expose these factors rather than abstract them away through endpoint accuracy alone.

7 Conclusion

Future progress in agentic AI requires scaling the harness: improving the execution architecture around models, not only increasing model capability. Evaluation should treat memory, context, routing, orchestration, verification, governance, and auditability as first-class concerns alongside one-shot success.

  • System scaling: Agentic AI is shifting from isolated model inference toward persistent system execution shaped by tools, memory, repositories, browsers, subagents, and external services.The paper identifies system scaling as improving context construction, trustworthy memory, skill routing, action verification, tool governance, role communication, and system evolution.
  • System scaling: Stronger foundation models remain essential, but model capability alone is insufficient for analyzing long-horizon agents.The conclusion preserves model scaling while arguing that it is no longer a sufficient unit of analysis.
  • Evaluation: Future benchmarks should evaluate memory, context, skill routing, orchestration, verification, and governance as first-class objects rather than measuring only one-shot success.The full execution system includes what agents remember, retrieve, expose to models, permit, verify, and leave auditable.
Loading 2605.26112v1…