PaperlayerView exact revision on arXiv

Source-linked AI summary

Resource Constraints and Performance in Agentic AI Systems

Amaz Salman, Malka Halgamuge, Teo Susnjak

arXiv:2608.27886v1cs.AI

TL;DR

Agentic-system evaluation must account for both task capability and resource burden, not completion alone. We compare OpenClaw and NanoBot as complete systems on paired benchmark prompts and an instrumented subset; full completion showed no established advantage, while OpenClaw had greater measured resource use.

  • Problem

    Evaluating increasingly autonomous agents by task completion alone is inadequate because practical capability also depends on complete-system mechanisms and their operational burden.

    Source: §8 · p. 15

  • Method

    We compared OpenClaw and NanoBot as complete agentic systems using paired primary-benchmark prompts and a 23-prompt instrumented subset with added execution and resource measures.

    Source: §4.1 · p. 5

  • Results

    Full completion differed by 6 percentage points (95% task-bootstrap interval −3 to 15), with no established advantage; OpenClaw-to-NanoBot geometric mean ratios were 2.98 for wall time and 19.44 for peak memory.

    Source: §5.1 · p. 8, §5.4 · p. 10

  • Takeaways & Limitations

    Completion alone is inadequate; evaluation should pair task outcomes with operational burden for complete agentic systems.

    Source: §6 · p. 13, §8 · p. 15

    Incomplete provenance records and unconfirmed equivalent task access mean outcomes may reflect environmental as well as system differences.

    Source: §7 · p. 15

Abstract

from arXiv · show

Progress toward more autonomous AI increasingly depends on agentic systems that combine a language model with tools, memory, state management, and multi-step execution. These mechanisms shape both task capability and operational burden. We compare OpenClaw and NanoBot as complete agentic systems using a paired primary benchmark and a more detailed instrumented subset of paired prompts. In the primary benchmark, the rate of full task completion was 31% for OpenClaw and 25% for NanoBot, a six-percentage-point difference with a 95% task-bootstrap interval from -3 to 15 percentage points, providing no statistically established full-completion advantage for either system. In the instrumented layer, both systems achieved 26% full completion, while NanoBot reached at least partial completion on 43% of prompts compared with 26% for OpenClaw. OpenClaw took longer on 83% of prompts and had a higher recorded peak-memory value on every prompt, with geometric mean ratios of 2.98 for wall time and 19.44 for peak memory. Among the ten detailed-layer prompts on which at least one system achieved partial or full completion, NanoBot weakly dominated on eight; across all 23 prompts, however, ten of its eighteen dominance cases were cheaper joint failures. Outcome labels differ across the two evidence layers, showing why agent-system evaluation should connect capability and resource measurements to attempt-level execution and scoring provenance. These findings show that progress toward more autonomous AI should be evaluated through verified task completion, observed resource use and records linking each result to the execution that produced it.

1 Introduction

Evaluating agentic AI requires assessing complete systems because their harnesses shape both task capability and operational burden. This study compares OpenClaw and NanoBot across paired task outcomes, resource use, and outcome provenance.

System-level evaluation

  • Because the software harness shapes an agent’s action space, memory path, tool interface, orchestration, recovery, and footprint, evaluation must cover the complete system.This shifts evaluation beyond the language model alone.Source: §1 · p. 1

Motivation

  • Grounding gaps and compounding multi-step errors motivate testing whether harness mechanisms convert state tracking, search, retries, feedback, and verification into task completion [3] [4].Predictive world models are treated as core to reasoning across states and planning over multiple time scales.Source: §1 · pp. 1–2

AGI context

  • The study treats harness evaluation as relevant to AGI research but does not claim to measure AGI; performance, generality, and autonomy are distinct dimensions [5].Agentic systems combine planning, memory, tool use, environmental interaction, and multi-step action.Source: §1 · p. 2

Evaluation rationale

  • The study pairs task outcomes with wall time and peak memory to evaluate the capability–cost trade-off of harnesses supporting autonomous action.The paper frames resource-bounded task completion as one empirical step toward evaluating architectures for more general autonomous intelligence.Source: §1 · p. 2

Deployment context

  • Capability–cost evaluation measures whether an agent reaches a defined outcome and the operational burden required, but a benchmark pass alone does not establish labor productivity or economic value.Organisational deployment also depends on infrastructure, human supervision, correction, and risk-control demands.Source: §1 · p. 2

Study scope

  • The study compares OpenClaw and NanoBot as complete systems across paired task outcomes, resource burden, two evidence layers, and outcome provenance.It asks how the systems differ in completion, ordinal outcomes, wall time, peak memory, resource-bounded completion, prompt-level dominance, and cross-layer outcome agreement.Source: §1 · p. 2

2 Background

This background treats the agent harness—not the language model alone—as the unit that turns model outputs into actions, and motivates evaluating complete systems through task outcomes, resource use, and reliable execution evidence.

2.1 Agent harnesses and autonomous action

  • Agent harnesses combine tool routing, memory, state, and execution control; richer tool use can aid performance but adds orchestration, resource demand, and failure or attack surfaces [7] [8] [10] [11] [12] [13] [14].Meng et al. use “agent harness” for a runtime combining an execution loop, tool registry, context manager, state store, lifecycle hooks, and evaluation interface.Source: §2.1 · pp. 2–3

2.2 Position within the agent-system ecosystem

  • OpenClaw and NanoBot are complete, runnable systems in adjacent harness families, unlike orchestration libraries; the comparison focuses on state, constraints across tool calls, and outcome verification.Software-development agents and platforms offer related tool-mediated planning and verification, but their scope differs from persistent, cross-domain assistance.Source: §2.2 · p. 3

2.3 Grounding, world models, and prospective action

  • Grounded action requires representing state, preconditions, and consequences; explicit search, planners, and verifiers can scaffold weak model reasoning but add calls, latency, memory demand, and failure paths [3] [4] [31] [35].PlanBench and CLadder expose difficulties with state transitions, executable plans, and causal queries, while chain-of-thought does not guarantee sound search or out-of-distribution constraint satisfaction [32].Source: §2.3 · pp. 3–4

2.4 Evaluation beyond final-answer success

  • Agent benchmarks increasingly assess tool use, error recovery, constraints, and externally checkable end states; this study combines paired capability transitions, operational ratios, resource-bounded completion, dominance, and lineage auditing.The broader evaluation literature includes τ-bench’s end-state verification and repeated-trial consistency, MASEval’s system-topology focus, and studies of multi-agent latency, planning, coordination, and scalability [8] [43] [44] [45].Source: §2.4 · p. 4

2.5 Cost, reliability, and failure modes

  • Reliability here means repeated attempts under pinned conditions, and agent evaluation should measure cost alongside accuracy; the study also distinguishes infrastructure fragility from decision fragility [12] [49].The study adds resource-budget completion curves, prompt-level dominance, and an audit of evidence-layer lineage.Source: §2.5 · p. 4
  • Benchmark capability, task-time reduction, and organizational value require separate measurement: workplace studies report mixed evidence, and time-horizon estimates do not establish independent operating time or delegation of all shorter tasks [51, 52, 53, 54, 55, 56].One randomized trial found measured completion time increased 19% with early-2025 AI tools; a later study could not firmly estimate newer-agent speedups because of participant selection and unreliable time measurement.Source: §2.5 · pp. 4–5

3 Systems and Empirical Scope

The empirical comparison covers two agentic systems, with detailed execution data identifying their container versions and primary benchmark records retaining less run-level detail.

3 Systems and Empirical Scope

  • The detailed execution data compare openclaw-container-v3 and nanobot-container-v3, both operating in agent mode.Table 2 summarizes the systems and comparison scope.Source: §3 · p. 5, Table 2 · p. 5
  • Primary benchmark records identify systems by product name but omit complete framework-version, model-setting, and attempt-level run information; implementation records provide partial details summarized in Table 4.Source: §3 · p. 5

4 Data and Methods

The methods pair OpenClaw and NanoBot by prompt across a 100-prompt primary benchmark and a 23-prompt instrumented subset, then analyze capability and operational burden separately.

4.1 Evidence layers and analytical grain

  • 100 paired prompt observations per system form the primary layer, while 23 matched prompts add execution measures; records link the layers at task, not attempt, level.The primary layer records outcomes and operational fields; the detailed layer adds execution, resource, trace, retry, termination, and manually assessed outcome data. Both systems used gpt-4o-mini, but inference settings and tool environments differed.Source: §4.1 · pp. 5–6
  • The 23-prompt layer is analyzed separately, not pooled or treated as repeated trials; scores match primary records on 15 OpenClaw and 13 NanoBot prompts.The subset has 7 short, 8 medium, and 8 long prompts, versus 23, 27, and 27 among the remaining 77; category coverage and outcome composition also differ, favoring NanoBot in the subset.Source: §4.1 · p. 6

4.2 Data cleaning and quality controls

  • Ten full-benchmark records with simultaneous zero resource values were treated as missing for operational analysis, while their failure outcomes were retained.Peak-memory values were converted from KB to MiB assuming KiB; CPU usage was excluded because its denominator was undefined and values exceeded 100%. Failure causes were available for only seven of 30 failed detailed executions, so failure types were not compared.Source: §4.2 · p. 7

4.1 Evidence layers and analytical grain

  • Full completion is the primary capability outcome, partial-or-better completion is secondary, and mean ordinal score is a secondary summary.Full completion means reaching the specified end state, partial completion means verifiable task-relevant progress short of that end state, and failure means no verifiable progress or inability to execute.Source: §4.3 · p. 7, §4.1 · p. 7

4.3 Statistical analysis

  • Paired binary outcomes use 20,000 prompt-level bootstrap resamples and exact McNemar tests; resource ratios use paired bootstrap intervals and exact sign tests.Intervals reflect sensitivity to the observed prompt composition, not run-to-run reliability or future-task uncertainty. Resource-bounded completion and weak dominance are also assessed; category analyses are exploratory, with nine of 17 categories containing fewer than five prompts.Source: §4.3 · pp. 7–8

5 Results

The primary benchmark shows a small, uncertain full-completion difference, while the detailed subset shows substantially higher recorded resource use for OpenClaw and more partial completions for NanoBot. Results also vary across horizon strata, and outcome labels differ between datasets for some prompts.

5.3 Category and process patterns

  • Among categories with at least seven prompts, paired estimates favoured OpenClaw in six categories and NanoBot in Memory, but all intervals included zero.Larger contrasts appeared in Tool Use, Accuracy, Tool Routing, Navigation, Orchestration, and Reliability, each with only three or four prompts.Source: §5.3 · p. 8

5.4 Operational burden

  • OpenClaw’s median wall time was 34.063 seconds versus 10.446 seconds for NanoBot, and it took longer on 19 of 23 prompts.The geometric mean OpenClaw-to-NanoBot time ratio was 2.98 (95% paired task-bootstrap interval 1.78 to 5.06); recorded peak memory was higher for OpenClaw on all 23 prompts, with a geometric mean ratio of 19.44. These measurements cover a purposively selected, horizon-balanced subset and use the stated source-unit assumption for memory.Source: §5.4 · pp. 8, 10, Figure 4 · p. 11

5.2 Performance across benchmark horizon strata

  • OpenClaw full completion fell from 53% of short prompts to 14% of long prompts, while NanoBot fell from 47% to 6%.Failure shares rose from 23% to 69% for OpenClaw and from 43% to 63% for NanoBot; the strata differ in task composition, so these are associations rather than causal effects of horizon.Source: §5.2 · p. 8, Figure 3 · p. 10

5.1 Overall benchmark outcomes

  • 31% of prompts reached full completion with OpenClaw versus 25% with NanoBot, a 0.06 paired risk difference (95% task-bootstrap interval −0.03 to 0.15).The interval includes zero; Figure 1 shows that both systems failed 28 prompts and completed 17, with disagreements running in both directions.Source: §5.1 · p. 8, Figure 1 · p. 9, Table 6 · p. 11

5.5 Resource-bounded completion and dominance

  • NanoBot reached partial-or-better completion on nine prompts within 30 seconds versus five for OpenClaw, while both reached five full completions.Among ten prompts where either system achieved partial or full completion, NanoBot weakly dominated on eight; across all 23 prompts, ten of its 18 dominance cases were cheaper joint failures.Source: §5.5 · pp. 10, 12, Figure 5 · p. 12

5.6 Outcome consistency across datasets

  • Across the same 23 prompts, OpenClaw’s outcome differed between datasets on eight (34.8%) and NanoBot’s on ten (43.5%).The records do not establish whether differences reflect new executions, changed evaluation conditions, or rescoring, so the datasets are analysed separately rather than as repeated-run reliability evidence.Source: §5.6 · p. 12

6 Discussion

The discussion treats OpenClaw and NanoBot as complete systems, emphasizing their observed task and resource outcomes while cautioning against attributing differences to individual components. It argues that evaluation should connect completion with operational burden and execution evidence.

6 Discussion

  • Eight of ten prompts with verifiable progress weakly favored NanoBot; OpenClaw recorded greater wall time and peak memory without a demonstrated completion advantage.Because inference settings, tools, and execution conditions differed, this is not a causal estimate of the cost of harness sophistication.Source: Figure 6 · p. 13, §6 · p. 13
  • The study compares complete model–runtime–tool stacks on shared prompts and resource budgets, so observed differences cannot be assigned to a particular harness component.Tools, memory, state control, retries, and verification shape whether model outputs become successful actions and what those actions cost.Source: §6 · p. 13

6.1 Implications for AGI evaluation

  • About threefold wall time and nineteenfold recorded peak memory accompanied equal full-completion counts; long-horizon failure was dominant for both systems, but horizon effects remain confounded.These findings motivate evaluating completion and operational burden together, not measuring AGI, grounding, causal reasoning, or internal world-model quality.Source: §6.1 · p. 13
  • Harnesses may support state tracking through recorded observations, precondition checks, alternatives, and action-effect verification, but the data neither isolate these mechanisms nor reveal internal representations.Linked execution traces could support behavioural analysis of state errors, constraint violations, and missing verification.Source: §6.1 · p. 13

6.2 Design requirements for reproducible agent-system evaluation

  • Table 8 is labeled as presenting design requirements for reproducible agent-system evaluation; the supplied passage does not specify those requirements.Source: Table 8 · p. 14

6.3 Implications for organisational deployment

  • Table 9 connects agent benchmarks to organisational deployment, and its caption says the final column states the evidence included in this study.Source: Table 9 · p. 14

7 Limitations

The benchmark estimates comparative performance from single evaluations, while its detailed operational findings apply only to a purposive 23-prompt subset. Incomplete provenance, possible environmental differences, subjective scoring, and limited resource measures constrain interpretation.

7 Limitations

  • Each system–prompt pair was evaluated once, and the purposive 23-prompt detailed subset differs in category and outcome mix, limiting operational findings to that subset.Horizon also overlaps with task category and environmental demand.Source: §7 · p. 15
  • Incomplete implementation and data provenance, uncertain equivalent access to credentials or external resources, and single-assessor non-blinded scoring leave possible environmental and assessment differences.No independent inter-rater agreement was available, so borderline partial-completion decisions may be subjective.Source: §7 · p. 15
  • Operational comparisons cover only wall time and recorded peak memory; unavailable comparable CPU, token, and monetary-cost measures limit the benchmark to technical capability and selected resource demands.The memory ratio assumes both systems used the same unit and collection procedure; the benchmark does not estimate organisational value or test internal model mechanisms.Source: §7 · p. 15

8 Conclusion

Evaluating autonomous agents by task completion alone is inadequate because practical capability depends on system mechanisms and their operational burden. This benchmark found no statistically conclusive full-completion difference, while operational costs and outcome-label changes underscore the need to report resource use and attempt-level provenance.

8 Conclusion

  • Practical agent capability depends on the complete system, including mechanisms for tools, memory, state, retries, and verification, as well as the burden they introduce.As agents take on longer, more autonomous action sequences, task completion alone becomes increasingly inadequate as an evaluation measure.Source: §8 · p. 15
  • Operational burden differed by roughly threefold in wall time and nineteenfold in recorded peak memory, without a demonstrated full-completion advantage for the higher-use OpenClaw configuration.The lightweight system weakly dominated most prompts on which either system succeeded.Source: §8 · p. 15
  • Outcome labels changed across evidence layers for over a third of prompts, making run and scoring lineage a first-order requirement of agent evaluation.The proposed repeated-run protocol operationalizes reporting capability jointly with resource budgets and attempt-level provenance.Source: §8 · p. 15
  • The benchmark did not isolate whether higher infrastructure use arose from harness architecture, inference settings, tool environments, or other configuration factors.Both evaluated open-source systems were built on the same foundation model and assessed as complete systems.Source: §8 · p. 15

Statements and Declarations

The declarations assign distinct methodological, technical, analytical, writing, validation, visualization, and supervisory contributions to the three authors.

Statements and Declarations

  • The three authors contributed across methodology, software, investigation, data curation, analysis, writing, validation, visualization, and supervision.Amaz Salman: Methodology, Software, Investigation, Data curation, Formal analysis, Writing – original draft, Visualization. Teo Susnjak: Conceptualization, Methodology, Formal analysis, Validation, Writing – original draft, Writing – review & editing, Visualization, Supervision. Malka N. Halgamuge: Validation, Writing – original draft, Writing – review & editing.Source: Statements and Declarations · p. 15

A Additional analyses … A.5 Cross-layer outcome concordance

The additional analyses characterize the detailed subset’s composition, exploratory category patterns, capability-cost position, and cross-layer outcome disagreements. They also show that the detailed subset differs in outcome composition and that attempt-level lineage is needed to interpret cross-layer differences.

A.1 Extended implementation and evaluation provenance

  • Table 10 presents extended implementation and evaluation provenance for OpenClaw and NanoBot.Source: Table 10 · p. 20

A.2 Subset composition and simultaneous-zero audit

  • The 23-prompt subset contains 7 short, 8 medium, and 8 long prompts, versus 23, 27, and 27 in the remaining 77 prompts.It omits Accuracy, Continuity, and Startup, includes the only Traceability prompt, and has higher shares of Auditability, Reliability, and Tool Use.Source: §A.2 · p. 21
  • In the subset, OpenClaw has 10 failures, 5 partial completions, and 8 passes; NanoBot has 9 failures, 7 partial completions, and 7 passes.The distinct outcome composition, particularly for NanoBot, defines the scope of the detailed resource results.Source: §A.2 · p. 21
  • The ten simultaneous-zero operational rows comprise nine OpenClaw and one NanoBot record, all associated with failures; excluding affected prompts leaves 90 pairs and 30 versus 24 full completions.The resulting risk difference is 0.067, with exact McNemar p = 0.263.Source: §A.2 · p. 21

A.3 Exploratory category-level effects

  • Category estimates range from 0.50 for Tool Use to −0.75 for Reliability, while Orchestration is −0.50, Recovery 0.115, and Memory −0.077.The estimates cover 16 categories with at least three prompts; sparse cells, wide intervals, and overlapping category and horizon membership make them task-dependent exploratory patterns.Source: §A.3 · p. 21, Figure 7 · p. 22

A.4 Capability-cost position

  • In the detailed subset, NanoBot’s mean score is 0.348 versus 0.261, median wall time is 10.446 versus 34.063 seconds, and median peak memory is 136.1 versus 2926.6 MiB.These aggregate capability-cost positions use 23 paired prompts; peak memory is reported under the stated unit assumption.Source: §A.4 · p. 21, Figure 8 · p. 22

A.5 Cross-layer outcome concordance

  • Across layers, OpenClaw agrees on 15 of 23 prompts and NanoBot on 13; their disagreements do not form a uniform upward or downward shift.The changes include outcomes moving in both directions; rerunning, rescoring, environment change, or a combination could produce the pattern, so attempt-level lineage is needed before drawing reliability conclusions.Source: §A.5 · p. 21, Figure 9 · p. 23

Related papers

Loading 2608.27886v1…