Source-linked AI summary
Resource Constraints and Performance in Agentic AI Systems
Amaz Salman, Malka Halgamuge, Teo Susnjak
TL;DR
Agentic-system evaluation must connect verified task completion with the resources and execution records that produce it. This paper compares OpenClaw and NanoBot as complete systems across paired primary and instrumented benchmarks. Full-completion rates did not show a statistically conclusive advantage, while the OpenClaw configuration used substantially more wall time and recorded peak memory.
Problem
As agents perform longer tool-mediated sequences, evaluating task completion alone omits the complete system mechanisms and operational burden shaping autonomous action.
Method
The study compares OpenClaw and NanoBot as complete systems using a 100-prompt paired benchmark and a 23-prompt instrumented subset with task, resource, and provenance measures.
Results
The primary benchmark found 31% versus 25% full completion with a six-percentage-point difference and no statistically established advantage, while OpenClaw recorded 2.98× wall time and 19.44× peak memory.
Takeaways & Limitations
Agent-system evaluation should pair verified outcomes with observed resource use and records linking each result to the execution that produced it.
Takeaways & Limitations
The comparison does not isolate harness architecture because configurations differed in inference settings, tool environments, and other execution details.
Abstract
from arXiv · showhide
Progress toward more autonomous AI increasingly depends on agentic systems that combine a language model with tools, memory, state management, and multi-step execution. These mechanisms shape both task capability and operational burden. We compare OpenClaw and NanoBot as complete agentic systems using a paired primary benchmark and a more detailed instrumented subset of paired prompts. In the primary benchmark, the rate of full task completion was 31% for OpenClaw and 25% for NanoBot, a six-percentage-point difference with a 95% task-bootstrap interval from -3 to 15 percentage points, providing no statistically established full-completion advantage for either system. In the instrumented layer, both systems achieved 26% full completion, while NanoBot reached at least partial completion on 43% of prompts compared with 26% for OpenClaw. OpenClaw took longer on 83% of prompts and had a higher recorded peak-memory value on every prompt, with geometric mean ratios of 2.98 for wall time and 19.44 for peak memory. Among the ten detailed-layer prompts on which at least one system achieved partial or full completion, NanoBot weakly dominated on eight; across all 23 prompts, however, ten of its eighteen dominance cases were cheaper joint failures. Outcome labels differ across the two evidence layers, showing why agent-system evaluation should connect capability and resource measurements to attempt-level execution and scoring provenance. These findings show that progress toward more autonomous AI should be evaluated through verified task completion, observed resource use and records linking each result to the execution that produced it.
1 Introduction
Agentic AI evaluation must examine complete systems rather than language models alone, because harnesses shape both autonomous task execution and operational burden. The study therefore compares capability and cost in OpenClaw and NanoBot as complete agentic systems.
- Agentic systems plan, use tools, inspect intermediate results, and continue acting until reaching a verifiable end state.
- Autonomous action exposes weaknesses in state grounding, action preconditions, and prediction of downstream effects.
- Capability-cost evaluation measures whether an agentic system reaches a defined outcome and records the operational burden required to do so.
- OpenClaw and NanoBot are compared as complete systems across paired task outcomes, operational burden, and two evidence layers.
2 Background
Agent harnesses connect language models to tools, memory, state, and verification, extending functional reach while introducing additional resource demands and failure paths. Background research motivates evaluating these complete systems in executable environments with attention to cost, reliability, and failure modes.
- Agent harnesses route tool calls, maintain memory, manage execution state, retry failed steps, and terminate or escalate stalled tasks.
- Richer harnesses can improve task performance through iterative tool use and reflection, but may increase invocations, latency, memory demand, and failure surfaces.
- OpenClaw and NanoBot represent persistent general-purpose and smaller-runtime architectural families, respectively.
- World models connect observations, available actions, and effects, while external memory, state tracking, search, and verification can compensate for model weaknesses.
- Executable-agent benchmarks judge intermediate steps, tool use, recovery, constraint adherence, and externally checkable end states rather than one-shot text alone.
- Repeated attempts are needed to assess reliability, while benchmark horizon strata express categorical execution demand rather than independent operating time.
3 Systems and Empirical Scope
The empirical comparison treats OpenClaw and NanoBot as complete agentic systems and defines scope through their recorded system identities and benchmark layers. The detailed data identify containerized agent-mode configurations, while the broader benchmark records systems by product name.
- The detailed execution data identify openclaw-container-v3 and nanobot-container-v3, both operating in agent mode.
- The primary benchmark records systems by product name, while complete framework-version, model-setting, and attempt-level run information is not retained.
- Table 2 summarises the systems and the scope of the comparison.
4 Data and Methods
The study combines a 100-prompt paired benchmark with a 23-prompt instrumented layer, analyzing capability outcomes and resource measures at the prompt level. Cleaning, paired bootstrap inference, and provenance checks are used while preserving the layers as separate analyses.
- Evidence layers and analytical grain: 100 primary-layer prompts and 23 detailed-layer prompts were evaluated once by each system, with prompt identifiers linking the layers at task level.
- Evidence layers and analytical grain: The detailed layer adds execution duration, CPU and memory measures, traces, retries, termination reasons, failure types, and manually assessed outcomes.
- Evidence layers and analytical grain: Outcome scores differ across layers, so the 100-prompt benchmark and 23-prompt execution data are analyzed separately rather than pooled.
- Scope and interpretation: Because inference settings, tool interfaces, and execution details differed, the comparison does not isolate the causal effect of harness architecture alone.
- Outcome definitions and inference: Full completion is the primary capability outcome, while paired risk differences use 20,000 prompt-level bootstrap resamples preserving system pairing.
- Data cleaning and quality controls: Resource analysis excludes inconsistent full-benchmark fields, including ten simultaneous-zero records, while detailed peak-memory values are converted from KB to MiB.
- Outcome definitions and inference: A system weakly dominates when it achieves an equal or higher outcome with equal or lower wall time and peak memory, with at least one strict improvement.
5 Results
Across the paired benchmark, OpenClaw has a small, statistically unresolved completion advantage, while prompt-level outcomes vary substantially. In the detailed layer, NanoBot achieves comparable full completion with lower observed resource use and more partial progress, though dominance partly reflects cheaper joint failures and dataset outcomes differ.
- Overall benchmark outcomes: 31% versus 25% full completion favors OpenClaw by six percentage points, but the 95% interval spans −3 to 15 percentage points.The paired risk difference is 0.06, and the exact McNemar test gives p = 0.286.
- Overall benchmark outcomes: 15 OpenClaw failures become NanoBot partial completions, while 15 NanoBot failures become OpenClaw partial completions.Both systems fail 28 prompts and reach full completion on 17, while ten NanoBot failures become OpenClaw full completions and five reverse.
- Performance across benchmark horizon strata: Each secondary ordinal-effect interval crosses zero, with estimates ranging from 0.133 in short prompts to 0.014 in long prompts and −0.087 in the detailed subset.The change in direction indicates dependence on the evidence layer and prompt set.
- Performance across benchmark horizon strata: OpenClaw and NanoBot both score lower from short to long prompts, while full completion falls to 14% and 6%, respectively.OpenClaw failure rises from 23% to 69%, and NanoBot failure rises from 43% to 63%; strata combine execution depth with task and environmental differences.
- Operational burden: 34.063 seconds versus 10.446 seconds is the median wall time for OpenClaw and NanoBot, respectively, across 23 detailed prompts.OpenClaw takes longer on 19 prompts; its geometric mean time ratio is 2.98, while its peak-memory ratio is 19.44 and memory is higher on all prompts.
- Resource-bounded completion and dominance: Six full completions are achieved by each system in the detailed subset, while NanoBot adds four partial completions and weakly dominates on eight of ten non-joint-failure prompts.Across all 23 prompts, NanoBot weakly dominates on 18, including ten cheaper joint failures.
- Outcome consistency across datasets: The same 23 prompts receive different outcome labels across datasets: OpenClaw differs on 8 prompts and NanoBot on 10.The records do not establish whether differences reflect new executions, environment changes, or rescoring, so the datasets are analyzed separately.
6 Discussion
The discussion frames agent evaluation as a joint capability–resource question for complete systems, while cautioning against attributing observed costs to harness sophistication. Detailed results show similar completion counts but large operational differences and motivate provenance-aware, controlled evaluation.
- Comparative outcomes: NanoBot weakly dominates on eight of ten prompts where at least one system makes verifiable progress.The comparison is limited to prompts with partial or full progress and does not establish a causal advantage for a particular harness design.
- Comparative outcomes: The OpenClaw configuration recorded greater wall-time and peak-memory values without a demonstrated completion advantage under the available measurement conditions.Inference settings, tools, and incompletely documented execution conditions prevent interpreting the difference as the causal cost of harness sophistication.
- Resource-aware evaluation: Harness mechanisms can support autonomous action through external state, search, verification, permissions, and tool feedback, but their resource burden must be measured.The study pairs task outcomes with wall time and peak memory to evaluate this systems trade-off.
- Resource-aware evaluation: Threefold wall-time and nineteenfold peak-memory differences accompany equal full-completion counts in the detailed results.The pattern supports evaluating capability together with operational burden rather than task outcomes alone.
- Evaluation implications: Linked execution traces could support behavioural analysis of state errors, constraint violations, and missing verification without claiming access to internal representations.The present data do not isolate these mechanisms or reveal either model’s internal representation.
7 Limitations
Operational comparisons are constrained by incomplete provenance, heterogeneous task environments, single-run measurement, and subjective completion assessment. The detailed subset also has limited scope because horizon overlaps with task category and environmental demand.
- Scope and reliability: Each system–prompt pair was evaluated once, so the benchmark estimates comparative performance rather than repeated-run reliability.The purposive 23-prompt detailed subset also differs in category and outcome mix from the remaining prompts.
- Scope and reliability: Operational findings therefore apply to the detailed subset, whose horizon overlaps with task category and environmental demand.This overlap limits interpretation of horizon-related patterns as isolated causal effects.
- Data and measurement limitations: Incomplete configuration and provenance records, unequal confirmation of access, and external dependencies leave environmental and system differences entangled.The records lack exact repository commits, full configuration records, and a confirmed relationship between the two datasets.
- Data and measurement limitations: Single-assessor completion labels lack independent inter-rater agreement, leaving borderline partial-completion decisions potentially subjective.This limitation is especially relevant where outcome labels depend on assessing partial progress.
- Data and measurement limitations: The operational comparison covers only wall time and recorded peak memory, excluding comparable CPU, token-use, and monetary-cost analysis.The memory ratio additionally assumes the same unit and collection procedure for both systems.
8 Conclusion
The conclusion argues that longer-horizon agent evaluation should treat the complete model–runtime–tool stack as the unit of analysis. It reports higher recorded infrastructure use for OpenClaw without a demonstrated completion advantage and calls for capability, resource, and provenance measurements together.
- Conclusion: Task-completion-only evaluation becomes inadequate as agents undertake longer, more autonomous action sequences.Practical capability depends on tools, memory, state, retries, and verification as well as the operational burden those mechanisms introduce.
- Conclusion: The study evaluates two open-source agentic systems built on the same foundation model as complete systems, pairing task outcomes with measured operational burden.This whole-system comparison does not reduce results to the language model alone.
- Conclusion: OpenClaw recorded higher infrastructure use without a demonstrated completion advantage, while the available data do not isolate whether configuration factors caused the difference.Possible factors include harness architecture, inference settings, tool environments, and other configuration differences.
- Conclusion: Outcome labels changed across evidence layers for over a third of prompts, making run and scoring lineage a first-order requirement of agent evaluation.The conclusion therefore links capability and resource reporting to the execution attempt that produced each result.
Statements and Declarations
The authors report contributions spanning methodology, software, investigation, data curation, analysis, validation, visualization, writing, conceptualization, and supervision.
- Author contributions: Amaz Salman contributed methodology, software, investigation, data curation, formal analysis, writing, and visualization.
- Author contributions: Teo Susnjak contributed conceptualization, methodology, formal analysis, validation, writing, visualization, and supervision.
- Author contributions: Malka N. Halgamuge contributed validation and writing.
A.1 Extended implementation and evaluation provenance
Table 10 presents extended implementation and evaluation provenance for the two agentic systems.
- Table 10 summarizes extended implementation and evaluation provenance for both agentic systems.
A.2 Subset composition and simultaneous-zero audit
The detailed subset differs in outcome composition and category coverage, while a simultaneous-zero audit leaves the primary completion comparison statistically unresolved. Resource summaries favor NanoBot, and cross-layer outcome changes require attempt-level provenance before reliability conclusions.
- Subset composition: 23 prompts form the detailed subset, with 7 short, 8 medium, and 8 long prompts.The subset omits Accuracy, Continuity, and Startup, includes the only Traceability prompt, and overrepresents Auditability, Reliability, and Tool Use.
- Subset composition: NanoBot’s subset contains 9 failures, 7 partial completions, and 7 passes, versus OpenClaw’s 10 failures, 5 partial completions, and 8 passes.The subset therefore has a distinct outcome composition, particularly for NanoBot, constraining the scope of its resource results.
- Simultaneous-zero audit: 30 versus 24 full completions remain after excluding affected prompts, with a risk difference of 0.067 and exact McNemar p = 0.263.This sensitivity analysis excludes the union of affected prompt identifiers from both systems, leaving 90 paired prompts.
- Category-level outcomes: 0.50 for Tool Use and −0.75 for Reliability are the extreme category-level ordinal-score differences, each based on four prompts.Orchestration favors NanoBot by −0.50 across three prompts, while Recovery and Memory differ by 0.115 and −0.077 across 13 prompts each; sparse cells and overlapping category and horizon demand limit interpretation.
- Capability-cost position: NanoBot’s mean score is 0.348 versus 0.261, median wall time is 10.446 versus 34.063 seconds, and median peak memory is 136.1 versus 2926.6 MiB.Figure 8 summarizes these capability-cost positions using the detailed subset’s 23 observed prompt pairs and bootstrap resamples.
- Cross-layer concordance: OpenClaw agrees across layers on 15 of 23 prompts, while NanoBot agrees on 13, with disagreements shifting outcomes in both directions.The cross-tabulation is a concordance audit; repeated-run interpretation requires shared attempt-level lineage.