PaperlayerView exact revision on arXiv

Source-linked AI summary

ClawEnvKit: Automatic Environment Generation for Claw-Like Agents

Xirui Li, Ming Li, Ion Stoica, Cho-Jui Hsieh, Tianyi Zhou

arXiv:2604.18543v4cs.AIcs.CL

TL;DR

ClawEnvKit addresses the labor and rigidity of manually built claw-agent environments by generating verified environments from natural-language specifications. Its Parser, Generator, and Validator support large-scale benchmarking and on-demand evaluation, with structured harnesses improving performance over ReAct while completion remains the main variation axis. The framework also exposes limitations from mock services and from tasks designed for at most 20 tool-calling rounds.

  • Problem

    Manual environment construction is labor-intensive and static, while training is constrained by whatever tasks users happen to perform.

  • Method

    ClawEnvKit parses natural-language requests, generates task environments, and validates their structure, coverage, feasibility, and consistency.

  • Results

    15.7 percentage points is the maximum structured-harness gain over the ReAct baseline across 8 harness frameworks and 4 model families, while completion remains the primary axis of variation.

  • Takeaways & Limitations

    ClawEnvKit makes evaluation continuously refreshable and user-driven while generating task distributions that adapt to agents’ current weaknesses.

  • Takeaways & Limitations

    Evaluation uses predictable mock services, and the current generation and scoring pipeline is designed for tasks completable within 20 tool-calling rounds.

Abstract

from arXiv · show

Constructing environments for training and evaluating claw-like agents remains a manual, human-intensive process that does not scale. We argue that what is needed is not just a dataset, but an automated pipeline capable of generating diverse, verified environments on demand. To this end, we introduce ClawEnvKit, an autonomous generation pipeline that instantiates this formalism from natural language descriptions. The pipeline comprises three modules: (1) a parser that extracts structured generation parameters from natural language input; (2) a generator that produces the task specification, tool interface, and scoring configuration; and (3) a validator that enforces feasibility, diversity, structural validity, and internal consistency across the generated environments. Using ClawEnvKit, we construct Auto-ClawEval, the first large-scale benchmark for claw-like agents, comprising 1,040 environments across 24 categories. Empirically, Auto-ClawEval matches or exceeds human-curated environments on coherence and clarity at 13,800x lower cost. Evaluated across 4 model families and 8 agent harness frameworks, we find that harness engineering boosts performance by up to 15.7 percentage points over a bare ReAct baseline, completion remains the primary axis of variation with no model saturating the benchmark, and automated generation enables evaluation at a scale previously infeasible. Beyond static benchmarking, ClawEnvKit enables live evaluation: users describe a desired capability in natural language and obtain a verified environment on demand, turning evaluation into a continuous, user-driven process. The same mechanism serves as an on-demand training environment generator, producing task distributions that adapt to an agent's current weaknesses rather than being bounded by existing user logs.

1 Introduction

ClawEnvKit addresses the cost and rigidity of manually constructing claw-agent environments with an automated, on-demand generation pipeline. It produces verified environments and Auto-ClawEval, while supporting live evaluation and adaptive training.

  • Motivation: Manual environment construction limits the scalability of claw-agent training and evaluation because benchmarks require substantial labor and become static after release.Training is constrained by observed user tasks, while benchmarks require hundreds of person-hours and remain fixed once released.
  • ClawEnvKit: ClawEnvKit converts natural-language specifications into verified environments through Parser, Generator, and Validator modules.The pipeline structures specifications, instantiates task environments, and enforces structural and semantic correctness in isolated sandboxes.
  • Auto-ClawEval: 1,040 environments across 24 semantic categories form Auto-ClawEval, the first large-scale cross-harness and cross-backbone benchmark for claw-like agents.The benchmark evaluates 8 agent harness frameworks and 4 model families.
  • Empirical findings: 15.7 percentage points is the maximum gain from structured harnesses over the ReAct baseline, while the benchmark remains unsaturated by current models.Full and compact benchmark scores differ by less than 2%, supporting scalable evaluation without sacrificing evaluation quality.
  • Beyond static benchmarks: Live evaluation lets users request verified environments in natural language, and the same mechanism generates training distributions adapted to agents’ weaknesses.This shifts evaluation from a static benchmark toward a continuous, user-driven process.

2 Relate Work

Prior claw-agent training and evaluation resources are either labor-intensive and fixed or limited by available user interaction data. ClawEnvKit extends automatic environment synthesis to claw-like agents and combines generated tasks with broad evaluation support.

  • Manual construction: Hand-crafted interactive, GUI, and web-agent environments require substantial engineering effort for each domain.These resources support multi-turn or functional-task evaluation but do not scale cheaply across domains.
  • Automatic synthesis: Automatic synthesis systems generate environments for GUI, API, coding, software-engineering, and terminal-use agents through specialized pipelines.Examples include AgentStudio, SWE-smith, R2E-Gym, RandomWorld, Agent World Model, and Endless Terminal.
  • ClawEnvKit’s position: Auto-ClawEval combines auto-generated tasks, universal verification, continuous scoring, safety gates, robustness testing, and support for the full claw-agent family.The comparison identifies this combination as unique among the listed frameworks.
  • Training resources: Existing claw-agent training uses real-user trajectories, whose diversity and volume constrain scaling.OpenClaw-RL and MetaClaw collect trajectories from real interactions for agent training.
  • Benchmark limitations: Fixed human-authored task distributions limit the scalability, diversity, and real-world coverage of recent claw-agent benchmarks.The cited benchmarks include ClawArena, ClawsBench, Claw-Eval, and SkillsBench.

3 Formalizing Environments for Claw-like Agents

ClawEnvKit formalizes an agent environment as a task specification, interaction interface, and evaluation functional rather than an explicitly enumerable state space. This representation supports environments whose effective state is driven by natural language, tool outputs, and interaction history.

  • Motivation: Modern claw-agent environments operate over effectively infinite state spaces shaped by natural-language context, tool outputs, and interaction history.Their implementations remain finite even though the agent-facing state is not explicitly enumerable.
  • Implication: The formalism makes automated generation tractable by generating a finite (P, M, C) specification without explicitly modeling every state transition.A correct state-based grader would require understanding the environment’s full execution semantics.
  • Environment formalism: An environment is defined as the three-tuple E = (P, M, C).P is the task specification, M the interaction interface, and C the evaluation functional.
  • Environment formalism: P is a natural-language task specification, while M = (T, O) contains callable tools and an audit log of tool calls and server outcomes.The audit log records parameters and server-side results for the interaction.
  • Environment formalism: C comprises weighted evaluators that score agent trajectories against the audit log.Each component evaluates a trajectory property using the trajectory and recorded observations.

4 ClawEnvKit: A Scalable Framework for Automated Environment Generation

ClawEnvKit automates environment construction from natural-language specifications and verifies the resulting task, tools, data, and scoring logic. It then executes agents in isolated sandboxes and grades their trajectories using safety, completion, and robustness checks.

  • Framework overview: Manual construction requires writing instructions, verification logic, and correctness checks, whereas ClawEnvKit generates verified environment sets for evaluation and reinforcement-learning training.The system produced 1,040 environments at 80 dollars in API costs using claude-sonnet-4.6.
  • Environment generation: The Parser extracts actions, objects, and constraints from natural-language requests into typed intent units.These units specify what the agent should do, what the environment contains, and what must be satisfied.
  • Environment generation: The Generator creates task specifications, tools, pre-loaded data, and scoring criteria while controlling diversity through API-action coverage and deduplication.It can also design and test new mock services for addition to the service library.
  • Environment generation: The Validator checks format, intent coverage, and feasibility before accepting an environment.It verifies required fields and weights, safety checks, intent realization, solvability, and new-service endpoint behavior; failed environments are retried up to three times.
  • Pipeline output: The three modules produce a verified E = (P, M, C) in one pipeline invocation with contamination-free execution and extensibility to new services.The resulting environments support both agent evaluation and reinforcement-learning training.
  • Execution: Agents execute through sandbox initialization, harness preparation, native agent execution, and trajectory collection.The execution design supports multiple harness integration tiers and records server-side audit logs alongside final outputs.
  • Grading: Grading applies a safety gate, evaluates completion components, and measures robustness from recovery after injected API errors.Checks draw on audit logs, agent outputs, and filesystem state; LLM-judge weight is capped at 55%.

5 Experiments

The experiments evaluate automated benchmark quality, scale, and agent performance across models, harnesses, and task categories. Auto-ClawEval supports broad evaluation while its compact variant enables controlled comparison with human-curated tasks.

  • Quality of Generated Environments: Auto-ClawEval-Mini achieves 100% validity, higher Coherence (0.59 vs 0.51), and higher Clarity (3.54 vs 3.38) than Claw-Eval.The comparison is count-matched and uses the three primary quality dimensions: Validity, Coherence, and Clarity.
  • Benchmark Construction: Auto-ClawEval contains 1,040 environments, while Auto-ClawEval-Mini provides a controlled 104-task comparison with Claw-Eval.The full benchmark targets broader coverage and scaling studies; the compact benchmark preserves the automated construction process at comparison scale.
  • Harness Effects: 15.7 points: all structured harnesses outperform the ReAct Agent Loop baseline of 53.3%, with NemoClaw reaching 69.0%.Structured harnesses shift score distributions toward fully solved tasks rather than merely increasing average scores.
  • Performance Dimensions: Completion ranges from 34% to 76%, while safety and robustness remain near-perfect at ≥83% across models and harnesses.The results identify completion as the primary axis of variation and indicate that current frontier models do not saturate Auto-ClawEval.
  • Benchmark Scaling: Auto-ClawEval and Auto-ClawEval-Mini differ by less than 2% across all models and harnesses, supporting the compact benchmark as a lower-cost substitute.The two variants are therefore consistent proxies for evaluation at different scales.
  • Task Diversity: Category difficulty varies substantially: C16 scores 10–71% across harnesses, whereas C21 and C32 are reliably solved above 85%.Different harnesses can have close aggregate scores while exhibiting divergent category-level error patterns.

6 Environment Automation makes a Live Testbed for Agents

Automation turns evaluation from a fixed benchmark into a continuously refreshable, user-driven testbed. ClawEnvKit generates executable environments for custom use cases on demand, expanding coverage beyond existing task distributions.

  • Static test sets can become stale, contaminated, or partially memorized as benchmark data are repeatedly absorbed through model development and deployment.This motivates evaluation that is not tied to a single frozen release.
  • Figure 5 compares harness and model performance against efficiency, while Figure 6 shows per-task score distributions and harness means across 1,040 tasks.The figures support comparing performance-efficiency trade-offs and score variability across harnesses.
  • Automation decouples evaluation from frozen benchmarks and adapts it to users’ custom needs.This supports testing use cases not covered by existing benchmarks and services under development.
  • ClawEnvKit lets users describe workflows and obtain multiple executable task instances with interactively synthesized mock services.The system proposes, adjusts, and confirms endpoints and services without manual rubric writing.
  • Live generation expands evaluation into previously uncovered use cases while remaining continuously refreshable as user needs and real-world environments evolve.The resulting process is described as an “alive” form of evaluation rather than merely a cheaper one.

7 Conclusion

ClawEnvKit automates verified environment construction from natural-language specifications and underpins Auto-ClawEval, a large cross-agent benchmark. The paper frames evaluation as continuously refreshable, user-driven infrastructure that can co-evolve with agent capabilities.

  • ClawEnvKit automates verified environment construction for claw-like agents from natural-language specifications by decoupling what to verify from how to verify it.The framework reduces construction from hours to minutes while matching or exceeding human-written environments on Validity, Coherence, and Clarity.
  • 1,040 environments across 24 semantic categories form Auto-ClawEval, the first large-scale cross-agent, cross-backbone benchmark in the claw ecosystem.The benchmark is built on the automated construction framework.
  • Evaluation becomes continuously refreshable, user-driven, and scalable rather than a frozen artifact that saturates and leaks.The conclusion presents environment generation, training, and evaluation as processes that can co-evolve.

A Limitations and Future Work

The current system is bounded by deterministic mock services, limited task-domain coverage, and task horizons of at most 20 tool-calling rounds. Future work must improve realism, broaden services, and support scalable long-horizon evaluation.

  • Mock services vs. real-world services: Mock services provide reliable deterministic evaluation but omit production conditions such as rate limits, authentication flows, schema drift, and changing external state.Agents that exploit predictable mock responses may still fail on real services.
  • Coverage of real-world task diversity: Auto-ClawEval covers 24 categories, but real workloads also include voice interfaces, GUI automation, multi-agent delegation, and specialized legal, medical, and financial workflows.Extending coverage requires expanding the service library or automating service generation from real OpenAPI specifications.
  • Generation of long-horizon tasks: Current tasks are designed for completion within 20 tool-calling rounds, whereas real workflows may span hours or days with persistent state and human approvals.The isolated-container model supports long-horizon execution in principle, but generation and scoring are not yet designed to produce or evaluate such tasks at scale.

B Automated Evaluation in Context

Automated evaluation is positioned as a scalable, reproducible, continuously refreshable pre-deployment layer that complements human judgment and other evaluation methods. The paper operationalizes environment quality through validity, coherence, and clarity checks, then generates benchmark environments from natural-language specifications.

  • Automated evaluation in context: Automated evaluation complements production monitoring, user feedback, and human studies rather than replacing them.ClawEnvKit targets the pre-deployment layer, which can run on every agent change as capabilities and task distributions evolve.
  • Environment quality dimensions: Validity requires environments to be executable, with valid API references and scoring weights that sum to one; invalid environments are discarded and regenerated.Validity is treated as a prerequisite for coherence and clarity.
  • Environment quality dimensions: Coherence measures whether exposed resources and scoring configurations match the task prompt and its intended outcome.An LLM judge checks that tools supply the resources implied by the prompt and that scoring does not reward an unintended proxy.
  • Environment quality dimensions: Clarity measures whether prompts are understandable and actionable on a 1–5 scale, because ambiguity can increase score variance without measuring capability.The same LLM judge assesses whether a capable agent would understand task success unambiguously.
  • Benchmark construction: Auto-ClawEval contains 1,040 environments across 24 categories, with tasks organized by service, category, and composition type.The benchmark uses 15 mock services, while its task composition is summarized across API-based and file-dependent settings.
  • Generation pipeline: The parser converts one natural-language request into structured services, difficulty, and specific action, object, and constraint atoms.For example, a meeting request yields calendar, contacts, and Gmail services plus scheduling, notification, attendee, and no-deletion requirements.
  • Generation pipeline: The generator produces task specifications, tool interfaces, fixtures, scoring components, and safety checks, while service generation creates missing mock APIs.Diversity is promoted through service-order shuffling, focus-action rotation, and deduplication using recent task names.
  • Validation pipeline: The validator performs sequential structural checks and semantic coverage checks, regenerating configurations when required fields, intent atoms, or safety constraints are uncovered.Action atoms must be callable and scored, objects must appear in fixtures or prompts, and constraints must be enforced.

E.2.2 Error Injection

The evaluation setup combines generated tool interfaces, deterministic and LLM-based grading, robustness scoring, repeated trials, and representative tasks spanning API, coordination, and file-dependent workflows. Error injection and audit-based checks test recovery while preserving safety constraints.

  • Harness integration: Generated tools are exposed through native plugins, stdio MCP servers, or prompt-injected command documentation across agent harnesses.The three tiers adapt the same task tool definitions to different harness integration mechanisms.
  • Execution settings: Agent runs use temperature 0, a 300-second timeout, and up to three retries per LLM API call.These settings are intended to standardize execution across runs.
  • Scoring: The GradingEngine supports 15 check types, with deterministic audit checks and an LLM judge whose total weight is capped at 55%.The judge uses the agent output, audit summary, and task rubric as inputs.
  • Scoring: The LLM judge scores outputs on six allowed levels from 0.0 to 1.0 and returns 0.5 when its API call fails.Audit context prevents rewarding claims about actions that were not actually performed.
  • Error injection: Robustness equals recovered/total_errors, counting recovery when the same action succeeds within the next five audit-log entries.If no errors are injected, robustness defaults to 1.0.
  • Repeated evaluation: Pass3 requires success in all three independent runs, and aggregation reports mean, minimum, and per-dimension scores.The default threshold is 0.5, reducing the influence of lucky outcomes under random error injection.
  • Task coverage: The benchmark includes single-service, cross-service, and file-dependent tasks testing API use, multi-hop coordination, shell interaction, and report generation.The representative examples cover todo auditing, calendar-contact-Gmail coordination, and SQLite WAL recovery.
  • Task coverage: File-dependent tasks use mounted files and native shell or file tools rather than mock service APIs.Their safety scoring can prohibit credential leakage in the final output.

G.1 Evaluation Infrastructure

The infrastructure runs each task in an isolated Docker environment with local mock services, audit logging, injected failures, standardized model access, and explicit timeout handling. Two benchmark variants preserve the same scenario distribution while differing in scale.

  • Sandbox and services: Each task runs in a harness-specific Docker container bundling the agent runtime, ClawEnvKit infrastructure, and mock services.This isolates execution while supporting different harness environments.
  • Sandbox and services: Mock services run on localhost:9100 behind a multi-service router, with health checks and audit logs recording every API call.Audit records include endpoint, request body, response body, and timestamp.
  • Error injection: 25% of POST requests receive injected failures or delays, distributed across HTTP 429, HTTP 500, and delayed HTTP 200 responses.The injection scheme tests rate-limit handling, error recovery, and latency tolerance.
  • Model execution: Models are accessed through OpenRouter with deterministic temperature-0 function calling, 4,096-token call limits, and 20 tool-calling rounds per task.Some text-formatted XML tool calls are parsed into standard tool-call objects before execution.
  • Failure handling: LLM calls retry with exponential backoff and jitter, while task timeouts kill the container and assign a score of 0.Per-call timeout is 120 seconds and per-task timeout is 300 seconds.
  • Task composition: The task mix contains approximately 370 single-service API, 400 cross-service API, and 270 file-dependent tasks.Cross-service tasks add multi-service audit checks and coordination rubrics.
  • Reproducibility: Judge nondeterminism, unseeded error injection, provider routing, and model-dependent API costs limit exact reproducibility across runs.Estimated cost for a 1,040-task run ranges from $20–50 with Haiku to $100–300 with Opus.

H.1 False Negative Analysis

The false-negative analysis examines high-effort, low-scoring trajectories and finds that their low scores correspond to actual execution failures rather than valid alternative solutions being penalized. Mock services are further justified as an API-evaluation proxy, while schema drift and authentication remain gaps relative to real services.

  • False-negative analysis: The analysis defines potential false negatives as trajectories with at least 10 tool calls and a final score below 0.4.It manually inspects 52 such cases to identify their root causes.
  • False-negative analysis: 0% of high-effort low-score cases are genuine false negatives; 82.7% involve incorrect API parameters, 9.9% failure to retry, and 7.4% other execution errors.The authors report that every low score corresponds to a real agent failure in this analysis.
  • Mock-service validity: Mock services preserve endpoint paths, parameter schemas, and response structures, so evaluation targets interface-level tool selection, parameter construction, recovery, and coordination.The paper argues that these skills are determined by the API contract rather than server-side implementation.
  • Scope boundary: The remaining proxy gaps are schema drift in changing real APIs and authentication complexity such as OAuth flows and API keys.The paper treats these as outside the core tool-use capabilities targeted by the benchmark.
  • Scale consistency: Auto-ClawEval and Auto-ClawEval-Mini differ by less than 2% in scores across models and harnesses, supporting scale consistency for the mock-service infrastructure.The authors interpret this scale invariance as evidence against systematic bias from increasing environment count.
Loading 2604.18543v4…