PaperlayerView exact revision on arXiv

Source-linked AI summary

ClawGym: A Scalable Framework for Building Effective Claw Agents

Fei Bai, Huatong Song, Shuang Sun, Daixuan Cheng, Yike Yang, Chuan Hao, Renyuan Li, Feng Chang, Yuan Wei, Ran Tao, Bryan Dai, Jian Yang, Wayne Xin Zhao, Ji-Rong Wen

arXiv:2604.26904v3cs.CLcs.AIcs.LG

TL;DR

Claw-style agents lack scalable, verifiable data and evaluation for multi-step workspace tasks. ClawGym unifies task synthesis, agent training, and benchmark evaluation, with trained agents showing substantial gains on two benchmarks.

  • Problem

    Claw-style agent development lacks large-scale, workspace-grounded task data that captures personalized workflows and supports reliable verification.

  • Method

    ClawGym combines persona-driven and skill-grounded task synthesis with mock workspaces, hybrid verification, rollout-based training, and benchmark construction.

  • Results

    54.68% improvement on PinchBench and 25.96% on ClawGym-Bench are reported for Qwen3-30B-A3B trained with ClawGym data.

  • Takeaways & Limitations

    ClawGym provides an integrated framework and resources for developing and diagnosing Claw-style personal agents across workspace-grounded tasks.

  • Takeaways & Limitations

    Evaluation primarily focuses on final-state correctness, leaving action safety, efficiency, and error recovery for future work.

Abstract

from arXiv · show

Claw-style environments support multi-step workflows over local files, tools, and persistent workspace states. However, scalable development around these environments remains constrained by the absence of a systematic framework, especially one for synthesizing verifiable training data and integrating it with agent training and diagnostic evaluation. To address this challenge, we present ClawGym, a scalable framework that supports the full lifecycle of Claw-style personal agent development. Concretely, we construct ClawGym-SynData, a diverse dataset of 13.5K filtered tasks synthesized from persona-driven intents and skill-grounded operations, paired with realistic mock workspaces and hybrid verification mechanisms. We then train a family of capable Claw-style models, termed ClawGym-Agents, through supervised fine-tuning on black-box rollout trajectories, and further explore reinforcement learning via a lightweight pipeline that parallelizes rollouts across per-task sandboxes. To support reliable evaluation, we further construct ClawGym-Bench, a benchmark of 200 instances calibrated through automated filtering and human-LLM review. Relevant resources have been released at https://github.com/ClawGym.

1. Introduction

Claw-style agents operate across tools, files, web services, and persistent workspaces, but remain unreliable on ambiguous, multi-step, long-horizon tasks. ClawGym addresses this gap by unifying verifiable task synthesis, agent training, and evaluation through SynData, ClawGym-Agents, and ClawGym-Bench.

  • Motivation: Claw-style agents must manage tools, local files, opaque interfaces, unexpected workspace states, execution errors, and cross-session dependencies.These requirements motivate workspace-grounded evaluation beyond static reasoning and structured agent-loop benchmarks.
  • Motivation: Existing studies lack large-scale Claw-style task data grounded in users’ workspaces, making personalized, representative, and verifiable task synthesis difficult.The challenge is driven by requirements spanning diverse professions, routines, and realistic workspace conditions.
  • Framework: ClawGym unifies task synthesis, agent training, and performance evaluation through a dual-route strategy combining persona-driven intents with skill-grounded operations.The framework is designed to generate tasks that are both diverse and verifiable.
  • Contributions: ClawGym-Agents use filtered black-box OpenClaw rollouts for supervised fine-tuning, while ClawGym-Bench retains discriminative tasks after automated calibration and LLM-assisted human review.Larger models show gains of 54.68% on PinchBench and 25.96% on ClawGym-Bench.
  • Contributions: 13.5K executable tasks comprise ClawGym-SynData, generated by an automated pipeline that combines persona-driven intents with skill-grounded operations.The dataset is presented as large-scale, diverse, verifiable, and suited to Claw-style training.

2. Preliminary

Claw-style agent tasks are environment-grounded instruction-execution problems in which agents use workspace tools to transform an initial state, with success judged primarily by the resulting final state. ClawGym organizes development around synthesized training data, trajectory-trained agents, and benchmark evaluation.

  • Task Definition: A Claw-style task provides a user instruction and initial workspace state, then allows the agent to use available computer tools to satisfy the request.Tasks may involve organizing files, extracting information, editing documents, generating reports, or configuring software; workspaces can include files, folders, web interfaces, configurations, and other artifacts.
  • Task Definition: The agent produces action and observation segments, supporting non-alternating execution in which multiple tool calls may precede corresponding feedback.Each action segment contains consecutive executable actions, while each observation segment contains environment observations.
  • Task Definition: The primary task output is the final environment state after execution, although the agent may also return a natural-language response.The final state can include created files, modified documents, reorganized directories, generated reports, or other persistent workspace changes.
  • Verification Criteria: ClawGym primarily evaluates final-state correctness because intermediate safety, efficiency, and robustness are difficult to verify consistently across different valid trajectories.A task receives full credit when the task-specific verifier assigns v = 1 to the final state and optional response.
  • Framework Overview: The framework includes ClawGym-SynData with 13.5K training tasks, ClawGym-Agents trained from black-box rollout trajectories, and ClawGym-Bench with 200 evaluation instances.These three components support synthesized data generation, agent training through the OpenClaw harness, and benchmark evaluation of Claw-style agents.

3. ClawGym-SynData: Scaling Effective Task Synthesis for Claw Agents · 3.1. Task Generation

ClawGym-SynData scales task synthesis through persona-driven top-down and skill-grounded bottom-up pipelines, then supports execution with prepared resources, hybrid verification, and automated quality filtering. Persona synthesis expands structured user contexts into realistic multi-step tasks, while skill synthesis composes filtered OpenClaw capabilities into operationally grounded workflows.

  • 3. ClawGym-SynData: Scaling Effective Task Synthesis for Claw Agents: The synthesis framework has four stages: task generation, resource preparation, verification design, and automated quality assessment.It creates workspace artifacts, code-based and rubric-based checks, and filtered training and benchmark data.
  • 3.1.1. Persona-Driven Top-Down Synthesis: Persona-driven top-down synthesis generates task seeds from user profiles, scenario categories, and atomic operations before expanding them into concrete multi-step instructions.The seed specifies who the user is, the scenario type, and the operations the agent may perform.
  • 3.1.1. Persona-Driven Top-Down Synthesis: Each persona-driven seed combines a user persona u∈U, a scenario category c∈C, and a basic-operation set G = {g_1, g_2, . . . , g_n}.Personas cover varied occupations, workflows, preferences, and daily needs, while the scenario taxonomy contains 9 major classes and 43 s…
  • 3.1.1. Persona-Driven Top-Down Synthesis: The persona and scenario define realistic user-facing intent, while selected operations constrain generated instructions to plausible Claw-style workspace actions.Different combinations of personas, categories, and operation sets produce diverse concrete user requests.
  • 3.1.2. Skill-Grounded Bottom-Up Synthesis: Skill-grounded bottom-up synthesis starts from reusable capabilities collected from ClawHub rather than user contexts and scenario intentions.Raw skills are commonly organized in directories containing SKILL.md, README.md, or other usage descriptions.
  • 3.1.2. Skill-Grounded Bottom-Up Synthesis: Approximately 30K raw OpenClaw skills are annotated and filtered to retain 16K synthesizable skills for downstream task synthesis.Annotations summarize capability content, constraints, input-output characteristics, and a binary synthesizability label.
  • 3.1.2. Skill-Grounded Bottom-Up Synthesis: Each skill-composition task uses one primary skill and up to three optional supporting skills to generate a user-facing workflow.The primary skill sets the central capability and objective, while supporting skills add auxiliary operations or contextual requirements.
  • 3.1.2. Skill-Grounded Bottom-Up Synthesis: GPT-5 grounds skill-composed instructions in original or annotated skill content, preserving operational details, usage constraints, and input-output requirements.Using annotated core content can abstract the skill into a more concise capability representation.

3.2. Resource Preparation · 3.3. Verification Design

ClawGym prepares self-contained task workspaces with lightweight, task-specific mock files generated from explicit resource requirements. It verifies completion through executable checks and qualitative rubrics, aggregating them with code-based verification weighted more heavily in hybrid tasks.

  • 3.2. Resource Preparation: ClawGym generates lightweight mock files tailored to each task instead of relying on real user files or large external datasets.Examples include documents, tables, configuration files, records, and reference materials.
  • 3.2. Resource Preparation: Resource specifications capture filenames, directory paths, formats, key fields, and content constraints before files are materialized.An LLM-based resource generator, GPT-5, places the resulting files at specified workspace paths.
  • 3.2. Resource Preparation: Generated resources encode task entities, constraints, schemas, fields, and values so checkers can recompute statistics and validate output consistency.This controlled construction makes each task self-contained, reproducible, and consistent with its requirements.
  • 3.3. Verification Design: ClawGym uses GPT-5 to generate executable checkers and rubric rules for workspace-grounded task outcomes.These signals address generated files, modified artifacts, data transformations, structured outputs, and final responses.
  • 3.3. Verification Design: Code-based verification decomposes objective requirements into deterministic checks covering files, JSON fields, computed statistics, required content, and resource-output consistency.The checker can cross-validate generated outputs against implicit requirements in the provided files.
  • 3.3. Verification Design: Rubric-based verification evaluates qualitative requirements such as professional tone, concise organization, and faithful summaries that executable code cannot reliably judge.Rubric rules define quality dimensions and assign ordinal scores combined through a normalized weighted average.
  • 3.3. Verification Design: λ = 0.7 weights code-based checks in hybrid verification, while the rubric component receives weight 0.3.Tasks with only code-based verification use the code score directly; hybrid tasks combine objective and subjective scores.

3.4. Automated Quality Assessment

Automated quality assessment filters low-quality synthesized tasks and unreliable verification artifacts before training or evaluation. It evaluates task novelty, plausibility, difficulty, and checker quality through embedding-based tests, LLM judgments, and executable or rubric-based checks.

  • Quality Assessment: The pipeline assesses both synthesized-task quality and verification-artifact reliability before samples enter training or evaluation.Task assessment examines whether requests are reasonable, self-contained, executable, and clearly specified, while verification assessment tests generated code checkers and rubric rules.
  • Task Quality: Task quality is measured through novelty, plausibility, and difficulty to reduce redundancy, remove impractical requests, and maintain a balanced task distribution.These dimensions target distinct coverage, practical executability, internal consistency, realistic environment assumptions, and expected completion complexity.
  • Task Quality: Novelty filtering compares a new task’s embedding with the retained pool and removes it when maximum cosine similarity exceeds the retention threshold.Tasks below the threshold are retained as sufficiently novel, reducing redundant patterns and improving coverage of distinct tasks.
  • Task Quality: GPT-5.4 judges plausibility and difficulty, checking realistic, internally consistent requests and estimating complexity from required steps, operation diversity, and planning demands.Difficulty scores support a mixture of simple, moderate, and challenging samples, while implausible tasks relying on unavailable components are removed.
  • Verification Quality: Verification assessment tests whether code-based and rubric-based checkers measure completion correctly without rewarding superficial outputs or rejecting valid solutions.Code checkers undergo executability and task-alignment assessment, while rubric judges assess complementary qualitative signals such as tone, clarity, organization, faithfulness, and completeness.

3.5. Synthesized Task Analysis

ClawGym analyzes synthesized task coverage, diversity, and quality using persona-driven and skill-grounded distributions alongside human evaluation of sampled training tasks.

  • Persona-Driven and Skill-Grounded Task Distribution: Persona-driven synthesis covers diverse user-facing scenario categories, with no single category dominating and the largest category representing 12.5% of tasks.Figure 2 reports both scenario-category and atomic-action distributions.
  • Human-Sampled Task Quality Analysis: Human evaluation randomly samples 50 synthesized training tasks and assesses task reasonableness, execution feasibility, resource consistency, and verification quality.Each dimension is scored on a 1–5 scale.

4. ClawGym-Agents: Training Claw Agents with Synthesized Tasks

ClawGym-Agents are trained from high-quality trajectories collected through black-box OpenClaw rollouts, reconstructed and reward-filtered for supervised fine-tuning. The framework also explores reinforcement learning with sandbox-parallel rollouts, improving both vanilla and supervised-starting agents.

  • Black-box rollout collection: OpenClaw is executed as a distributed black-box system, preserving its original control flow, tool interface, and agent-environment interaction semantics.Each synthesized task receives its specification and workspace materials in a designated Docker environment.
  • Black-box rollout collection: A proxy layer captures complete multi-turn rollout streams, including model inputs, outputs, tool invocations, and environment feedback, without modifying agent logic.The recorded requests and responses support reconstruction of coherent trajectories from fragmented logs.
  • Trajectory filtering: 24.5K high-fidelity interaction trajectories are retained for SFT by filtering valid, complete executions whose final verifier scores exceed a predefined reward threshold.The hybrid verification protocol produces continuous scores in [0, 1] by combining code-based and rubric-based verifiers.
  • Supervised fine-tuning: Multi-turn SFT on Qwen3-series models, with environment-feedback loss masking, produces ClawGym-4B, ClawGym-8B, and ClawGym-30B-A3B.For Qwen3-8B, YaRN extends the native 32K-token context window to 64K tokens.
  • Reinforcement learning: Sandbox-parallel RL improves evaluation performance from both Qwen3-4B-2507-Instruct and ClawGym-30B-A3B starting points.The pipeline virtualizes each task into an independent sandbox while keeping the OpenClaw agent loop black-box, enabling concurrent non-interfering rollouts.

5. ClawGym-Bench: A Reliable Benchmark for Claw Agents

ClawGym-Bench is constructed through stricter automated and human-LLM quality control to calibrate difficulty, ensure verifier reliability, and cover diverse workspace-grounded scenarios. The resulting 200-task benchmark uses executable verification and checks that each task is feasibly solvable.

  • Benchmark Construction: ClawGym-Bench applies stricter construction than training data, targeting difficulty calibration, verifier reliability, and diverse workspace-grounded scenarios.It is built from the synthesized task pool after task-quality and verifier-quality assessment.
  • Difficulty-Aware Filtering: n=4 rollouts with strong and smaller LLM agents estimate task difficulty and retain samples that are neither trivial nor unrealistically difficult.The filtering compares average completion scores to preserve tasks exposing a meaningful capability gap.
  • LLM-Assisted Human Review: Final candidates undergo human-LLM collaborative review because benchmark instances combine instructions, resources, executable checkers, and optional rubric rules.The review addresses subtle checker and task-verifier alignment issues that may require code-level reasoning.
  • Benchmark Composition: 200 benchmark tasks remain after difficulty-aware filtering and Human-LLM review, including 156 code-checked tasks and 44 hybrid-verification tasks.Each instance contains a user instruction, mock resources, and a corresponding verifier.
  • Verifiable Solvability: Each task is checked for verifiable solvability, using successful strong-agent rollouts or human-constructed reference completions to establish a feasible path to full score.This prevents flaws in instructions, resources, or verification code from making full success impossible.

6. Experiment

The experiments evaluate ClawGym’s data quality, training effectiveness, benchmark discrimination, and generalization using hybrid verification on ClawGym-Bench and PinchBench. Results show gains from synthesized data, complementary synthesis strategies, tuned training scale and reward filtering, while benchmark performance varies across capabilities and transfers beyond synthesized scenarios.

  • Effectiveness of Synthesized Data: ClawGym-4B, 8B, and 30A3B achieve average ClawGym-Bench scores of 47.73, 50.24, and 56.82, respectively, outperforming their Qwen baselines.ClawGym-30A3B also surpasses the much larger Qwen3-235B-A23B.
  • Discriminative Capacity of ClawGym-Bench: ClawGym-Bench distinguishes models by category and capability: Claude-4.7-Opus has the highest overall average, GPT-5.4 leads Product. & Collab., and Gemini-3-Flash leads Software Dev.No single agent dominates every dimension, indicating that categories probe distinct capabilities and expose model-specific strengths.
  • Generalization beyond Synthesized Scenarios: ClawGym-Agents trained exclusively on ClawGym-SynData perform robustly on external PinchBench, indicating transfer beyond the synthesized task distribution.The results suggest that the data supports acquisition of transferable agentic principles rather than only task-specific behavior.
  • Synergy of Synthesis Strategies: Mixed synthesis outperforms either persona-driven or skill-grounded synthesis alone, demonstrating the complementary value of both trajectory sources.The comparison uses models trained on trajectories from each individual strategy and their combination.
  • Impact of Reward Thresholding: A reward threshold of 0.5 yields the best results on both benchmarks, balancing execution completeness against data diversity.Thresholds from 0.4 to 0.9 are evaluated using continuous hybrid-verifier scores in [0, 1].

7. Behavioral Analysis of Claw Agents

Behavioral analysis on ClawGym-Bench identifies three requirements for effective Claw agents: coherent tool use, recovery during long-horizon workflows, and satisfaction of fine-grained constraints. Representative cases contrast stronger trajectories that maintain valid workflows with weaker ones that accumulate errors or produce invalid artifacts.

  • Tool-Use Appropriateness: Claw tasks require coherent tool use across discovery, inspection, computation, and verification rather than isolated tool invocation.In the CI artifact audit case, the stronger trajectory expands file patterns, inspects the JSON schema, and runs Python aggregation.
  • Long-Horizon Execution Robustness: The stronger long-horizon trajectory recovers from tool failures and completes an idempotent workflow, whereas the weaker trajectory accumulates unresolved errors and fails to reach stable completion.The representative support-ticket task requires processing two input batches and safely rerunning the automation.
  • Long-Horizon Execution Robustness: Long-horizon robustness depends on interpreting feedback, recovering from disruptions, and continuing valid workflows while preserving task context.These workflows coordinate file reading, scripts, artifact generation, intermediate checks, and workspace state across reruns.
  • Fine-Grained Instruction Following: Violating a small requirement can invalidate downstream outputs, as weaker reorder-plan trajectories propagate rows that fail the Quantity <= ReorderPoint rule.The resulting reorder artifacts may appear plausible while remaining incorrect.
  • Fine-Grained Instruction Following: Fine-grained instruction following requires satisfying filtering rules, output schemas, numeric formulas, and cross-file consistency, not merely creating requested artifacts.The reorder-plan case requires Quantity <= ReorderPoint filtering, exact TargetStock, OrderQty, and LineTotal computations, supplier aggregation, and per-supplier JSON outputs.

8. Conclusion

ClawGym streamlines end-to-end personal-agent development in Claw-style environments by integrating task synthesis, trajectory collection, agent training, and evaluation. Its synthesis combines topic-driven generation with skill composition, automated environment construction, and hybrid verification.

  • Framework overview: ClawGym integrates task synthesis, trajectory collection, agent training, and evaluation into one end-to-end development framework for personal agents.The framework operates within Claw-style environments.
  • Task synthesis: Its twofold synthesis approach combines top-to-bottom topic-driven generation with bottom-to-top skill composition.The approach joins complementary generation directions for constructing tasks.
  • Task synthesis: The synthesis stage is supported by automated environment construction and hybrid verification mechanisms.These components support the generation process within Claw-style environments.

A. Evaluation Prompt

The evaluation prompt instructs a strict rubric-based evaluator to grade an OpenClaw agent using only supplied task evidence, then return rubric scores and concise notes in a final JSON object.

  • Output Format: The response must end with exactly one standalone JSON object containing exactly `scores` and `notes`, with nothing after it.The `scores` object maps every rubric id to an allowed numeric anchor, while `notes` concisely summarizes the main reasons.
  • Evaluation Constraints: The evaluator must grade the OpenClaw agent without calling tools, browsing, inspecting files, or requesting additional context.Evaluation is restricted to the task, final output files, optional transcript evidence, and included rubrics.
  • Evaluation Procedure: Before the final judgement, the evaluator must concisely explain how the outputs satisfy or fail each rubric criterion.
  • Output Format: The evaluator must not include or compute an overall score because aggregation is handled separately by post-processing.
Loading 2604.26904v3…