PaperlayerView exact revision on arXiv

Source-linked AI summary

ClawLoop: Self-Improving Agents via Harness-Native Training Loop

Hansi Yang, Linji Hao, Hongyan Hao, An Wang, Han Wu, Qi Gu, Xunliang Cai

arXiv:2610.14118v1cs.AIcs.CL

TL;DR

Harness-agent training pipelines are scarce, often one-shot and dependent on stronger external teachers. ClawLoop replaces that pattern with a teacher-free cycle of task synthesis, training, and diagnosis; it improves across three benchmarks and its tasks also benefit smaller models.

  • Problem

    Harness-agent post-training data is difficult to scale, while existing pipelines are scarce, often one-shot, and depend on external stronger models for training signal.

    Source: §1 · p. 1

  • Method

    ClawLoop iteratively synthesizes and validates tasks, trains the agent, and uses rollout diagnosis to target tasks for the next cycle without an external teacher.

    Source: §1 · p. 2, Domain-grounded synthesis · p. 4, §4.3 · p. 6

  • Results

    ClawLoop consistently improves on SkillsBench, Claw-Eval-Live, and ClawGym-Bench, and Qwen3.5-4B and 9B models trained on its tasks achieve comparable or superior performance to models trained on ClawGym data.

    Source: §5.1 · pp. 6–7

  • Takeaways & Limitations

    The reported gains across model scales and benchmarks support the quality of ClawLoop's iteratively generated tasks beyond the loop model itself.

    Source: §1 · p. 2, §5.1 · p. 7

Abstract

from arXiv · show

The rapid advancement of Large Language Models (LLMs) has driven their deployment toward general-purpose harness environments, where LLM-based agents operate in sandbox workspaces and interact directly with file systems and shells. While numerous post-training pipelines exist for tool-use agents, their counterparts for harness-based agents remain scarce and are predominantly one-shot and teacher-dependent: a stronger external model synthesizes tasks and collects trajectories, the target agent is trained once, and evaluation results are consumed by human researchers rather than fed back into the loop. Such pipelines are inherently bounded by the capability of the external model and the scope of human prior, leaving no room for self-improvement. To address these limitations, we propose ClawLoop, a framework that enables self-improving without any external teacher model and with minimal prior dependencies. ClawLoop composes three phases into a self-sustaining cycle: Task Synthesis, which generates a diverse pool of training tasks via two complementary construction routines, applies diagnosis-conditioned difficulty growth followed by both agent-free and agent-based task validation; Training, which updates the agent via SFT and/or RL; and Diagnosis and Feedback, which analyzes the agent's rollout behaviors to produce a capability analysis report that will be used as feedback for task synthesis in the next iteration. Using Qwen3.6-27B as the loop model, ClawLoop yields consistent improvements on SkillsBench, Claw-Eval-Live, and ClawGym-Bench. The iteratively refined tasks are further transferred to smaller-scale Qwen3.5-4B and 9B models with gains across all benchmarks, validating data quality independently of the loop model.

1 Introduction

Harness agents can tackle more complex tasks than tool-use agents, but training data for their open-ended environments remains difficult to obtain. ClawLoop addresses one-shot, teacher-dependent training with an iterative, teacher-free loop that synthesizes tasks, trains agents, and feeds self-diagnosis into the next cycle.

1 Introduction

  • Harness agents can combine file operations, shell commands, skill execution, and multi-step reasoning, exceeding the scope of pipelines built around fixed tool domains.Their audited, extensible permissions let them operate directly on real file systems and shells.Source: §1 · p. 1
  • Existing harness training pipelines are scarce, depend on stronger external models, and run once, with evaluation feedback going to researchers rather than back into training.The paper identifies the external model's capability and human prior as bounds on these pipelines.Source: §1 · p. 1
  • ClawLoop closes the cycle by having the agent synthesize tasks, collect trajectories, diagnose rollout failures, and generate targeted tasks for the next training round without an external teacher.The harness already supports workspace creation, trajectory execution, and outcome verification; the missing mechanism is feedback from identified weaknesses.Source: §1 · pp. 1–2
  • Its task-synthesis mechanism combines seed-task generation, diagnosis-conditioned difficulty growth, and self-reflection-based validation.The paper says these tasks support code-level verifiability while maintaining complexity for deep reasoning.Source: §1 · p. 2
  • Using Qwen3.6-27B in the loop, ClawLoop improves across diverse benchmarks, while its generated tasks also yield gains for Qwen3.5-4B and 9B models across all benchmarks.The smaller-model results are presented as evidence of task quality independent of the loop model.Source: §1 · p. 2

2 Related Work

Related work spans task and environment synthesis, self-improving language models, and benchmarks for harness-based agents. These lines provide executable settings and evaluation tools, but commonly focus on repository coding, predefined tool ecosystems, or environment scaling.

Task Synthesis and Environment Generation

  • Task synthesis work covers repository coding, predefined-tool environments, and capability-oriented tasks, while ClawGym pairs synthesized tasks with mock workspaces and hybrid verifiers (Bai et al. 2026).The paper characterizes these approaches as generally focused on repository-level coding, predefined tool ecosystems, or environment scaling.Source: Task Synthesis and Environment… · p. 2

Self-Improving Language Models

Evaluation Benchmarks for LLM-Based Agents with Harness

3 Preliminary

The paper treats a harness as a sandbox execution environment for an LLM agent and defines each task in terms of its prompt, initial workspace, actions, transitions, and verifier. ClawLoop's three phases operate on this shared harness substrate.

3 Preliminary

  • A harness is a sandbox environment that can provide a Docker container, shell, file system, package manager, network access, and APIs for workspace provisioning and trajectory recording.The same harness is used for training rollouts and evaluation to ensure distribution match.Source: §3 · p. 2
  • A task is the tuple τ = ⟨p, s0, A, F, Vτ⟩, covering an instruction, initial workspace state, action space, deterministic transition function, and completion verifier.The workspace state can include files, environment variables, and installed tools; actions include shell commands, file edits, and tool invocations.Source: §3 · pp. 2–3
  • ClawLoop's overview places task synthesis and selection, training, and diagnosis and feedback on the same harness substrate in a closed, teacher-free loop.The task instruction is exposed to the agent as its initial user prompt.Source: §3 · p. 2, Figure 1 · p. 3

4 ClawLoop Framework

ClawLoop repeats task synthesis, training, and diagnosis in a loop, using evaluation failures to guide later task synthesis. Its task pool combines complementary construction routines with difficulty growth and validation; the agent trains on its own successful trajectories and eligible rollouts.

4 ClawLoop Framework

  • Each iteration synthesizes training and held-out tasks, updates the agent with SFT and/or RL, then uses held-out rollouts to diagnose failures and inform the next iteration.This closes the feedback loop between evaluation and task synthesis.Source: §4 · p. 3

4.1 Phase 1: Task Synthesis

  • Two complementary routines create seed tasks: capability-driven synthesis varies domains around a workflow, while domain-grounded synthesis derives tasks from skills in executable professional domains.The capability route separates workflow patterns from surface domains; the domain route organizes tools and data into skills, then builds tasks from abstract action graphs grounded in the domain.Source: §4.1 · p. 4, Capability-driven synthesis · p. 4, Domain-grounded synthesis · p. 4
  • Runnable seeded generators compute answer keys from synthesized data, while answer contracts specify hidden decision rules and output requirements without revealing answers.Generators must create cross-referenced sources; failed generation errors feed into repair.Source: Capability-driven synthesis · p. 4
  • Difficulty growth adds multi-hop retrieval and controlled workspace noise, while validation checks task integrity and retains tasks the current agent can sometimes—but not always—solve.Agent-free checks include oracle verification and leakage audits; agent-based validation filters by trial pass counts.Source: Growing Difficulty from Seed Tasks · p. 5, Execution-grounded validation · p. 5

4.2 Phase 2: Train

  • SFT trains on successful validated trajectories and masks environment observations; GRPO uses grouped rollouts from validated tasks to refine behavior beyond imitation of existing successes.SFT computes loss on the agent’s action tokens, while GRPO compares each rollout’s advantage with the group mean.Source: §4.2 · p. 5

4.3 Phase 3: Diagnosis and Feedback

  • Diagnosis summarizes each failed held-out trajectory’s divergence, root cause, and missing capability, then aggregates recurring failure patterns by skill category to guide targeted task synthesis.These summaries serve as diagnostic evidence rather than reward signals.Source: §4.3 · p. 6

5 Experiments

Experiments evaluate ClawLoop on three harness-agent benchmarks and test how synthesis choices and iteration count affect outcomes. The results also assess whether its generated tasks transfer to smaller models.

5 Experiments

  • Qwen3.6-27B ran three ClawLoop iterations, generating 1,024 training tasks per iteration for evaluation on SkillsBench, Claw-Eval-Live, and ClawGym-Bench.The benchmarks cover professional-domain skills, real-life work scenarios, and tasks constructed using the ClawGym pipeline.Source: §5 · p. 6

5.1 Main Results

  • ClawLoop surpasses ClawGym across all three benchmarks, while its tasks train Qwen3.5-4B and 9B models comparably or better than ClawGym-generated data.The smaller-model comparison uses tasks produced without ClawGym’s stronger external teacher models.Source: §5.1 · pp. 6–7

5.2 Ablation Study

  • Removing either synthesis routine degrades overall performance: capability-driven synthesis matters most for Claw-Eval-Live, while domain-grounded synthesis matters most for SkillsBench and ClawGym-Bench.The authors relate this asymmetry to broad agent capabilities versus benchmark emphasis on specific skill categories.Source: §5.2 · p. 7
  • Increasing task difficulty lengthens trajectories, but raising difficulty by at most one level per iteration best balances challenge with success rates for learning.Figure 2 tracks average rollout turns for constructed tasks and average Claw-Eval-Live scores as difficulty growth varies.Source: §5.2 · p. 7, Figure 2 · p. 7
  • SkillsBench continues improving with more iterations, whereas Claw-Eval-Live and ClawGym-Bench plateau around K = 3; the experiments use K = 3 to balance cost and performance.The authors attribute SkillsBench’s continued gains to its broader domain coverage and larger space for curriculum exploration.Source: §5.2 · p. 7, Figure 3 · p. 7

6 Conclusion

ClawLoop combines task synthesis, training, and self-diagnosis in a teacher-free loop. Its Qwen3.6-27B run improves benchmark performance, and its tasks also benefit smaller models.

6 Conclusion

  • ClawLoop combines complementary task synthesis, training, and self-diagnosis to let agents identify and address weaknesses without human intervention during training.Its tasks pass through diagnosis-conditioned difficulty growth and agent-free and agent-based validation before training.Source: §6 · p. 7
  • Qwen3.6-27B improves across diverse harness-agent benchmarks, and ClawLoop-generated tasks yield gains for Qwen3.5-4B and 9B models across benchmarks.The smaller-model results are presented as evidence of task quality beyond the loop model.Source: §6 · p. 7
  • Longer training horizons with adaptive stopping criteria remain future work.The appendix outlines harness and infrastructure, training configurations for all model scales, and the evaluation protocol.Source: §6 · p. 7, A Experimental Details · p. 10

A.1 Harness and Infrastructure

ClawLoop runs its phases in isolated, network-restricted containers with fixed execution budgets. Rollout collection, especially validation, dominates per-iteration compute cost.

  • Each rollout runs in an isolated Docker container with a shell, Python 3.11, standard command-line tools, and network isolation.Containers have 24-hour and 300-turn limits; exceeding either ends the trajectory as a failure.Source: §A.1 · p. 10
  • Each iteration comprises task synthesis, rollout collection, and gradient updates, with per-iteration costs reported for the Qwen3.6-27B loop model.The costs use a shared 32×H800 allocation, with GPU hours calculated as wall-clock hours scaled by 32.Source: §A.1 · p. 10, Table 4 · p. 10
  • Rollout collection dominates cost: validation rollouts account for roughly 41% of the per-iteration total and exceed combined SFT and GRPO training time.Diagnosis is comparatively cheap because it reruns only a small held-out task split disjoint from the training pool.Source: §A.1 · p. 10

A.2 Training Configuration

ClawLoop trains on successful self-generated trajectories using SFT followed by GRPO, and reuses each task’s rollouts across training and validation. Its SFT setup masks environment observations and preserves full trajectories.

  • ClawLoop applies SFT to successful self-generated trajectories, then GRPO for refinement beyond the SFT ceiling.Each task receives eight rollouts reused for validation, SFT data, and GRPO advantage estimation to keep compute tractable.Source: §A.2 · p. 10
  • The per-token SFT mask excludes environment observations, which can span thousands of tokens and otherwise dominate the loss.Table 5 separates loop-level settings from stage-specific SFT and GRPO optimization settings.Source: §A.2 · p. 10, Table 5 · p. 10

A.3 Evaluation Protocol

ClawLoop evaluates policies on three external benchmarks using a fixed harness configuration and runs an iterative cycle of task synthesis, training, and diagnosis. Its training stages use self-generated successes and group-based reinforcement learning, while the synthesized curriculum and the models evaluated at smaller scales have important scope boundaries.

A.3 Evaluation Protocol

  • The three external benchmarks are held out from training, task selection, and difficulty scheduling, and evaluation uses a fixed harness configuration.SkillsBench is reported as Avg@5 over five samples per task; Claw-Eval-Live and ClawGym-Bench use their official graders.Source: §A.3 · pp. 10–11

B Complete Algorithm

  • Each iteration synthesizes and validates tasks, trains on collected rollouts, then diagnoses failures on a held-out split to inform the next round.The algorithm pseudocode labels diagnosis as using the held-out split and aggregates recurring failure patterns.Source: B Complete Algorithm · p. 11

C SFT and RL within the Loop

  • The 27B model yields 892–1,978 successful trajectories per iteration, while GRPO supplies smaller models with learning signal from mixed-success rollout groups.SFT imitates successful trajectories; GRPO can use the contrast among eight rollouts even when only one succeeds.Source: C SFT and RL within the Loop · p. 11
  • The admission band retains tasks with uncertain outcomes, excluding tasks solved on every attempt or none because unanimous groups yield zero advantage.With N = 8 rollouts, the stated setting is Nl = 1 and Nh = 5; the band moves with the current policy.Source: C SFT and RL within the Loop · p. 11
  • The smaller models inherit a curriculum shaped by the 27B model and do not run their own diagnosis phase, so their gains are evidence about curriculum transfer rather than full-loop self-improvement.The results do not separate curriculum-producer scale from curriculum-consumer scale.Source: C SFT and RL within the Loop · p. 11

D Loop Dynamics

  • Tracking task-pool and trajectory changes across iterations distinguishes curriculum progression from mere pool growth.These quantities are described as the loop’s internal state.Source: D Loop Dynamics · p. 11

D.1 Per-Iteration Statistics

The loop’s admission and trajectory statistics vary as difficulty growth overshoots and is damped, while task synthesis yields both measurable filtering and concrete aggregation tasks with explicit requirements.

D.1 Per-Iteration Statistics

  • Agent-free validation admits about 89% of candidates consistently, while agent-based validation retains only tasks inside the [1, 5] admission band.The first stage checks generator determinism and verifier executability; the second selects tasks relative to the current policy.Source: §D.1 · p. 12
  • 65.6% admission at k=1 fell to 45.8% at k=2, then recovered to 55.4% at k=3 as task difficulty overshot and was damped.Mean turns rose from 5.6 to 11.4 before settling at 8.7; the table reports rates relative to 1,024 synthesized candidates per iteration.Source: Table 6 · p. 12, §D.1 · p. 12

D.2 Diminishing Returns

  • Only 45.8% to 65.6% of candidates land inside the admission band, so one-third to one-half of the fixed synthesis budget yields no training signal.Tasks solved on every attempt or none produce zero advantage because the rollout group is unanimous; the account identifies difficulty tracking as the limiting factor.Source: §D.2 · p. 12

E Case Study

  • The two aggregation-task instances differ in difficulty, with added difficulty appearing mainly in a longer verification phase rather than more task steps.The instances are illustrative, not a controlled comparison.Source: E Case Study · p. 12

E.1 Two Instances of an Aggregation Task

  • Both tasks require joining separate data sources, and workspace skill files specify otherwise underdetermined decisions needed for exact verification.The stated rules cover thresholds, rounding, invalid records, and empty tiers; without them, multiple defensible outputs could be penalized.Source: §E.1 · p. 12

E.2 The Two Trajectories

Both aggregation-task trajectories solved their task and entered the training pool, but the harder instance devoted more effort to verification and deliberation.

E.2 The Two Trajectories

  • Both agents solved their instance, and both trajectories were admitted to the training pool.Source: §E.2 · p. 13
  • Six of Instance A’s fifteen turns (40%) were verification, compared with one of Instance B’s six turns (17%).The trajectories have the same phase structure and diverge almost entirely in verification.Source: Table 7 · p. 13, §E.2 · p. 13
  • 2.5× turns, 1.7× tool calls, 4.5× shell executions, and nearly 17× reasoning tokens distinguish the harder instance from the easier one.The text characterizes the difference as more deliberation per action, much of it spent checking prior actions.Source: Table 8 · p. 13, §E.2 · p. 13

E.3 What the Verification Phase Contains

The harder trajectory’s verification included independent recomputation, cross-checking, and review of the checking process itself. The two examples are illustrative rather than a controlled comparison.

E.3 What the Verification Phase Contains

  • The agent recomputed the total as 6984.997, confirmed it rounded to 6985.00, and cross-checked category and tier totals.It also found its own checking script had incorrectly rejected equal medians despite a valid alphabetical tie-break.Source: §E.3 · p. 13
  • Tasks between instant success and complete failure elicit self-checking trajectories that enter the SFT set.The account describes the agent as auditing its own audit and correctly concluding that its solution, not the check, was sound.Source: §E.3 · p. 13
  • The two instances are not a controlled comparison, so their verification-effort difference cannot be attributed to difficulty alone with precision.The text points to aggregate trajectory-length and admission-rate trends as the quantitative counterpart.Source: §E.3 · p. 13

Related papers

Loading 2610.14118v1…