Source-linked AI summary
ClawGym II: Exploring Black-Box RL on Agent Harness
Huatong Song, Fei Bai, Ming Yang, Renyuan Li, Jia Deng, Jujie He, Zhange Zhang, Daixuan Cheng, Yan Xing, Qi Yun, Xuxing Chen, Danyang Li, Feng Chang, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, Mingjie Tang, Ji-Rong Wen
TL;DR
Reinforcement learning through complex agent harnesses remains difficult, particularly when harnesses use heterogeneous protocols and workflows. This paper presents a black-box RL framework that decouples optimization from opaque harness execution and reconstructs trajectories for PPO, GRPO, and mix-harness training. It improves ClawGym-Bench Pass@1 by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200–400 optimization steps.
Problem
Optimizing agents across heterogeneous harnesses is challenging because their interaction protocols, tool interfaces, context management, and execution workflows differ substantially.
Method
The framework decouples policy optimization from opaque harnesses, reconstructs forked trajectories with prefix trees, and adapts PPO, GRPO, and mix-harness training.
Results
Pass@1 on ClawGym-Bench rises by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, with stable training across 200–400 optimization steps.
Takeaways & Limitations
Black-box RL provides a practical, scalable approach for optimizing general autonomous agents through complex deployment harnesses.
Takeaways & Limitations
The framework excludes auxiliary compaction and subagent trajectories from policy optimization, leaving their effective use for future work.
Abstract
from arXiv · showhide
Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.
1. Introduction
Agent harnesses coordinate complex, long-horizon interactions between models and environments, but their opacity and heterogeneity make stable, scalable RL difficult. The paper addresses these challenges with a unified black-box RL framework and validates it across OpenClaw and Claude Code.
- Agent harnesses integrate prompts, tools, context management, workflow orchestration, and recovery mechanisms into a unified runtime for autonomous task solving.
- Challenges: Black-box RL must handle unreliable concurrent rollouts, fragmented and redundant traces, training–inference inconsistency, and heterogeneous harness protocols.
- Framework: The framework uses temporary sandboxes to isolate task environments and harnesses, supporting stable large-scale concurrent execution and faithful model behavior.
- Evaluation: 11.71 and 17.28 points are the Pass@1 improvements on PinchBench for models trained through OpenClaw and Claude Code, respectively.The evaluated models start from the Qwen3-30A3B backbone; the corresponding ClawGym-Bench improvements are 9.98 and 14.
- Contributions: The framework supports individual opaque harnesses and mix-harness training, enabling joint optimization across heterogeneous execution systems.
2. Preliminaries
The preliminaries formalize agent tasks as multi-turn interactions with isolated stateful environments, whose final workspace states receive rollout-level rewards. They then characterize harness-mediated execution and contrast critic-based PPO with critic-free GRPO for optimizing agent policies over interaction histories.
- Agent-task formulation: A general agent task pairs a user instruction with an isolated, stateful environment containing the workspace and task-specific execution resources.The agent interacts through external tools across multiple turns.
- Agent-task formulation: Agent execution alternates between observations and actions, terminates upon completion or stopping, and evaluates the final workspace state to produce a rollout-level reward.Rewards may be rule-based, using deterministic checks, or rubric-based, using an evaluator such as an LLM-as-a-Judge.
- Harness-mediated execution: Modern harnesses such as OpenClaw, Claude Code, and Codex mediate model–workspace interaction through tool orchestration, context management, skills, delegation, and failure recovery.The harness executes model actions, updates workspace and context, and returns processed observations, making trajectories jointly determined by model and harness.
- RL preliminaries: PPO is presented as critic-based reinforcement learning using a policy model and auxiliary value model, whereas GRPO is critic-free and estimates advantages from groups of sampled rollouts.Both define policy optimization over states represented by interaction histories.
- RL preliminaries: PPO and GRPO use token-level masks to exclude invalid or environment-generated positions, while GRPO additionally constrains policy drift with a reference policy and KL divergence.The mask also prevents invalid positions from contributing to PPO temporal credit assignment.
3. Unified Black-Box RL through Complex Agent Harnesses
The framework directly optimizes a trainable model through an unmodified, opaque agent harness while decoupling policy optimization from harness execution. It supports scalable rollouts, reconstructs training trajectories from captured model calls, and enables training across heterogeneous harnesses.
- The formulation optimizes a trainable model through an unmodified, opaque agent harness while decoupling policy optimization from harness execution.
- The section develops scalable black-box rollouts, recovery and optimization of trajectories from captured model calls, and training across heterogeneous harnesses.
3.1. Infrastructure for Scalable Black-Box RL Execution
The framework enables scalable black-box RL by isolating stateful task environments and harnesses in temporary sandboxes while separating policy optimization from harness execution. A serving proxy captures model-boundary interactions and rewards for reconstructing trainable trajectories without instrumenting opaque harness internals.
- Sandboxed execution: Temporary sandboxes isolate each task’s stateful environment and selected harness for large-scale concurrent rollouts.Rollouts may modify files, processes, tool states, and other task-specific resources across many interaction steps.
- Execution–optimization decoupling: Separating training and inference engines from harness control flow enables different black-box harnesses to integrate with minimal adaptation.The harness retains control over tool use, context management, retries, and environment interaction.
- Boundary-level capture: A serving proxy at the model boundary exposes model behavior without requiring access to the harness’s internal control flow.The proxy serves as the harness’s model endpoint and intercepts every model-generated action.
- Boundary-level capture: The proxy records exact input tokens, generated tokens, rollout log-probabilities, and task metadata for policy optimization.It invokes the current rollout policy and returns responses in the protocol expected by the harness.
- Rollout-to-training pipeline: After rollout completion, a verifier converts the final workspace state into a reward, which the training pipeline combines with captured calls to reconstruct multi-turn trajectories.The reconstructed trajectories are then used by the training engine to update the policy model.
3.2. Bridging Black-Box Harness Execution and Policy Optimization
The framework reconstructs black-box model calls as prefix-tree trajectories, filters invalid or auxiliary branches, and optimizes the retained structure with adapted GRPO or PPO. It preserves rollout-level reward semantics and training–inference token consistency despite harness-side transformations.
- Trajectory Reconstruction: Prefix trees reconstruct shared interaction histories, preserve distinct continuations, and recover harness-introduced tool outputs and environment feedback between model calls.Each root-to-leaf path forms a candidate multi-turn trajectory, while shared prefixes are stored once.
- Trajectory Filtering: Trajectory filtering removes retry-induced dead leaves, over-branching rollouts, and auxiliary subagent or compaction interactions that create noisy or ambiguous training signals.Only main-agent trajectories aligned with the rollout-level task reward are optimized.
- Tree-Structured Optimization: Recovered trajectories retain their multi-trajectory tree structure, with all valid paths from one rollout sharing its terminal reward while shared-prefix tokens contribute to the loss only once.This preserves rollout-level reward semantics while allowing multiple reconstructed trajectories to contribute to optimization.
- Tree-Structured Optimization: GRPO assigns each rollout-level advantage across its retained trainable token nodes, whereas PPO treats same-rollout trajectories independently with γ=1 and λ=1 and performs separate GAE backups.PPO does not propagate advantage signals across branch points and may increase advantage variance.
- Training–Inference Consistency: The black-box token-in-token-out discipline trains on inference-generated tokens directly while decoding a separate harness-facing representation that is never re-encoded.This prevents tool-call normalization and assistant-message re-serialization from breaking training–inference sequence identity.
3.3. Mix-Harness Training
Mix-harness training jointly optimizes one shared policy across heterogeneous harnesses through a common serving and trajectory pipeline. It mixes task–harness instances while preserving comparable rollout groups for advantage estimation.
- Motivation and design: A common model-serving boundary and recovered trajectory representation integrates different harnesses without changing the underlying policy optimization procedure.Harnesses may differ in interaction protocols, tool interfaces, context management, and control flows.
- Motivation and design: Mix-harness training jointly optimizes a shared policy using rollouts from multiple heterogeneous harnesses in one training run.This reduces specialization to any single harness’s tool conventions, context strategy, or execution workflow.
- Batch construction: Task–harness instances pair each task environment with compatible harnesses, then randomly mix these instances within each training batch.All instances are jointly processed through the shared training pipeline.
- Rollout grouping: Optimization groups are defined by task–harness pairs, with advantages normalized separately even when different harnesses process the same task in one batch.This keeps rollouts comparable and prevents harness-dependent interaction patterns and reward distributions from distorting relative advantages.
- Representative harnesses: The framework instantiates individual-harness and mix-harness training with OpenClaw and Claude Code.OpenClaw supports diverse workspace-grounded tasks, whereas Claude Code targets long-horizon coding and terminal interaction.
3.4. Safeguards for Reliable Training
The framework adds engineering safeguards to separate infrastructure failures from policy behavior and improve reliable rollout collection and training. These safeguards address sandbox faults, streaming-parser errors, and incomplete asynchronous trajectory records.
- Robust Sandbox Execution and Fault Handling: Infrastructure-level failures such as timeouts, connection losses, and transient platform errors are handled with fail-safe mechanisms to prevent stalled collection or invalid training trajectories.These failures arise from remotely provisioned sandbox execution rather than the policy model.
- Buffered Pseudo-Streaming Parsing: Pseudo-streaming parsing buffers output tokens while generating to prevent rare incremental-parser errors from prematurely terminating otherwise correct tool calls.The method preserves synthetic Server-Sent Events during generation, though the supplied passage is truncated before describing the full procedure.
- Complete Trajectory Capture via Settling: Trajectory assembly waits until record counts remain unchanged across several checks and no records are pending, with a fixed timeout preventing indefinite waiting.This settling procedure addresses asynchronous retries and background fault handling that can delay trajectory writes.
4. Experiments · 4.1. Overall Performance · 4.2. Training Dynamics Across Harnesses
The experiments show that black-box RL substantially improves agent performance across heterogeneous harnesses, initialization strategies, and model scales. Training remains stable for approximately 200–400 steps under both PPO and GRPO, with upward reward and evaluation trends.
- 4.1. Overall Performance: The evaluation uses Pass@1 on ClawGym-Bench and PinchBench under a hybrid code-verification and rubric-judgment protocol.ClawGym-Bench combines code checks and rubrics with weights 0.7 and 0.3 when both apply.
- 4.1. Overall Performance: The study trains Qwen3-8B and Qwen3-30A3B with OpenClaw and Claude Code, forming the ClawII-OC and ClawII-CC model families.ClawII-OC starts from a cold-started policy, whereas ClawII-CC trains directly from the corresponding base models.
- 4.1. Overall Performance: 9.98 and 14.81 points are the ClawGym-Bench gains of 30A3B ClawII-OC and ClawII-CC over their respective initial policies under OpenClaw and Claude Code.ClawII-OC-30A3B also exceeds the ClawGym-30A3B SFT baseline by 5.80 points under OpenClaw, with gains transferring to PinchBench.
- 4.1. Overall Performance: 7.92 and 9.98 points are the OpenClaw gains for 8B and 30A3B, while 23.51 and 14.81 points are the Claude Code gains.The resulting 30A3B models outperform Qwen3-235A23B by 8.14 and 6.28 points in the two corresponding settings.
- 4.1. Overall Performance: Black-box RL improves consistently with and without cold-start initialization across model scales and task categories.A lightweight cold start strengthens OpenClaw initialization, but a specialized warm-up stage is not required for effective training.
- 4.2. Training Dynamics Across Harnesses: The training-dynamics study compares critic-based PPO and critic-free GRPO with Qwen3-30A3B under OpenClaw and Claude Code.Both algorithms optimize over recovered multi-trajectory tree structures, using matched rollout budgets per update despite different batch configurations.
- 4.2. Training Dynamics Across Harnesses: Approximately 200–400 steps are stable for both PPO and GRPO, with upward training-reward and downstream-evaluation trends and broadly comparable final results.PPO has smoother entropy dynamics, while GRPO varies more, including a late-stage entropy decline under OpenClaw; Claude Code produces higher entropy than OpenClaw.
4.3. Mix-Harness Training · 4.4. Training on More Challenging Tasks
The framework supports joint optimization across heterogeneous harnesses, with mixed training matching individual-harness performance. It also transfers directly to structurally diverse JobBench and OfficeQA tasks, producing substantial evaluation gains with stable reward trends.
- 4.3. Mix-Harness Training: The study tests whether one model can be jointly optimized through multiple heterogeneous harnesses within a unified black-box RL pipeline.This experiment uses Qwen3-30A3B without cold-start initialization.
- 4.3. Mix-Harness Training: GRPO training randomly mixes OpenClaw and Claude Code task–harness pairs within each training batch.The same task environment is executed through both harnesses, with GRPO grouping and relative-advantage normalization applied across the mixed data.
- 4.3. Mix-Harness Training: Mixed-harness training achieves rewards comparable to individual-harness models and matches or slightly exceeds their downstream evaluation under both harnesses.Under OpenClaw, the mixed model narrows its initial gap; under Claude Code, its reward remains on par with or higher than the Claude-Code-only model.
- 4.4. Training on More Challenging Tasks: The framework is evaluated on more challenging, structurally diverse task settings from JobBench and OfficeQA.JobBench involves heterogeneous workspaces and multiple file formats, while OfficeQA emphasizes answer-centric reasoning.
- 4.4. Training on More Challenging Tasks: Black-box RL trains Qwen3-30A3B with Claude Code on synthesized JobBench-style and OfficeQA-style tasks.The respective task formats guide the construction of the training tasks.
- 4.4. Training on More Challenging Tasks: A new task requires only an instruction, initialized workspace, and verifier, while evaluation improves from 20.46 to 27.20 on JobBench-Easy and from 8.53 to 21.54 on OfficeQA-Full.Training rewards maintain clear and stable upward trends throughout optimization.
4.5. Effect of Cold-Start Initialization · 4.6. Comparison with White-Box AgentLoop RL
Cold-start supervised initialization improves the training dynamics of OpenClaw black-box RL relative to direct base-model initialization. White-box AgentLoop RL achieves stronger same-harness performance and transfers to OpenClaw, indicating gains across harness settings.
- 4.5. Effect of Cold-Start Initialization: Cold-start initialization uses lightweight supervised training on ClawGym-SynData trajectories, whereas direct initialization starts from the base model.The comparison uses Qwen3-30A3B with OpenClaw and GRPO.
- 4.5. Effect of Cold-Start Initialization: Cold-started models begin with higher training reward and follow smoother optimization trajectories than models trained directly from the base model.Both initialization strategies benefit from reinforcement learning.
- 4.5. Effect of Cold-Start Initialization: Cold-started models maintain relatively stable policy entropy, while direct base-model training shows larger fluctuations and a pronounced late-stage entropy decline.The observed decline indicates more volatile entropy dynamics and potentially less stable optimization.
- 4.6. Comparison with White-Box AgentLoop RL: White-box AgentLoop RL exposes the system prompt, tools, observations, context management, and workflow, unlike black-box RL, which observes only serving-boundary model calls.These components can be independently designed and assembled into different agent loops.
- 4.6. Comparison with White-Box AgentLoop RL: White-box rollouts directly record model generation, tool execution, observations, and context updates as complete multi-turn trajectories for optimization.The instantiated loop includes bash, search, fetch, code interpreter, and file-operation tools in sandbox-based environments.
- 4.6. Comparison with White-Box AgentLoop RL: 59.90 average score: WhiteBox-30A3B improves 18.21 points over Qwen3-30A3B and outperforms black-box-trained ClawII-OC-30A3B by 8.53 points under the white-box harness.The improvement is consistent across all six task categories, and GRPO and PPO provide stable optimization dynamics.
- 4.6. Comparison with White-Box AgentLoop RL: 50.33 average score: WhiteBox-30A3B exceeds Qwen3-30A3B by 5.22 points under OpenClaw without additional training.Performance improves in five of the six categories, suggesting transfer of underlying agentic capabilities across harnesses.
5. Conclusion
The paper establishes black-box reinforcement learning as a practical, scalable approach for training general autonomous agents through complex deployment harnesses. It decouples policy optimization from opaque harness execution and identifies broader interaction types, tasks, and execution settings for future study.
- Conclusion: Black-box reinforcement learning trains general autonomous agents directly through complex deployment harnesses.The native harness remains unmodified and opaque during rollout.
- Conclusion: The framework decouples policy optimization from harness execution while isolating rollouts in temporary sandboxes and capturing model behavior at the serving boundary.It also reconstructs forked multi-turn trajectories, as described in the conclusion passage.
- Conclusion: Future work will incorporate auxiliary interactions, including compaction and subagent trajectories, into policy optimization rather than excluding them from training.The authors also plan to evaluate broader general-agent tasks and more diverse execution settings.
A. Evaluation Prompt
The evaluation prompt requires strict rubric-based grading of an OpenClaw agent using only supplied task evidence, with concise criterion-level analysis followed by an exact two-key JSON judgment.
- Output format: The final response must end with exactly one standalone JSON object containing only `scores` and `notes`.The `scores` map every rubric ID to an allowed numeric anchor, while `notes` concisely summarize the main reasons.
- Evaluation constraints: The evaluator must grade an OpenClaw agent result without calling tools, browsing, inspecting files, or requesting additional context.Grading is limited to the task, final output files, optional transcript evidence, and rubrics included in the prompt.
- Evaluation procedure: The evaluator must explain concisely how the final outputs satisfy or fail each rubric criterion before issuing the final judgment.
- Output format: The evaluator must not include or compute an overall aggregated score because aggregation is handled separately.