PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenForgeRL: Train Harness-native Agents in Any Environment

Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao

arXiv:2607.21557v3cs.AIcs.CL

TL;DR

OpenForgeRL addresses the difficulty of training harness-based agents end-to-end when open RL stacks cannot express stateful, multi-process inference or remote rollout environments. Its proxy and Kubernetes orchestration framework connects real harnesses and environments to standard RL codebases, yielding performance above similarly sized open baselines across tool-use and GUI benchmarks, including 37.7 on OSWorld-Verified.

  • Problem

    Open training stacks cannot natively express stateful, multi-process harness inference or scalable containerized rollouts, forcing simplified training harnesses and creating a train–deploy mismatch.

  • Method

    OpenForgeRL decouples harness inference from training with a proxy that reconstructs trajectories for standard RL codebases and a Kubernetes orchestrator for remote rollout containers.

  • Results

    OpenForgeRL models outperform similarly sized open baselines on nearly all tool-use and GUI benchmarks, reaching 37.7 on OSWorld-Verified.

  • Takeaways & Limitations

    RL improves agentic reliability through self-verification, broader tool coverage, and multi-step plan completion, while error recovery remains weak.

  • Takeaways & Limitations

    The framework discards trajectories ending in network issues, harness crashes, or timeouts because better credit assignment for partial rollouts remains unresolved.

Abstract

from arXiv · show

Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems. While powerful, these complex harnesses also make agents hard to train end-to-end with open infrastructure, whose SFT/RL stacks cannot natively express stateful, multi-process harness inference. To address this, we present OpenForgeRL, an open-source framework for training harness-based agents end-to-end in diverse environments. OpenForgeRL achieves this with a lightweight proxy that serves the harness's model calls while recording them as training data for a standard RL codebase (e.g., veRL), and a Kubernetes orchestrator that runs each rollout in its own remote container, together enabling training on any harness in any environment at scale. By decoupling training and inference, OpenForgeRL allows researchers to easily train, study, and improve agents directly in the real harnesses and environments they are deployed with. We validate our framework across diverse, complex harnesses and environments, spanning tool/claw-based agents and multimodal GUI browser- and computer-use agents. Using only hundreds to a few thousand tasks, OpenForgeClaw reaches 31.7 pass^3 and 55.9 pass@3 on ClawEval and 33.7 on QwenClawBench. OpenForgeGUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager. Both outperform open baselines of similar size on nearly all benchmarks, and in the GUI setting match or surpass models several times larger. Beyond benchmarks, we analyze how harness choice (e.g., ZeroClaw, OpenClaw, Codex) and RL shape agent behavior. We find that some harnesses are substantially harder to learn than others, and that RL improves agentic reliability, such as self-verification, tool coverage, and completing multi-step plans, though critical abilities such as error recovery remain weak.

1 INTRODUCTION

OPENFORGERL enables end-to-end training of harness-based agents by decoupling stateful harness inference from standard RL training and orchestrating rollouts in remote environments. Across tool-use and GUI settings, it improves agent performance while showing that harness choice and RL affect learnability and reliability, with error recovery remaining difficult.

  • Motivation: OPENFORGERL addresses the mismatch between stateful, multi-process harness inference and open training stacks that cannot natively express nested tools, subagents, and long-horizon context.Existing approaches often require reimplementing a simplified harness for training.
  • Framework: A lightweight proxy decouples harness inference from training, records model calls, reconstructs trajectories, and connects arbitrary harnesses and environments to standard RL codebases such as veRL.The infrastructure is designed to pair real harness ecosystems with remote sandboxes in cloud-service providers.
  • Empirical validation: 31.7 pass^3 and 55.9 pass@3 on ClawEval, plus 33.7 on QwenClawBench, demonstrate OpenForge-Claw’s performance across ZeroClaw, OpenClaw, Codex, and ReACT settings.OpenForge-Claw is a 30B-A3B MoE trained across three popular harnesses and the standard ReACT loop.
  • Empirical validation: OPENFORGERL improves over open models of similar size on nearly all benchmarks and in several cases exceeds models several times larger across tool-use and GUI agents.The evaluation spans daily tool-use, claw-based agents, and multimodal browser- and computer-use agents.
  • Behavioral analysis: Simpler, better-aligned harnesses are easier to learn, training generalizes across similar harnesses, and RL improves self-verification, tool coverage, and task completion while error recovery remains challenging.These findings come from training in real deployment harnesses and analyzing how harness choice and RL shape agent behavior.

2 RELATED WORK

Prior work has developed increasingly capable inference harnesses for coding and everyday-task agents, while open-source RL frameworks provide advanced algorithms and distributed training for end-to-end agent learning. However, these RL frameworks assume relatively simple rollouts, creating a gap for training harness-based agents in complex environments.

  • Agents with inference harnesses: Inference harnesses improve agent task success through carefully designed agent-computer interfaces and increasingly refined tool-use recipes.SWE-Agent established this direction, followed by Claude Code and Codex for software engineering and OpenClaw for everyday tasks.
  • Agents with inference harnesses: Claude Code and Codex refine harness-based software-engineering agents, while OpenClaw extends the approach to everyday tasks.
  • Training harness-based agents: Open-source RL frameworks such as veRL and Slime support advanced algorithms including PPO and GRPO alongside asynchronous distributed training.These frameworks are designed for end-to-end training as agents operate in increasingly complex environments.
  • Training harness-based agents: Existing RL frameworks assume relatively simple rollouts, limiting their direct fit for training agents built around complex inference harnesses.

3 METHODS

OPENFORGERL connects standard RL frameworks to distributed, containerized harness rollouts through a proxy and Kubernetes orchestrator. It also provides mechanisms for scalable rollout management, failure handling, and task synthesis in data-scarce browser-, computer-, and claw-use domains.

  • Problem formulation: Harness-based agents follow multi-step environment interactions, receiving observations, generating actions, and transitioning through an MDP-like process.The framework models each step as an observation-conditioned action followed by an environment transition.
  • System architecture: OPENFORGERL addresses hidden harness control flows and resource-intensive rollouts with a proxy plus Kubernetes orchestrator connecting harnesses to RL frameworks such as veRL.The orchestrator creates rollout pods, while the proxy intercepts generation requests and routes them to the inference server.
  • Rollout orchestration: Kubernetes-based orchestration elastically creates, deletes, and allocates remote rollout containers without overloading training nodes.Rollout containers can run on cloud providers such as Microsoft Azure, supporting many concurrent environments.
  • Asynchronous rollout and timeouts: Wall-clock timeouts and error signaling prevent unresponsive remote rollouts from stalling batch collection and training.The proxy and trajectory-reconstruction module let training continue collecting from remaining rollouts after a terminated rollout.
  • Error handling: Failed trajectories caused by network issues, harness crashes, or timeouts are discarded to avoid misleading training signals from partial rollouts.The paper identifies better credit assignment for partial rollouts as a promising direction.
  • Task synthesis: A parallel task-synthesis pipeline proposes realistic instructions, removes duplicates, builds executable environments and verifiers, and tests tasks with separate open models.The pipeline targets data-scarce domains such as claw/daily tool use and computer use.

4 EXPERIMENTS

OpenForgeRL trains agents end-to-end across diverse text-based harnesses and multimodal GUI environments, then evaluates them on six benchmarks. The trained OpenForge agents outperform similar-size baselines on nearly all reported benchmarks, including with far fewer GUI training tasks than MolmoWeb.

  • 4 EXPERIMENTS: OpenForgeRL trains and evaluates agents across text-based tool-use and multimodal GUI domains spanning browser-use and computer-use tasks.The experiments cover six popular benchmarks and environments requiring visual perception, mouse and keyboard control, or diverse tool use.
  • Text-based harnesses: Text-based experiments use everyday tasks such as reading email, searching knowledge bases, and updating helpdesk tickets inside ZeroClaw, OpenClaw, and Codex.The evaluation includes ClawEval, QwenClawBench, and MCPAtlas, using their specified official protocols and harnesses.
  • Text-based harnesses: OpenForge-Claw outperforms similar-size models and the untrained Qwen3-30B-A3B-Thinking backbone across ClawEval, QwenClawBench, and MCPAtlas.The authors attribute the gains to curated tasks and executable environments that provide learning signal for everyday harness-mediated tasks.
  • Multimodal GUI environments: The GUI experiments cover OSWorld-Verified, Online-Mind2Web, and WebVoyager, using generated tasks and containerized browser-use environments.The resulting data pool contains 2,500 SFT tasks and 900 RL tasks, with GPT-4.1 evaluating task success.
  • Multimodal GUI environments: OpenForge-GUI outperforms similar-size computer-use and browser-use models on nearly all benchmarks, despite using only 2.5k tasks versus MolmoWeb’s over 200k.It surpasses MolmoWeb on Online-Mind2Web and remains competitive on WebVoyager.

5 DISCUSSION

The discussion finds that harness design strongly affects learnability and that training transfers across harnesses, with multi-harness training performing best. RL further improves tool selection and several reliability behaviors, although error recovery remains weak.

  • 5.1 Harness difficulty: Custom-tool harnesses ReACT and ZeroClaw reach the highest performance, while OpenClaw gains only moderately from SFT+RL despite much longer prompts and contexts.The comparison evaluates ReACT*, ZeroClaw, OpenClaw, and Codex, ordered by increasing sophistication.
  • 5.2 Transfer across harnesses: +3.3 and +4.6: ZeroClaw-only training improves the untrained base on unseen OpenClaw and Codex, respectively.Training on one harness therefore generalizes to other harnesses.
  • 5.2 Transfer across harnesses: +9.5 on OpenClaw and +20.3 on Codex: training on all three harnesses produces the largest gains over the base on the more complex harnesses.It also raises ZeroClaw to 48.5 versus 46.0 for ZeroClaw-only training.
  • 5.3 Capabilities added by RL: 22.6% to 13.9%: RL reduces generic shell calls and redirects tool use toward dedicated service tools under ZeroClaw.RL also slightly shortens trajectories, suggesting improved selection of specialized tools over a general-purpose shell.
  • 5.3 Capabilities added by RL: RL strengthens error recovery, self-verification, and coverage of required tools under Codex, but error recovery remains weak.Self-verification includes reading back the model’s own writes to confirm them.

6 CONCLUSION · A TRAINING DETAILS

OpenForgeRL trains LLM- and VLM-based agents end-to-end inside their deployment harnesses and environments using standard RL codebases. The framework also enables analysis of how harness choice and RL affect agent behavior, including reliability and transfer to unseen harnesses.

  • 6 CONCLUSION: OpenForgeRL trains LLM- and VLM-based agents directly inside the inference harnesses where they are deployed.This supports end-to-end optimization in real deployment settings rather than simplified reimplementations.
  • 6 CONCLUSION: The framework makes any harness × any environment trainable with standard RL codebases such as veRL.Training is performed in the agents’ real deployment settings.
  • 6 CONCLUSION: OpenForgeRL trains OpenForge-Claw and OpenForge-GUI models using only hundreds to a few thousand automatically curated tasks.The passage states that these models surpass open models, but the supplied text truncates the comparison.
  • 6 CONCLUSION: 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager are reported for OpenForge-GUI.These are the benchmark results explicitly supplied in the conclusion passage.
  • 6 CONCLUSION: OpenForgeRL enables analysis of how harness choice and RL shape agent behavior.The passage contrasts this capability with prior open work, which could not easily conduct such a study.
  • 6 CONCLUSION: Some harnesses are substantially harder to learn than others, while training gains transfer to harnesses unseen during training.Both findings are stated as conclusions from the framework’s behavioral analysis.
  • 6 CONCLUSION: RL primarily improves agentic reliability, including self-verification and broader action coverage.The supplied passage states that the model verifies its own actions and covers more, but truncates the full list of behaviors.

A.1 CLAW AGENT TRAINING DETAILS

OpenForge-Claw RL training uses veRL with Azure-hosted, containerized rollouts orchestrated through Kubernetes. Each task-specific rollout runs in an isolated resource-capped pod, while policy training uses a single multi-GPU node.

  • veRL serves as the training backend, with Microsoft Azure providing the cloud infrastructure for rollout containers.
  • Each rollout runs in its own container with the target harness pre-installed and is scheduled onto a Kubernetes pod capped at 2 CPUs and 2GB RAM.Supported harnesses include OpenClaw, Codex, and ZeroClaw.
  • Policy training runs on a single node of 8×B200 GPUs, while rollout pods are packed onto Azure D128ads v5 nodes.

A.2 GUI AGENT TRAINING DETAILS · B DATA DETAILS

OpenForge-GUI training uses veRL with Azure-hosted, Kubernetes-managed rollout environments tailored separately to computer-use and browser-use agents. Computer-use rollouts run harness-equipped containers, while browser-use rollouts operate through dedicated remote Chromium sessions.

  • A.2 GUI AGENT TRAINING DETAILS: OpenForge-GUI RL training uses veRL as its backend and Microsoft Azure for rollout containers.
  • A.2 GUI AGENT TRAINING DETAILS: Training curves cover computer-use training success rate, validation success rate, and training episode length, while Table A1 reports key RL hyperparameters.
  • A.2 GUI AGENT TRAINING DETAILS: Computer-use rollouts run in separate containers built from task-specific Dockerfiles with modified Kimi-Agent pre-installed.
  • A.2 GUI AGENT TRAINING DETAILS: Computer-use containers are scheduled on Kubernetes pods capped at 4 CPUs and 4GB RAM, packed onto Azure D64ads v5 nodes.
  • A.2 GUI AGENT TRAINING DETAILS: Policy training for computer-use agents runs on a single node equipped with 8×B200 GPUs.
  • A.2 GUI AGENT TRAINING DETAILS: Browser-use rollouts drive dedicated remote hosted-Chromium sessions through the Chrome DevTools Protocol.
  • A.2 GUI AGENT TRAINING DETAILS: Each browser-use environment manages session handling, action execution, screenshot capture, and reward judging inside a lightweight Kubernetes sandbox pod.
  • A.2 GUI AGENT TRAINING DETAILS: Browser-use sandbox pods are capped at 2 CPUs and 6 GB RAM and packed onto Azure D128ads v7 nodes.

B.1 MORE DETAILS ON DATA SYNTHESIS · B.2 MORE DETAILS ON CLAW DATA

OpenForgeRL synthesizes training tasks with parallel Claude Agent SDK agents and relies primarily on this pipeline because suitable large-scale public Claw-agent datasets were unavailable. Claw RL tasks require robust verifier refinement, making them substantially more costly than SFT tasks, while multiple harnesses are included for experimentation.

  • B.1 MORE DETAILS ON DATA SYNTHESIS: The synthesis pipeline uses Claude Agent SDK with Opus 4.6 as the backbone for five agent modules operating in parallel.It takes a target domain prompt and target task count N as inputs.
  • B.1 MORE DETAILS ON DATA SYNTHESIS: Agents propose candidate instructions by browsing domain-specific assets and reference pools rather than inventing tasks from scratch.For Claw agents, the pool includes ClawHub skills and ZClawBench tasks; computer-use pools include X search API access, 22k AgentNet instructions, and synthetic Synt files and data.
  • B.2 MORE DETAILS ON CLAW DATA: No suitable large-scale public dataset was available for Claw-agent training, so the authors primarily synthesize tasks for SFT and RL.The passage identifies this dependence as especially relevant to RL training.
  • B.2 MORE DETAILS ON CLAW DATA: RL tasks retain a full test-and-refine loop so their verifiers remain robust and reliable across repeated rollouts.SFT success signals are used for one-time filtering, whereas RL verifiers must consistently judge task success.
  • B.2 MORE DETAILS ON CLAW DATA: 16.1 minutes and 4.36 USD are the average costs to synthesize an RL task with verifier refinement.These figures contrast with the lower cost of synthesizing an SFT task without a verifier.
  • B.2 MORE DETAILS ON CLAW DATA: 5.2 minutes and 0.86 USD are the average costs to synthesize an SFT task without a verifier.A full task-distribution breakdown appears in Figure A6.
  • B.2 MORE DETAILS ON CLAW DATA: ZeroClaw, OpenClaw, and Codex are included alongside the default ReACT loop to test OpenForgeRL across popular harnesses.The default ReACT loop is implemented by most tool-use agents.

B.3 MORE DETAILS ON GUI DATA · C EVALUATION DETAILS

The GUI data pipeline uses lightweight Xvfb-based environments and customized harnesses for computer-use and browser-use tasks. It curates SFT trajectories with filtering and repetition control, then trains RL on 900 distinct tasks with a 20-turn limit.

  • B.3 MORE DETAILS ON GUI DATA: Xvfb renders virtual displays in memory, enabling lightweight parallel GUI containers for RL training.This avoids the heavier full Ubuntu virtual-machine setup commonly run under QEMU.
  • B.3 MORE DETAILS ON GUI DATA: Kimi-Agent serves as the primary computer-use harness, with added bash and other tools following Anthropic’s computer-use approach.The full action space and tool list are provided in Section C.3.
  • B.3 MORE DETAILS ON GUI DATA: Browser-use adapts MolmoWeb with JSON-formatted actions and Stealth Browsers, reducing IP and CAPTCHA blocks from 40% to nearly zero.The JSON action format avoids extra action-alignment training stages, while Stealth Browsers address CAPTCHA and website blocking.
  • B.3 MORE DETAILS ON GUI DATA: SFT curation removes evaluation-overlapping and decomposed subtasks, then retains popular websites from SimilarWeb Top100 and MOZ Top 500.The filtering reduces distractions from unusual long-tail websites.
  • B.3 MORE DETAILS ON GUI DATA: Kimi-K2.5 generates up to 30-turn trajectories across 4 repeats, with successful candidates reduced to the shortest trajectory per task.Repeated-action handling retains only the last turn after more than three consecutive identical actions and removes trajectories exceeding five identical actions.
  • B.3 MORE DETAILS ON GUI DATA: RL samples 600 Insta-v3 tasks and 300 PAE-WebVoyager tasks distinct from SFT data, while restricting trajectories to 20 turns.The same popular-website filter is applied during RL to balance performance and training cost.

C.1 CLAWEVAL AND QWENCLAWBENCH EVALUATION DETAILS … D ANALYSIS DETAILS

The evaluation spans claw-agent, MCP-tool, computer-use, and browser-use benchmarks, using official or explicitly standardized protocols and harness configurations. Additional analyses vary harness choice and document task subsets, tools, observations, and scoring procedures for reproducibility.

  • C.1 CLAWEVAL AND QWENCLAWBENCH EVALUATION DETAILS: OpenForge-Claw is evaluated on ClawEval, QwenClawBench, and held-out MCPAtlas using each benchmark’s official evaluation protocol.MCPAtlas serves as a held-out test set for novel tool use.
  • C.1 CLAWEVAL AND QWENCLAWBENCH EVALUATION DETAILS: ClawEval harness analysis compares ZeroClaw, OpenClaw, and Codex, exposing benchmark-specific tools through native registration or SKILL.md files.ZeroClaw natively supports registering new tools, whereas OpenClaw and Codex require adaptation through SKILL.md files.
  • C.2 MCP-ATLAS EVALUATION DETAILS: 89 of MCPAtlas’s 500 public tasks have fully supported expected tool calls under the default 20-server configuration, and every model uses this same set.The default configuration excludes optional servers requiring third-party credentials or service-specific data initialization.
  • C.2 MCP-ATLAS EVALUATION DETAILS: MCPAtlas evaluation uses its official ReACT-like harness with default prompts and tool configurations, without benchmark-specific demonstrations or fine-tuning.The harness executes policy-model tool calls in the MCPAtlas sandbox and returns observations until termination.
  • C.2 MCP-ATLAS EVALUATION DETAILS: MCPAtlas reports pass@1 as the fraction of tasks whose Gemini 2.5 Pro claim-coverage score reaches at least 0.75 under a fixed evaluation setup.All models share the same task subset, harness, judge, and threshold.
  • C.3 COMPUTER-USE EVALUATION DETAILS: OSWorld-Verified evaluates multimodal agents on open-ended Ubuntu desktop tasks using screen, mouse, and keyboard interaction scored by execution-based verifiers.OSWorld-Verified is described as an in-place upgrade with enhanced infrastructure and improved task quality.
  • C.3 COMPUTER-USE EVALUATION DETAILS: OSWorld-Verified uses Kimi-Agent as the main training and evaluation harness, augmented with bash and a string-replacement editor for direct file and data manipulation.The modification is inspired by Anthropic’s computer-use approach.
  • C.4 BROWSER-USE EVALUATION DETAILS: Browser-use evaluation covers 300 Online-Mind2Web tasks and WebVoyager’s 595-task Fara-595 subset using a modified MolmoWeb harness with screenshot-only observations.Each step provides a 1280 × 720 viewport screenshot without DOM or accessibility-tree information.

D.1 BEHAVIORAL ANALYSIS DETAILS

The section compares SFT and SFT+RL OpenForge-Claw checkpoints under ZeroClaw and Codex, analyzing tool reliance and long-horizon behavioral capabilities. It defines robustness and recovery metrics and reports a shift away from generic shell usage on successful ZeroClaw runs.

  • Comparison setup: SFT and SFT+RL OpenForge-Claw checkpoints are evaluated on ClawEval under both ZeroClaw and Codex harnesses.The analysis compares behavioral differences between the two checkpoints across the same benchmark setting.
  • ZeroClaw tool usage: Tool-usage percentages measure each tool’s share of all calls across rollouts, independently of whether tasks are solved.This measures tool reliance rather than task success.
  • Codex behavioral capabilities: Five Codex behavioral capabilities are reported as percentages from 0 to 100, with higher values indicating better performance across rollouts.The capabilities are computed for the two checkpoints on the same task set and summarized in Figure 5.
  • Codex behavioral capabilities: Format robustness measures rollouts avoiding malformed tool-call termination, while error recovery measures failed-command rollouts that still solve the task.Format robustness is defined as 100 minus the malformed-payload failure fraction; recovery is conditioned on at least one failed command.
  • ZeroClaw tool usage: 13.2% → 5.6%: successful ZeroClaw runs show a stronger shift away from the generic shell tool than the aggregate distribution.The comparison covers full tool-usage distributions for SFT versus SFT+RL on ClawEval.
Loading 2607.21557v3…