PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenClaw-RL: Train Any Agent Simply by Talking

Yinjie Wang, Xuyang Chen, Xiaolong Jin, Mengdi Wang, Ling Yang

arXiv:2603.10165v2cs.CLcs.AIcs.CVcs.LG

TL;DR

OpenClaw-RL addresses the lack of live online learning from next-state signals in agent interactions. It combines a server-client architecture, asynchronous signal extraction, and a hybrid stabilized RL objective, achieving efficient personal-agent optimization and unified general-agent training across diverse settings.

  • Problem

    Existing agentic RL systems do not recover user replies, tool outputs, or environment changes as live online learning sources, while infrastructure and methodology for real-time improvement remain little explored.

  • Method

    OpenClaw-RL streams deployed-agent interactions to an RL server, extracts evaluative and directive signals asynchronously, and combines them with overlap-guided hint selection and log-probability-difference clipping.

  • Results

    OpenClaw-RL aligns personal agents within around 10.3 sessions, outperforms memory and skill-evolution methods, and improves optimization over RLVR across terminal, GUI, SWE, and tool-call settings.

  • Takeaways & Limitations

    Deployed interactions can provide online supervision across personal and general agents by combining frequent evaluative signals with richer but sparser directive signals.

Abstract

from arXiv · show

Every agent interaction generates a next-state signal, namely the user reply, tool output, terminal or GUI state change that follows each action, yet no existing agentic RL system recovers it as a live, online learning source. We present OpenClaw-RL, a framework that employs next-state signals to optimize personal agents online through infrastructure and methodology innovations. On the infrastructure side, we extend existing RL systems to a server-client architecture where the RL server hosts the policy behind an inference API and user terminals stream interaction data back over HTTP. From each observed next state, the system extracts two complementary training signals, evaluative and directive, via a separate asynchronous server so that neither signal extraction nor optimization blocks inference. On the methodology side, we introduce a hybrid RL objective that unifies both signal types in a single update: directive signals provide richer, token-level supervision but are sparser, while evaluative signals are more broadly available. To stabilize distillation under teacher-student mismatch, we propose overlap-guided hint selection, which picks the hint whose induced teacher distribution maximally overlaps with the student's top-$k$ tokens, together with a log-probability-difference clip that bounds per-token advantages. Applied to personal agents, OpenClaw-RL enables an agent to improve simply by being used, recovering conversational signals from user re-queries, corrections, and explicit feedback. Applied to general agents, OpenClaw-RL is the first RL framework to unify real-world agent settings spanning terminal, GUI, SWE, and tool-call environments, where we additionally demonstrate the utility of next-state signals in long-horizon settings.

1. Introduction

OpenClaw-RL addresses the underexplored problem of learning from deployed agent interactions in real time. It combines asynchronous infrastructure with a hybrid objective that uses complementary next-state signals for efficient, stable online optimization.

  • Real-time use data, including user replies, tool results, and GUI transitions, remains little explored as a source for improving agents online.
  • OpenClaw-RL extends RL infrastructure with a server-client architecture that streams terminal interactions to an RL server for online training.The server hosts the model behind an inference API, while user terminals send interaction data over HTTP.
  • Next states yield evaluative signals from outcomes and directive signals from corrective hints, with evaluative signals more frequent and directive signals more informative.The hybrid objective combines both signal types rather than relying only on scalar rewards or sparse hints.
  • Overlap-guided hint selection and log-probability-difference clipping stabilize distillation under teacher-student mismatch.Hint selection favors teacher distributions overlapping the student’s top-k tokens, while clipping bounds token-level advantages.
  • The framework unifies terminal, GUI, SWE, and tool-call agents under one infrastructure and shows that next-state signals are valuable in long-horizon environments.Its deployment scope includes cloud-hosted parallel environments and diverse real-world agent settings.
  • OpenClaw-RL improves personal-agent optimization and general-agent optimization, outperforming memory and skill-evolution methods for personalization and RLVR methods in general settings.The reported personalization result reaches alignment within around 10.3 sessions, while the general-agent evaluation reports better optimization with stable training.

2. OpenClaw-RL Infrastructure: Unified System for Personal and General Agents

OpenClaw-RL uses a unified, decoupled server-client system to train personal and general agents from interaction streams. The infrastructure supports diverse agent frameworks, asynchronous signal processing, and scalable cloud-hosted environments.

  • A single framework unifies automatic optimization for personal OpenClaw agents and large-scale terminal, GUI, SWE, and tool-call agents.
  • The inference-API architecture lets user frameworks on devices, terminals, or cloud instances query a stateless policy server and stream interaction data over HTTP.Any agent capable of issuing API requests can serve as a data source.
  • Requests are classified into trainable main-line turns and non-training side turns, with session identifiers supporting concurrent multi-user streams.Main-line turns include primary responses and tool results; side turns include auxiliary queries and environment transitions.
  • Policy serving, environment hosting, reward judging, and policy training operate as independent asynchronous components without blocking live inference.The PRM judges prior responses while the trainer updates the policy, and serving receives weight updates at defined boundaries.
  • The system spans sparse, session-based personal interactions and dense, parallelized general-agent streams from cloud-hosted environments.This scalability supports diverse deployment settings, including terminal, GUI, and other real-world agent scenarios.

3. Methodology: Learning from Next-State Signals

OpenClaw-RL converts heterogeneous next-state signals into policy gradients through a hybrid objective combining dense evaluative rewards with sparse, token-level directive supervision. It stabilizes hint-conditioned distillation through overlap-guided selection and clipped log-probability differences, while integrating step-wise and outcome rewards for long-horizon agents.

  • Next-state signals: OpenClaw-RL converts next-state signals from conversations, terminals, GUIs, SWE tasks, and tool-call traces into policy gradients.The framework unifies these interaction streams within one methodology.
  • Two complementary signals: Evaluative signals provide scalar rewards on every scored turn, whereas directive signals provide richer token-level teacher guidance only when the next state contains an extractable correction.Examples include explicit or implicit success reactions for evaluative signals and error traces for directive signals.
  • Hybrid objective: The hybrid objective combines the evaluative advantage and on-policy distillation loss in a single per-token update, using their complementary frequency and information density.Both terms share the same trajectory and policy updates, with default weights w_RL = w_OPD = 1.
  • Overlap-guided distillation: Overlap-guided hint selection chooses candidate hints whose induced teacher distributions share the most top-k tokens with the student, using sequence-level or token-level selection.Sequence-level selection has similar performance and tends to be more stable in general agentic RL settings.
  • Distillation stability: A clipped log-probability difference bounds per-token distillation advantages, while PPO-style clipping bounds the surrogate update and preserves hint directionality.The method restricts distillation to a vocabulary subset and clips the teacher-student log-probability gap at C.
  • Long-horizon settings: For long-horizon agentic RL, OpenClaw-RL integrates process rewards with outcome rewards and standardizes actions by shared step index.Step-wise rewards provide dense credit assignment beyond terminal-only supervision, while direct action grouping avoids requiring clustered states.

4. Experiments

Experiments evaluate OpenClaw-RL for personal-agent personalization and general-agent reinforcement learning. The results show efficient preference alignment, broad agent-setting coverage, benefits from hybrid and process-reward training, and stabilization from overlap-guided selection and clipping.

  • Personal Agent Optimization: OpenClaw-RL reaches the desired personalization effect in about 10 joint-optimization sessions and 15 separate-optimization sessions.The metric is the minimum number of sessions required to achieve the defined optimization effect.
  • General Agent Setup: OpenClaw-RL supports terminal, GUI, SWE, and tool-call agents with cloud-hosted parallel environments.The experiments use 128 parallel terminal environments, 64 GUI and SWE environments, and 32 tool-call environments.
  • General Agent Setup: 0.25 and 0.33 are achieved with integrated rewards in tool-call and GUI settings, versus 0.19 and 0.31 with outcome-only optimization.These results come from long-horizon training with 250 tool-call steps and 120 GUI steps.
  • Personal Agent Optimization: Hybrid RL is more efficient than GRPO, OPD, memory, and skill-evolution baselines while updating only model weights.The cited comparison also notes that memory and skill-evolution methods impose additional inference-time context overhead.
  • Hybrid RL: Hybrid RL generalizes to agentic RL and RLVR, outperforming outcome-only and integrated-reward baselines in large-batch training.The experiments include multi-turn agentic RL and RLVR settings.
  • Hint Selection: Random hint selection destabilizes training, while sequence-optimal and token-optimal top-k-overlap selection improve optimization efficiency.Sequence-optimal selection tends to be more stable than token-optimal selection in large-batch training.
  • Stability: Clipping token-level log-probability differences suppresses extreme shifts and prevents uncontrolled response-length growth.Without sufficient control, final truncation ratios reach 0.2 for clipping and 0.5 for non-clipping methods in the Retool setting.
  • Robustness: Larger models do not guarantee faster personalization, and using Qwen3-8B as a PRM teacher performs similarly to using the target model as teacher.These findings are reported for Qwen3-32B joint optimization and Qwen3-4B-Thinking-2057 guidance.

5. Related Work

Related work spans general RL, agent-specific interaction paradigms, process-reward modeling, on-policy distillation, and scalable RL infrastructure. OpenClaw-RL positions itself as online next-state supervision combined with overlap-guided distillation and asynchronous training across heterogeneous agents.

  • Agentic RL: Earlier agent paradigms support multi-step tool interaction but rely on demonstrations rather than online reinforcement learning.The cited paradigms include ReAct, Toolformer, and FireAct.
  • Agentic RL: Prior RL work targets specific settings, whereas OpenClaw-RL unifies terminal, GUI, SWE, and tool-call agents.The cited preceding systems include SWE-agent, ReTool, DigiRL, WebRL, ArCHer, and LOOP.
  • Process Reward Models: OpenClaw-RL extends process-reward modeling to online supervision inferred from live next-state signals across heterogeneous long-horizon agents.This differs from process-reward approaches based on pre-collected ground truth.
  • On-Policy Distillation: On-policy distillation supplies token-level guidance that scalar-reward RLVR cannot provide, while OpenClaw-RL selects hints by teacher-student top-k overlap.The method combines overlap-guided selection with log-probability-difference clipping.
  • RL Infrastructure: OpenClaw-RL builds on decoupled RL infrastructure with asynchronous serving, rollout, PRM judging, and training loops for continuous live-interaction learning.The cited infrastructure includes OpenRLHF, AReal, veRL, ROLL, and slime.

6. Conclusion

OpenClaw-RL treats every agent interaction as a stream-agnostic next-state training signal and combines frequent evaluative feedback with richer directive feedback. The framework demonstrates online supervision across personal and general agents, while identifying feedback poisoning and privacy as deployment challenges.

  • Unified Online Learning: User replies, tool outputs, terminal changes, GUI interactions, SWE tasks, and tool-call traces can feed one shared training loop.The conclusion describes next-state signals as stream-agnostic.
  • Training Signals: Frequent evaluative signals and sparser directive signals provide complementary online supervision for deployed agents.Stable learning is supported by overlap-guided hint selection and log-probability-difference clipping.
  • Deployment Challenges: Negative or adversarial feedback may poison online updates, motivating stronger training-data filtering.The conclusion gives misleading corrections and malicious instructions as examples.
  • Deployment Challenges: Personalized models may encode user preferences and private information, making privacy protection and safe online learning necessary deployment priorities.The conclusion identifies such models as attractive targets for attacks.

A. Experiment Details

The appendix specifies the real-time OpenClaw training configuration, infrastructure components, evaluation setup, and cross-setting hyperparameter documentation. Training uses asynchronous updates after small data batches and separates policy, inference, PRM, and signal-serving roles.

  • Training Configurations: OpenClaw training uses w_RL = w_OPD = 1, clipping constant C = 1, k = 4, three hints per sample, and learning rate 1 × 10^-5.Asynchronous training begins after collecting 16 training samples.
  • Training Configurations: The real-time infrastructure includes policy actor, policy server, PRM actor, and PRM server components.The PRM server obtains evaluative and directive signals, while the policy server handles inference.
  • Evaluation Configurations: Evaluation deploys the RL server for inference and training before running OpenClaw on personal computers.The student evaluation uses a default limit of 72 conversation sessions.
  • General Agentic RL Hyperparameters: Table 6 provides the complete hyperparameter configuration across different agentic RL settings.The supplied appendix entry identifies this table as covering the general agentic RL settings.

A.3. Hybrid RL Extension Experiment Configurations

The hybrid RL extension combines evaluative and directive training signals in a token-level objective, with directive supervision built from hint-conditioned teacher distributions.

  • The token-level OPD variant is evaluated as a special case of directive-signal training.
  • Directive training appends a next-state-derived hint to the prompt before evaluating the teacher distribution.
  • The teacher uses the hint-augmented prompt, whereas the rollout student is evaluated under the original prompt.
  • The directive signal compares teacher and rollout-student log probabilities on sampled tokens.
  • The token-level OPD advantage is added to the scalar evaluative advantage in the hybrid objective.

A.5. Token-Level Log-Probability Shift Analysis

The token-level analysis examines how hints alter likelihoods, while the online pipeline asynchronously converts next states into evaluative rewards and directive hints for training.

  • Token-level analysis: The experiment uses Qwen2.5-7B-Instruct as the base generator and Qwen2.5-14B-Instruct as the hint-conditioned scorer on 32 MATH-500 examples.
  • Token-level analysis: The analysis measures per-token log-probability shifts between hint-conditioned and base models on the same generated response.The visualization separates all tokens, short-response tokens, and long-response tokens; orange lines mark medians, red diamonds means, and hollow circles outliers.
  • Online pipeline: The online pipeline streams each executed action’s next state to the RL server for asynchronous signal extraction.
  • Online pipeline: Samples are categorized as hybrid, directive-only, or evaluative-only according to whether hints and evaluative rewards are available.
  • Online pipeline: The pipeline freezes the rollout policy, computes advantages, applies clipped optimization, updates the policy, and synchronizes weights with the serving engine.
  • Signal extraction: The prompt templates classify next states and assign positive, negative, or neutral evaluations based on progression, correction, or ambiguity.

B.1. More Optimization Examples

The optimization examples compare pre- and post-optimization responses across probability, arithmetic, and explanatory-answer tasks.

  • Probability example: 100% more likely is the reported relative difference between rolling above 3 once and rolling two even numbers consecutively.
  • Arithmetic examples: 10 lawn mowings is the reported answer after accounting for the $70 total cost, $30 available, and $40 remaining.
  • Arithmetic examples: The after-optimization response explicitly walks through the costs, savings, earnings, remaining amount, and division by earnings per mow.
  • Explanatory examples: The after-optimization response praises the correct intermediate calculations and identifies the relevant problem structure.
  • Explanatory examples: 189 hours is the reported answer for the painter-time example, whose optimized explanation preserves the days-to-hours conversion and per-painter interpretation.

B.2. Response Length and Truncation Ratio Example

The response-length example reports that clipping prevents training from being dominated by self-distillation’s aggressive update, while non-clipping produces increasingly long outputs.

  • Clipping can prevent training from being dominated by self-distillation’s aggressive update.
  • 0.2 and 0.5 are the final truncation ratios for clipping and non-clipping, respectively.Without clipping, response length increases steadily because overly long outputs are not penalized by the outcome reward.

B.3. PRM Ablation Results

The PRM ablation compares process-reward-model configurations and reports similar performance with the larger Qwen3-8B model. The section also clarifies that OPD uses token-local supervision under fixed prefixes rather than counterfactual continuations.

  • Performance is similar when using the larger Qwen3-8B PRM model.
  • OPD objective: OPD computes a sum of token-local terms rather than a full sequence-level KL over entire continuations.
  • OPD objective: Each OPD term updates the student using the teacher’s next-token distribution under the observed fixed prefix.
  • OPD objective: Because downstream effects from alternative token choices are omitted, hint selection should measure teacher-student overlap at the token level.
Loading 2603.10165v2…