PaperlayerView exact revision on arXiv

Source-linked AI summary

RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System

Yinjie Wang, Tianbao Xie, Ke Shen, Mengdi Wang, Ling Yang

arXiv:2602.02488v1cs.LGcs.AIcs.CLcs.CV

TL;DR

Long-horizon interaction makes binary outcome rewards insufficient, while scalable step-wise supervision remains costly to obtain. RLAnything addresses this by jointly optimizing policy, reward model, and environment in a closed loop, combining outcome and step-wise feedback and adapting task difficulty through critic feedback. Across GUI, text-game, and coding settings, each dynamic component improves the system, with substantial gains on OSWorld, Alf World, and LiveBench and optimized reward-model signals outperforming human-labeled outcomes.

  • Problem

    Binary outcome rewards provide insufficient supervision for long trajectories, while training generative reward models usually requires costly task-specific supervision.

  • Method

    RLAnything jointly optimizes policy, reward model, and environment through closed-loop feedback, integrating outcome and step-wise rewards and adapting task difficulty.

  • Results

    RLAnything consistently improves performance across GUI agents, text-based LLM agents, and coding LLMs, including 9.1% on OSWorld, 18.7% on Alf World, and 11.9% on LiveBench.

  • Takeaways & Limitations

    Optimized reward-model signals outperform human-labeled outcome signals, while dynamic environment adaptation supports learning from experience and potential environment scaling.

  • Takeaways & Limitations

    Reward-model training can become unbalanced when policy-induced tasks are overly difficult or overly easy, motivating task-difficulty adaptation.

Abstract

from arXiv · show

We propose RLAnything, a reinforcement learning framework that dynamically forges environment, policy, and reward models through closed-loop optimization, amplifying learning signals and strengthening the overall RL system for any LLM or agentic scenarios. Specifically, the policy is trained with integrated feedback from step-wise and outcome signals, while the reward model is jointly optimized via consistency feedback, which in turn further improves policy training. Moreover, our theory-motivated automatic environment adaptation improves training for both the reward and policy models by leveraging critic feedback from each, enabling learning from experience. Empirically, each added component consistently improves the overall system, and RLAnything yields substantial gains across various representative LLM and agentic tasks, boosting Qwen3-VL-8B-Thinking by 9.1% on OSWorld and Qwen2.5-7B-Instruct by 18.7% and 11.9% on AlfWorld and LiveBench, respectively. We also that optimized reward-model signals outperform outcomes that rely on human labels. Code: https://github.com/Gen-Verse/Open-AgentRL

1. Introduction

RLAnything addresses insufficient supervision in long-horizon reinforcement learning by dynamically optimizing policy, reward model, and environment in a closed loop. It combines outcome and step-wise feedback, jointly improves reward modeling, and adapts tasks using critic feedback across diverse applications.

  • Motivation: Binary outcome rewards alone provide insufficient supervision when policies interact with environments iteratively over long trajectories.Generative reward models can provide step-wise signals but usually require high-quality, task-specific supervision.
  • RLAnything: RLAnything forges the environment, policy, and reward model through closed-loop optimization, with each component continuously receiving feedback from the others.The framework aims to amplify learning signals across complex LLM and agentic scenarios.
  • RLAnything: The policy combines verifiable outcome rewards with reward-model step-wise signals, while the reward model is optimized through consistency feedback from outcomes and self-consistency.The refined reward model then supplies step-wise supervision for policy learning.
  • RLAnything: Environment tasks are automatically adapted using critic feedback from the policy and reward model, with summarized policy failures guiding task perturbations.The approach adjusts task difficulty while preserving the task’s essence.
  • Empirical results: RLAnything improves performance across computer-use agents, text-based LLM agents, and coding LLMs, including 9.1% on OSWorld and 18.7% and 11.9% on Alf World and LiveBench.The reported gains use Qwen3-VL-8B-Thinking on OSWorld and Qwen2.5-7B-Instruct on Alf World and LiveBench.

2. RLAnything

RLAnything couples policy training, reward-model optimization, and automatic environment adaptation into one feedback-driven pipeline. Its reward design combines trajectory outcomes with step-level evaluations, while theory motivates balancing task difficulty to improve both policy and reward-model learning.

  • Policy feedback: The policy is trained with integrated feedback combining trajectory-level outcome rewards and reward-model step-wise signals.The framework samples trajectories, obtains outcomes, queries the reward model repeatedly for each step, and updates the policy.
  • Policy feedback: The step reward combines the outcome signal with nuanced step-wise feedback, and advantages are standardized across trajectories at the same step index.The default coefficient for combining the signals is λ=1.
  • Environment adaptation: New tasks are adapted from critic feedback and accepted-task scaling, with task counts growing approximately linearly with training steps.The reward model’s evaluative traces guide concrete task modifications.
  • Reward-model optimization: The reward model receives consistency feedback by comparing its step-level evaluations with the estimated overall quality of each trajectory step.The supervision signal is the product of the step-quality estimate and each independent reward-model evaluation.
  • Reward-model optimization: The reward model is evaluated on process accuracy and outcome accuracy, reflecting both step-wise quality and a step’s influence on the final outcome.The paper motivates optimizing both logical correctness and future impact prediction.
  • Theory: Reward precision improves as the number of independent evaluations increases when μ>1, with A ≥ 1 − e^(-m(μ−1)^2/4).The theorem states that A approaches 1 as m approaches infinity if and only if μ>1.
  • Environment adaptation: Overly difficult or easy tasks make importance sampling unbalanced, so moderating task difficulty can improve both policy training and process reward-model training.The framework estimates difficulty from rollout accuracy and uses thresholds to propose harder or easier tasks while preserving task essence.

3. Experiments

Experiments evaluate RLAnything across GUI, text-based game, and coding settings, testing dynamic reward modeling, environment adaptation, and policy optimization. Each dynamic component improves training or evaluation, with gains in OSWorld, reward-model quality, coding, task scaling, and reasoning behavior.

  • Experimental Settings: Experiments cover computer-use agents, text-based interactive games, and RLVR coding tasks with automatic environment adaptation.The GUI setting uses OSWorld, the LLM-agent setting uses Alf World, and coding uses unit-test-based reinforcement learning.
  • Datasets: Training and evaluation use separate datasets, including in-domain and out-of-distribution splits for OSWorld and Alf World.OSWorld evaluation contains 230 in-domain and 139 OOD tasks; Alf World contains 140 in-domain and 134 OOD evaluation tasks.
  • Reward Modeling: The reward model evaluates each policy step using step quality and potential outcome impact, with three independent evaluations per response.GUI inputs include prior actions, recent images, and the action being evaluated; Alf World uses summarized actions and consequences.
  • Policy Training: Jointly optimizing the reward model and environment produces stronger policy optimization and a higher convergence point than adding fewer dynamic components.Environment adaptation also benefits reward-model training, yielding a three-fold gain for policy training and stronger OOD generalization.
  • GUI Performance: 9.1% accuracy improvement is reported for the optimized model on OSWorld, including a 5.2% improvement on out-of-distribution tasks.The optimized model improves across all OSWorld task categories.
  • Reward and Environment Scaling: Optimized reward-model signals alone outperform verifiable outcome rewards, while accepted synthetic tasks grow approximately linearly and achieve pass-at-least-one rates of 96.0%, 96.7%, and 94.2%.The pass rates correspond to GUI, Alf World, and coding settings, respectively; optimized models also improve agentic coding performance.

4. Related Works

The paper situates RLAnything in the shift from single-turn RL toward multi-turn, long-horizon agentic tasks, where outcome-only rewards do not provide sufficient step-wise supervision. It addresses this gap by coupling environment, policy, and reward-model optimization to produce stronger signals.

  • RL has expanded from reasoning tasks to coding, retrieval, and multi-turn agentic settings.
  • Generative reward models can provide step-wise signals, but existing systems lack the step-wise signal needed for long-horizon interactive tasks.
  • Coupled optimization of the environment, policy, and reward model yields stronger signals for the whole system.

5. Conclusion

RLAnything jointly optimizes the environment, policy, and reward model in a dynamic closed loop to amplify learning signals. Integrated supervision, consistency feedback, and automatic environment adaptation support stronger training signals, and experiments verify effectiveness across several settings.

  • RLAnything jointly optimizes the environment, policy, and reward model to amplify learning signals.
  • The framework combines policy supervision, reward-model consistency feedback, and critic-guided environment adaptation.
  • Experiments across GUI agents, LLM agents, and coding LLMs verify RLAnything’s effectiveness on OSWorld, AlfWorld, and LiveBench.

A.1. Proof of Theorem 1

The proof models reward-evaluation differences as independent bounded variables and applies the strong law of large numbers and Hoeffding’s inequality. These steps establish concentration properties for the analyzed reward signals.

  • The proof defines X_j from differences between positive and negative step-evaluation variables.
  • X_j has mean 2(p+ + p− − 1) and variance bounded by 4.
  • The variables X_j are independent, enabling application of the strong law of large numbers.
  • The probabilities p+ and p− are defined as the positive-label probabilities for the corresponding step evaluations.
  • The proof also invokes Hoeffding’s inequality to obtain a concentration bound.

A.2. Proof of Theorem 2 and Remarks

The theorem proof expresses process-reward optimization through outcome-conditioned and self-consistency terms, then analyzes how λ controls their balance. The remarks connect the formal reward to coding-task unit-test rewards and report robustness under the stated condition.

  • Each process-reward evaluation assigns a step reward by multiplying the outcome reward R_τ by its step label S_τi,j.
  • For positive outcomes, the expected old-policy process signal is proportional to the probability of a positive step label.
  • For negative outcomes, the corresponding expected signal is proportional to the negative-step-label probability with a negative sign.
  • The ratio of positive to negative weighted terms approaches zero when failure probability approaches one and infinity when success probability approaches one.
  • As λ increases, policy supervision shifts toward process reward while reward-model supervision shifts toward self-consistency.
  • The reward system still optimizes future-outcome prediction when the process reward is better than random guessing, and λ ablations show a supervision trade-off.
  • For coding tasks, unit-test rewards simplify to b_p_c + b_p_d − 1, targeting the population quantity p+ + p−.

B. Additional Experimental Results

Additional experiments evaluate reward-model step-wise accuracy under different supervision sources and trajectory sources across GUI-agent and LLM-agent settings.

  • Reward-model step-wise accuracy is reported under different supervision sources across GUI-agent and LLM-agent settings.
  • The two evaluations separately examine supervision sources and trajectory sources as factors affecting reward-model accuracy.
  • Reward-model step-wise accuracy is also reported under different trajectory sources across GUI-agent and LLM-agent settings.

B.1. Ablation Studies on Using Different Models for Reward Model Evaluation

The ablation studies test whether reward-model evaluation remains accurate when supervisor and trajectory-generation models are varied. Results are consistent with the main findings, supporting the use of an LLM as judge.

  • Different supervisor models produce the same conclusion when evaluating the accuracy of assigned step-wise rewards.
  • The ablations use OpenCUA-72B in the GUI setting and gpt-oss-20b in the LLM-agent setting.
  • Varying the trajectory-generation models also yields results consistent with the main results in Table 1.
  • These results validate the approach of using an LLM-as-a-judge for evaluation.

B.2. Examples of Environment Adaptation

RLAnything adapts environments by making tasks easier or harder according to policy performance and critic feedback, with examples showing how targeted changes restore or increase learning signals.

  • GUI Task Adaptation Example 1 (Target: Easier): 0 → 0.125: Adding targeted tips to a GUI task increases policy accuracy when the original task yields no successful trajectories.The tips are generated from summarized policy error patterns identified by the reward model.
  • GUI Task Adaptation Example 2 (Target: Harder): Tasks that are too easy for the policy are made harder by switching to a more challenging task template.
  • GUI Task Adaptation Example 2 (Target: Harder): A GUI task becomes harder by adding a requirement to rename the copied Revenue column to Profit.
  • Task Difficulty Control: Difficulty is adjusted by varying the number of objects involved in the task.
  • Alf World Task Adaptation: AlfWorld adaptation modifies object and receptacle goal tokens while preserving the original goal framework.
  • Alf World Task Adaptation: Harder AlfWorld goals use valid object–receptacle pairs with fewer available instances, such as CreditCardType and ArmChairType.

C.2. Evaluation for Reward Models

Reward models are evaluated on step-wise and outcome-related accuracy using stronger evaluators or verifiable outcomes, with separate protocols for GUI, text-based, and coding settings. Coding evaluations additionally test generated code and unit tests within several agentic methods.

  • Evaluation Criteria: OSWorld and AlfWorld evaluations measure both process accuracy and outcome accuracy.Process labels come from a stronger reasoning model, while outcome labels use verifiable task outcomes.
  • GUI and LLM-Agent Evaluation: In OSWorld, Qwen3-VL-32B-Thinking provides eight independent evaluations per policy response, with majority vote producing the ground-truth label.Qwen3-VL-8B-Thinking generates 16 independent rollouts per task.
  • GUI and LLM-Agent Evaluation: In AlfWorld, Qwen3-32B evaluates trajectories generated by Qwen2.5-7B-Instruct on the out-of-distribution evaluation set.
  • Coding Evaluation: Coding evaluation labels solutions as correct when they pass all dataset-provided unit tests and labels generated tests by their behavior on correct and incorrect solutions.A perfect generated test is correct and rejects all non-ground-truth solutions.
  • Agentic Coding Applications: Optimized coding models are evaluated with MPSC, AlphaCodium, S★, and best of N.
  • Agentic Coding Applications: MPSC uses consistency scores over generated code, unit tests, and specifications to identify the best code solution.A specification consists of pre-condition and post-condition functions describing valid inputs and expected behavior.

C.8. Examples of Task Templates in GUI data

This appendix presents task templates for GUI and AlfWorld adaptation, then specifies a task-specific pipeline that samples trajectories, scores them, adapts difficulty, and updates the policy and reward model.

  • C.8. Examples of Task Templates in GUI data: GUI task templates pair each task template with an evaluator and a corresponding verifiable outcome file.The examples include spreadsheet filling, data concatenation, slide formatting, text alignment, and chart creation tasks.
  • C.8. Examples of Task Templates in GUI data: AlfWorld adaptation uses a task-difficulty adaptation prompt template alongside the GUI task templates.The supplied appendix identifies AlfWorld as an additional environment for task adaptation.
  • C.8. Examples of Task Templates in GUI data: The GUI examples vary evaluator scope, including full ranges, selected columns, selected slides, and partial date ranges.Several evaluators intentionally specify narrower or incomplete versions of the same task.
  • C.9. Task Specific Algorithm: The task-specific algorithm begins with an environment task set Q, policy π_θ, reward model r_ϕ, and high and low accuracy thresholds.A temporary task set T is initialized before sampling task subsets.
  • C.9. Task Specific Algorithm: For OSWorld and AlfWorld, the policy samples trajectories while the reward model samples reasoning and final scores used to compute policy rewards.For coding tasks, the reward model instead samples reasoning and unit tests, while execution results determine outcome supervision.
  • C.9. Task Specific Algorithm: Coding-task rewards use code accuracy and unit-test pass rates to assign trajectory-level and step-level rewards.Unit tests receive positive or negative rewards according to their pass behavior on successful or unsuccessful trajectories.
  • C.9. Task Specific Algorithm: Accepted new tasks are added to subsequent policy and reward-model updates together with the existing task set.The pipeline collects newly accepted tasks, resets temporary proposal sets, and updates both models.
  • C.9. Task Specific Algorithm: The algorithm summarizes step-wise errors and proposes harder or easier task variants from that feedback.Candidate tasks are accepted when their accuracy falls within threshold-dependent conditions relative to the original task.
Loading 2602.02488v1…