PaperlayerView exact revision on arXiv

Source-linked AI summary

RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents

Zijing Zhang, Ziyang Chen, Mingxiao Li, Zhaopeng Tu, Xiaolong Li

arXiv:2507.22844v1cs.LGcs.AI

TL;DR

Long-horizon agents may achieve successful outcomes through flawed reasoning paths, leaving a gap in coherent and generalizable problem solving. RLVMR adds dense, verifiable rewards for explicit meta-reasoning during end-to-end RL and achieves state-of-the-art results on ALFWorld and ScienceWorld. The reported gains coincide with fewer redundant actions and improved error recovery, producing more robust and efficient agents.

  • Problem

    Outcome-only RL can reinforce flawed reasoning while optimizing final success, contributing to brittle policies that fail to generalize.

  • Method

    RLVMR provides dense, process-level rewards for verifiable planning, exploration, and reflection behaviors expressed through explicit cognitive tags.

  • Results

    RLVMR achieves state-of-the-art performance across ALFWorld and ScienceWorld, with the 7B model reaching 83.6% success on the hardest unseen split.

  • Takeaways & Limitations

    Improved reasoning quality, including fewer redundant actions and better error recovery, accompanies more robust, efficient, and generalizable agents.

Abstract

from arXiv · show

The development of autonomous agents for complex, long-horizon tasks is a central goal in AI. However, dominant training paradigms face a critical limitation: reinforcement learning (RL) methods that optimize solely for final task success often reinforce flawed or inefficient reasoning paths, a problem we term inefficient exploration. This leads to agents that are brittle and fail to generalize, as they learn to find solutions without learning how to reason coherently. To address this, we introduce RLVMR, a novel framework that integrates dense, process-level supervision into end-to-end RL by rewarding verifiable, meta-reasoning behaviors. RLVMR equips an agent to explicitly tag its cognitive steps, such as planning, exploration, and reflection, and provides programmatic, rule-based rewards for actions that contribute to effective problem-solving. These process-centric rewards are combined with the final outcome signal and optimized using a critic-free policy gradient method. On the challenging ALFWorld and ScienceWorld benchmarks, RLVMR achieves new state-of-the-art results, with our 7B model reaching an 83.6% success rate on the most difficult unseen task split. Our analysis confirms these gains stem from improved reasoning quality, including significant reductions in redundant actions and enhanced error recovery, leading to more robust, efficient, and interpretable agents.

1 Introduction

Existing training paradigms can produce either brittle efficiency or inefficient generalization because outcome-only RL rewards successful trajectories without supervising coherent intermediate reasoning. RLVMR addresses this gap with verifiable process-level rewards and improves benchmark performance, reasoning quality, robustness, and efficiency.

  • Outcome-only RL can reinforce flawed, illogical, or redundant reasoning paths that achieve successful final outcomes.
  • RLVMR provides dense, programmatic rewards for verifiable meta-reasoning behaviors such as planning, exploration, and reflection.Agents articulate cognitive states through special tags, enabling rewards for behaviors such as discovering new states or correcting prior mistakes.
  • RLVMR achieves state-of-the-art performance across ALFWorld and ScienceWorld, including 83.6% success for the 7B model on the hardest unseen split.
  • Analysis links RLVMR’s gains to improved reasoning quality, including fewer repetitive actions and enhanced error recovery.These changes improve agent robustness and efficiency.

2 Inefficient Exploration in Long-Horizon Agents

Long-horizon agents face a trade-off: SFT is efficient but brittle on unfamiliar tasks, whereas outcome-only RL generalizes better while producing redundant and invalid actions. Trajectory analysis and benchmark results motivate process-level supervision for coherent reasoning.

  • 2.1 Experimental Setup: ALFWorld evaluates embodied household-task capabilities and generalization across seen and increasingly novel task splits.L0 contains seen variants and categories, L1 holds out variants, and L2 holds out both variants and categories.
  • 2.1 Experimental Setup: The repetitive action rate measures meaningless repeated actions, while the invalid action rate measures actions invalid in the current state.
  • 2.2 Observing the Inefficient Exploration Problem: A GRPO trajectory repeatedly attempts to return to dresser 1 after already arriving there, despite needing to collect another keychain.The sequence illustrates redundant actions and limited self-evaluation during an unseen L2 task.
  • 2.3 The Problem of Inefficient Exploration: SFT improves seen-task success but produces brittle policies that fail to generalize to unseen tasks.For the 7B model, success rises from 23.1% to 63.3% on L0 but falls to 37.5% on L2.
  • 2.3 The Problem of Inefficient Exploration: GRPO improves generalization but reinforces inefficient reasoning under outcome-only rewards.The 7B model reaches 77.3% on L1 and 52.3% on L2, while its L2 repetitive action rate is 31.2%.
  • 2.3 The Problem of Inefficient Exploration: Increasing model size improves success but does not resolve the underlying reasoning deficiencies.For GRPO on L2, the 7B model reaches 52.3% success versus 29.7% for 1.5B, but repetitive actions increase from 27.1% to 31.2%.
  • 2.3 The Problem of Inefficient Exploration: These trade-offs establish a need for process-level supervision that rewards coherent and efficient reasoning rather than only final outcomes.

3 Methodology

RLVMR formalizes long-horizon agent interaction as an MDP, structures reasoning with explicit meta-reasoning tags, and trains the agent with composite outcome and process-level rewards. GRPO-MR combines trajectory performance with context-aware reasoning quality to optimize the policy.

  • 3.1 Task Formulation as a Markov Decision Process: RLVMR models long-horizon agent-environment interaction as an MDP whose states, actions, and observations are natural-language sequences.The transition and reward functions define environment dynamics and feedback over these language-based spaces.
  • 3.2 Meta-Reasoning Framework: The framework extends ReAct with explicit tags for planning, exploration, reflection, and monitoring, separating cognitive functions from actions.These tags enable fine-grained analysis and supervision of the reasoning process.
  • 3.3 Cold Start: Training begins with supervised fine-tuning on successful trajectories annotated by a teacher model with meta-reasoning tags.The target model learns to imitate the annotated reasoning and action-generation patterns before reinforcement learning.
  • 3.4 Reinforcement Learning: The reinforcement-learning signal combines a sparse binary outcome reward with dense process rewards for planning, non-redundant exploration, reflection-based correction, and valid output formatting.Planning is rewarded when successful, exploration targets new objects or locations, reflection precedes corrective action after failures, and malformed tag-action structure receives a penalty.
  • 3.4.2 Group Relative Policy Optimization with Meta-Reasoning: GRPO-MR computes step-level advantages by combining normalized trajectory-level outcome performance with normalized rewards grouped by meta-reasoning tag.A weighting hyperparameter balances global outcome information against local reasoning quality, followed by clipped policy optimization with KL regularization.

4 Experiment

Across ALFWorld and ScienceWorld, RLVMR outperforms baseline methods, especially on unseen tasks, while producing more efficient and adaptive reasoning behavior. Ablations show that process-level rewards work best alongside outcome rewards and a lightweight cold-start phase.

  • Main Results: RLVMR consistently achieves state-of-the-art performance across ALFWorld and ScienceWorld, outperforming all baselines across benchmarks and model sizes.
  • Generalization: 83.6% success on ALFWorld’s unseen-L2 split is achieved by the RLVMR 7B model, improving 16.4 percentage points over GiGPO.
  • Model Efficiency: 87.9% success on ALFWorld’s unseen-L1 split is achieved by RLVMR-trained Qwen-1.5B, versus 66.0% for GPT-4o using standard ReAct prompting.
  • Ablation and Attribution: 56.3% success on ALFWorld unseen-L2 with RLVMR 1.5B nearly doubles Vanilla-GRPO’s 29.7% and exceeds GiGPO’s 48.1%.
  • Exploration Efficiency: RLVMR reduces repetitive and invalid actions, with exploration and reflection rewards supporting purposeful behavior, self-correction, and error recovery.A case study shows the agent triggering reflection after an ineffective action rather than entering an error loop.
  • Training Efficiency: 15.4 average actions on ALFWorld unseen-L2 is achieved by RLVMR, 28.1% fewer than GRPO’s 21.7 and 20.6% fewer than GiGPO’s 19.4.RLVMR also shows a stable decline in action counts during training, whereas baselines can be unstable or increase action counts.
  • Ablation Study: Removing AMC lowers success by 11.0 points on ALFWorld, while removing AT collapses success to 12.5% on ALFWorld and 7.8% on ScienceWorld.Removing the supervised cold-start phase also decreases performance by 15.7 points on ALFWorld and 7.7 points on ScienceWorld.

5 Related Work

Prior work uses reinforcement learning to improve LLM capabilities and deploys LLMs across many agentic domains, but multi-turn long-horizon tasks remain difficult because rewards are sparse and delayed.

  • Reinforcement learning has been used to align LLMs with human preferences and improve capabilities such as reasoning and emotional intelligence.
  • Multi-turn, long-horizon tasks remain challenging because reinforcement-learning rewards are sparse and delayed.
  • LLM-based agents support applications including code generation, web interaction, embodied intelligence, and emotional intelligence.

6 Conclusion

The paper addresses inefficient exploration by adding dense, verifiable supervision for meta-reasoning behaviors to reinforcement learning. Experiments on ALFWorld and ScienceWorld report state-of-the-art performance, better unseen-task generalization, fewer redundant actions, and improved error recovery.

  • RLVMR addresses inefficient exploration by rewarding explicit planning, exploration, and reflection during end-to-end policy optimization.
  • RLVMR combines a lightweight cold-start phase with end-to-end policy optimization to shape more robust, efficient, and adaptive reasoning.
  • RLVMR establishes a new state of the art on ALFWorld and ScienceWorld, improving success rates and generalization to unseen tasks.
  • The reported reasoning-quality improvements include fewer redundant actions and markedly better error recovery.

A.1 Dataset Details

ALFWorld and ScienceWorld are text-based environments for household and scientific reasoning tasks, respectively. Both include seen and unseen variants, with L2 consisting entirely of held-out task types for testing generalization.

  • ALFWorld requires textual exploration and common-sense reasoning for household tasks, while providing feedback only on final task completion.
  • ScienceWorld evaluates scientific reasoning across 10 domains and 30 subcategories through tasks involving multiple subgoals.
  • Both benchmarks include seen and unseen variants, with L2 comprising entirely held-out task types.
  • ALFWorld holds out Cool & Place and Pick Two & Place, while ScienceWorld reserves the final task type of each topic for unseen evaluation.

A.2 Implementation Details

Experiments use Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct models, with specified cold-start and reinforcement-learning training configurations.

  • Experiments use Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct models.
  • During cold start, training uses batch size 16 per GPU, learning rate 1 × 10^-5, and 5 epochs.
  • During reinforcement learning, batch size remains 16 per GPU, with 16 environments and 8 trajectories rolled out per environment at each step.

B Detailed Experiment Results

RLVMR generally outperforms other methods across ALFWorld tasks, with particularly strong performance on more complex tasks, indicating enhanced robustness and adaptability.

  • RLVMR generally outperforms other methods across ALFWorld tasks.The comparison covers success rates reported for multiple tasks and both Qwen2.5-1.5B and Qwen2.5-7B base models.
  • RLVMR shows particularly strong performance on more complex ALFWorld tasks.The reported pattern is presented as evidence that rewarding high-quality reasoning behaviors enhances robustness and adaptability.

C Prompts

The prompts structure agent interaction around tagged reasoning followed by an admissible action, using environment-specific action formats, state history, and an overall plan.

  • ALFWorld action space: ALFWorld actions include moving, picking up, putting down, pouring, dunking, mixing, navigating, eating, flushing, and focusing on objects.The prompt represents these actions as structured action-description pairs.
  • ALFWorld prompt: ALFWorld prompts provide task descriptions, recent observations and action history, current state, and the previous overall plan.The agent is instructed to adhere strictly to that plan while taking the next action.
  • ScienceWorld prompt: ScienceWorld prompts define a text-based elementary-science environment and provide task, step count, recent history, current observation, and available actions.Available actions include operations such as opening, activating, connecting, looking, reading, and using objects.
  • Reasoning protocol: Agents must express one concise reasoning sentence using a single tag pair before selecting an action.The prompt defines planning, exploration, and monitoring as distinct meta-reasoning modes.
  • Action protocol: Agents must present an appropriate current-step action within <action> ...</action> tags.The action must be selected from the situation’s admissible actions.
Loading 2507.22844v1…