PaperlayerView exact revision on arXiv

Source-linked AI summary

ProAct: Agentic Lookahead in Interactive Environments

Yangbin Yu, Mingyu Yang, Junyou Li, Yiming Gao, Feiyu Liu, Yijun Yang, Zichuan Lin, Jiafei Lyu, Yicheng Liu, Zhicong Lu, Deheng Ye, Jie Jiang

arXiv:2602.05327v1cs.AI

TL;DR

Long-horizon LLM agents face compounding simulation errors and difficult value estimation in interactive environments. ProAct addresses these challenges with grounded lookahead distillation and a Monte-Carlo Critic, and achieves strong performance across stochastic and deterministic benchmarks, including unseen environments.

  • Problem

    Long-horizon LLM agents struggle with compounding simulation errors, while reinforcement learning faces significant value-estimation hurdles in long-horizon scenarios.

  • Method

    ProAct uses GLAD to distill MCTS trajectories into concise, environment-grounded reasoning chains and MC-Critic to provide a low-variance value estimator for multi-turn reinforcement learning.

  • Results

    A 4B parameter model trained with ProAct outperforms existing open-source baselines, compares favorably with state-of-the-art proprietary models, and generalizes to unseen environments across 2048 and Sokoban.

  • Takeaways & Limitations

    ProAct provides accurate lookahead capabilities and stable policy optimization across stochastic and deterministic long-horizon interactive environments.

  • Takeaways & Limitations

    Trajectory-level rewards lack discriminative credit assignment across time steps, can lead to model collapse, and make combination with MC-Critic difficult.

Abstract

from arXiv · show

Existing Large Language Model (LLM) agents struggle in interactive environments requiring long-horizon planning, primarily due to compounding errors when simulating future states. To address this, we propose ProAct, a framework that enables agents to internalize accurate lookahead reasoning through a two-stage training paradigm. First, we introduce Grounded LookAhead Distillation (GLAD), where the agent undergoes supervised fine-tuning on trajectories derived from environment-based search. By compressing complex search trees into concise, causal reasoning chains, the agent learns the logic of foresight without the computational overhead of inference-time search. Second, to further refine decision accuracy, we propose the Monte-Carlo Critic (MC-Critic), a plug-and-play auxiliary value estimator designed to enhance policy-gradient algorithms like PPO and GRPO. By leveraging lightweight environment rollouts to calibrate value estimates, MC-Critic provides a low-variance signal that facilitates stable policy optimization without relying on expensive model-based value approximation. Experiments on both stochastic (e.g., 2048) and deterministic (e.g., Sokoban) environments demonstrate that ProAct significantly improves planning accuracy. Notably, a 4B parameter model trained with ProAct outperforms all open-source baselines and rivals state-of-the-art closed-source models, while demonstrating robust generalization to unseen environments. The codes and models are available at https://github.com/GreatX3/ProAct

1 Introduction

ProAct addresses compounding simulation errors in long-horizon interactive tasks through grounded lookahead distillation and Monte-Carlo value estimation. Experiments show that a 4B model trained with ProAct outperforms open-source baselines, matches closed-source performance, and generalizes to unseen configurations.

  • Motivation: LLM agents in long-horizon environments accumulate simulation errors, causing plans to diverge from reality and producing suboptimal decisions.Ungrounded reasoning can further degrade performance through context drift and hallucination.
  • ProAct: ProAct uses Grounded LookAhead Distillation to compress environment-based search trajectories into concise reasoning chains during supervised fine-tuning.The chains include optimal paths and dead-ends, grounding action-outcome predictions in environmental feedback.
  • Evaluation: ProAct is evaluated on long-horizon games including stochastic 2048 and deterministic Sokoban.These testbeds provide objective measures across different environment dynamics.
  • ProAct: MC-Critic is a plug-and-play auxiliary value estimator introduced to stabilize and accelerate multi-turn agentic reinforcement learning.It leverages environment interaction to improve value estimation during policy optimization.
  • Results: A 4B parameter model trained with ProAct outperforms all open-source baselines and achieves performance comparable to state-of-the-art closed-source models.The model also exhibits strong generalization to unseen environment configurations.

2 Related Work

Related work develops multi-turn reinforcement learning, distills expensive search into efficient policies, and improves value estimation for agentic RL. ProAct contributes an environment-calibrated reasoning approach aimed at cumulative simulation errors in long-horizon internal simulations.

  • Multi-Turn Agentic Reinforcement Learning: Multi-turn agentic RL addresses stability and exploration challenges in trajectory-level optimization, with prior work expanding interaction horizons and identifying shallow-reasoning overfitting.AgentGym-RL uses curriculum-based horizon expansion, while RAGEN identifies the Echo Trap phenomenon.
  • ProAct’s Position: Unlike implicit-reward or pure-trial-and-error methods, ProAct introduces environment-calibrated reasoning to mitigate cumulative errors in long-horizon internal simulations.The work focuses on the quality of the agent’s internal reasoning process.
  • Reasoning Distillation from System 2 to System 1: Reasoning distillation transfers expensive inference-time search capabilities into more efficient policy intuition.Prior methods use explicit search or self-generated rationales, while related approaches also ground reasoning with internal world models.
  • Reasoning Distillation from System 2 to System 1: ProAct advances reasoning distillation by using MCTS to calibrate the agent’s thought process against ground-truth environmental dynamics.Its approach compresses search into environment-grounded reasoning rather than simply cloning verbose traces or explicit world-model states.
  • Value Estimation in Agentic Reinforcement Learning: Value estimation remains a bottleneck for agentic reinforcement learning, especially when sparse rewards make exploration and credit assignment difficult.Traditional neural critics can suffer from high bias and slow convergence in high-dimensional language spaces.

3 The ProAct Framework

ProAct internalizes multi-turn lookahead through grounded search distillation followed by online reinforcement learning with a Monte-Carlo Critic. The framework grounds reasoning in environment trajectories, compresses them into causal chains, and uses auxiliary value estimates to refine long-term decisions.

  • ProAct models LLM decisions as deliberation over reasoning chains followed by action execution in an interactive environment.
  • Grounded Lookahead Distillation: Grounded Lookahead Distillation uses environment-based MCTS to generate future trajectories and supervise compressed reasoning chains.The search records both optimal and suboptimal paths, which provide grounded alternatives for decision-making.
  • Grounded Lookahead Distillation: The distillation process compresses verbose search traces into natural-language chains linking observations, analyses, conclusions, and rejected alternatives.This preserves causal and counterfactual reasoning while reducing dependence on inference-time search.
  • Monte-Carlo Critic: MC-Critic is a plug-and-play value estimator that supports reinforcement-learning algorithms such as PPO and GRPO.It estimates values from environment interaction rather than relying only on a parameterized critic.
  • Monte-Carlo Critic: In MC-GRPO, incorporating MC-Critic changes advantage calculation and incentivizes prioritizing long-term returns over immediate single-step gains.

4 Experiments

ProAct is evaluated on complementary long-horizon environments that test planning under stochasticity, deterministic dynamics, sparse rewards, and error accumulation, with additional tests for generalization.

  • 2048 tests planning under uncertainty across trajectories containing hundreds of turns.
  • Sokoban tests deterministic planning with shorter trajectories and sparse rewards.
  • Additional unseen environment configurations are used to evaluate generalization beyond the training distribution.

4.1 Experimental Setup

ProAct evaluates long-horizon planning in stochastic 2048 and deterministic Sokoban, using a two-stage pipeline that combines GLAD supervised fine-tuning with MC-Critic-enhanced reinforcement learning.

  • Environments: 2048 introduces stochasticity through random tile spawning after each move, whereas Sokoban requires deterministic box-pushing toward target locations.The benchmarks therefore test planning under uncertainty and deterministic sparse-reward control.
  • Environments: Generalization tests modify 2048 grid size or minimum tile value and alter Sokoban levels, action spaces, or symbolic map representations.These variants include 3×3 2048, 3072, unseen Sokoban levels, modified actions, and changed map symbols.
  • Training: ProAct trains Qwen3-4B-Instruct in two stages: SFT with GLAD followed by reinforcement learning enhanced with MC-Critic.All methods use the same backbone architecture and parameter count, with end-to-end fine-tuning.
  • Training: GLAD constructs training samples from search-based trajectories containing textual states, distilled lookahead reasoning chains, and resulting actions.The dataset contains 25K samples for 2048 and 8K for Sokoban.
  • Training: MC-Critic is integrated with PPO and GRPO, and experiments compare GLAD-initialized reinforcement learning with training directly from the base model.The setup evaluates both in-distribution and unseen Sokoban levels, while using environment-specific scores for 2048 and Sokoban.
  • Evaluation: The study compares ProAct with open- and closed-source instruction models and evaluates MC-Critic against standard PPO and GRPO under a shared environment interface.The evaluation temperature is fixed to 0.6.

4.2 Results

GLAD improves lookahead reasoning and generalizes across environment variants, while MC-Critic further improves reinforcement-learning performance in both GLAD-initialized and from-scratch settings. Its effectiveness depends on rollout-horizon and sample-count choices, especially across long-horizon 2048 and sparse-reward Sokoban.

  • 4.2.1 GLAD: GLAD-trained 4B models outperform all open-source baselines and several closed-source models across standard and varied 2048 and Sokoban settings.Reported gains extend to reduced grids, 3072, unseen levels, modified action spaces, and altered symbolic representations.
  • 4.2.1 GLAD: GLAD produces more compact and accurate intermediate analysis that simulates multiple candidate actions, compares their consequences, and selects an action with consistent justification.The qualitative comparison contrasts this behavior with base-model redundancy, hallucinated configurations, and unstable trajectories.
  • 4.2.2 MC-Critic: MC-Critic consistently improves performance for both PPO and GRPO on 2048 and Sokoban when training starts from GLAD SFT checkpoints.The gains also transfer to variant environments, supporting additional refinement beyond the GLAD prior.
  • 4.2.2 MC-Critic: MC-PPO achieves the highest scores across both environments when training from scratch, while MC-GRPO outperforms other GRPO variants on 2048.On Sokoban, MC-GRPO performs comparably to Traj-GRPO because short trajectories reduce accumulated reward variance.
  • 4.2.2 MC-Critic: MC-Critic outperforms baseline methods on 2048 and Sokoban variants when trained from scratch, demonstrating robustness and generalization.The comparison covers reinforcement-learning methods with and without MC-Critic across environment variants.
  • 4.2.2 MC-Critic: On 2048, increasing Monte-Carlo sample count M from 10 to 100 and 1000 lowers value-estimate variance and improves MC-GRPO performance over Step-GRPO.On Sokoban, larger M dilutes sparse positive returns, while smaller M preserves distinctions between actions.
  • 4.2.2 MC-Critic: On 2048, performance improves as rollout horizon T increases and saturates around T = 100, whereas Sokoban performs best at T = 5.Longer horizons can increase single-trajectory variance or exceed the useful planning depth of short Sokoban levels.

5 Conclusion

ProAct combines grounded lookahead distillation with a low-variance auxiliary critic to improve long-horizon decision-making and policy optimization. Across stochastic and deterministic benchmarks, the approach supports strong performance and generalization from a 4B parameter model.

  • ProAct addresses compounding simulation errors and high-variance value estimation through a two-stage training paradigm.The framework targets accurate lookahead reasoning and stable policy optimization in long-horizon interactive environments.
  • GLAD distills explicit MCTS trajectories into concise, environment-grounded reasoning chains, connecting inference-time search with efficient policy intuition.
  • MC-Critic provides a low-variance, plug-and-play value estimator that enhances the stability of multi-turn agentic reinforcement learning algorithms.
  • A 4B parameter model trained with ProAct outperforms existing open-source baselines and achieves generalization comparable to state-of-the-art proprietary models.The reported results span stochastic 2048 and deterministic Sokoban benchmarks.

A.1 Environment and Evaluation Details

The appendix specifies textual environment interfaces, structured agent outputs, action validity and termination rules, task-specific scoring, and simplified Sokoban training levels. It also identifies the symbol mappings used for Sokoban variants.

  • Agents interact with 2048 and Sokoban through textual observations and textual action outputs in a structured response format.The required format is thought followed by a selected action.
  • Invalidly formatted or state-incompatible actions are penalized, and episodes terminate at terminal states or when the maximum trajectory length is exceeded.
  • 2048 evaluation uses cumulative merge score, including the standard protocol across 3 × 3 grid and 3072 environment variants.A merge receives a reward equal to the resulting tile value.
  • Sokoban evaluation reports the average number of boxes placed on targets per level across independent runs, capped at 200 steps.When a run reaches the step limit, scoring records the maximum number of boxes placed during the episode.
  • For reinforcement learning from scratch, simplified Sokoban levels solvable within 20 steps are used for training, while evaluation remains on the original test distributions.The training-only trajectory limit is T = 20.
  • Table 4 defines symbol mappings for the standard Sokoban environment and its symbolic-variant setting.

A.2 Training Config

The appendix presents the training configurations for supervised fine-tuning and reinforcement learning, with separate tables for RL experiments with and without GLAD supervision.

  • Table 5 presents the training configurations used in supervised fine-tuning experiments.
  • Table 6 presents the training configurations used in reinforcement learning experiments with GLAD supervision.
  • Table 7 presents the training configurations used in reinforcement learning experiments without GLAD supervision.

A.3 Prompt Template and Example Outputs

The appendix gives the full prompt template and representative 2048 outputs from a baseline model and the same model after GLAD supervised training. The examples show differing reasoning about candidate moves and board consequences.

  • The appendix identifies the first representative output as baseline Qwen3-4B-Instruct and the second as the same model after GLAD supervised training.
  • The baseline example evaluates candidate moves by discussing tile alignment, merges, empty spaces, and preservation of the top-left corner.
  • The displayed reasoning includes revisions and uncertainty while calculating the effects of right, down, and left moves on the board.
  • The prompt asks the agent to analyze the current 2048 board and return reasoning in a specified thought-and-move format.Strategy tips include preserving the highest tile in a corner, creating merge opportunities and empty spaces, and considering chain reactions.
  • One example selects up because it merges the two 2 tiles, increases empty spaces from 7 to 8, and preserves the corner strategy.
Loading 2602.05327v1…