PaperlayerView exact revision on arXiv

Source-linked AI summary

ProAct: Agentic Lookahead in Interactive Environments

Yangbin Yu, Mingyu Yang, Junyou Li, Yiming Gao, Feiyu Liu, Yijun Yang, Zichuan Lin, Jiafei Lyu, Yicheng Liu, Zhicong Lu, Deheng Ye, Jie Jiang

arXiv:2602.05327v1cs.AI

TL;DR

Long-horizon LLM agents face compounding errors when simulating future states. ProAct distills environment-based search into grounded reasoning chains and adds MC-Critic; its 4B model outperforms open-source baselines and performs comparably to closed-source models, with generalization to unseen environments.

  • Problem

    In long-horizon tasks, LLM agents’ simulated future states can accumulate errors, limiting reliable lookahead reasoning.

    Source: §1 · p. 2

  • Method

    ProAct pairs GLAD, which distills environment-based MCTS trajectories into concise grounded reasoning chains, with MC-Critic, a low-variance value estimator for stable multi-turn RL.

    Source: §1 · p. 2, §5 · pp. 13–14

  • Results

    A 4B model trained with ProAct outperformed open-source baselines, achieved performance comparable to state-of-the-art closed-source models, and generalized to unseen environments.

    Source: §1 · p. 2, §5 · p. 14

  • Takeaways & Limitations

    The findings support ProAct’s applicability across stochastic and deterministic long-horizon interactive environments.

    Source: §5 · p. 14

    The trajectory-level reward approach lacks individual-state value estimates, making it difficult to combine with MC-Critic.

    Source: §3.3.2 · p. 7

Abstract

from arXiv · show

Existing Large Language Model (LLM) agents struggle in interactive environments requiring long-horizon planning, primarily due to compounding errors when simulating future states. To address this, we propose ProAct, a framework that enables agents to internalize accurate lookahead reasoning through a two-stage training paradigm. First, we introduce Grounded LookAhead Distillation (GLAD), where the agent undergoes supervised fine-tuning on trajectories derived from environment-based search. By compressing complex search trees into concise, causal reasoning chains, the agent learns the logic of foresight without the computational overhead of inference-time search. Second, to further refine decision accuracy, we propose the Monte-Carlo Critic (MC-Critic), a plug-and-play auxiliary value estimator designed to enhance policy-gradient algorithms like PPO and GRPO. By leveraging lightweight environment rollouts to calibrate value estimates, MC-Critic provides a low-variance signal that facilitates stable policy optimization without relying on expensive model-based value approximation. Experiments on both stochastic (e.g., 2048) and deterministic (e.g., Sokoban) environments demonstrate that ProAct significantly improves planning accuracy. Notably, a 4B parameter model trained with ProAct outperforms all open-source baselines and rivals state-of-the-art closed-source models, while demonstrating robust generalization to unseen environments. The codes and models are available at https://github.com/GreatX3/ProAct

1 Introduction

ProAct targets long-horizon interactive tasks, where inaccurate simulated states compound into ungrounded plans, by training agents to internalize grounded lookahead. Its two-stage approach combines search-derived reasoning chains with environment-rollout value estimates.

1 Introduction

  • Long-horizon lookahead remains challenging because small errors in simulated environment dynamics compound, causing plans to diverge from reality.Deeper ungrounded reasoning can worsen performance through context drift and hallucination, while single-step reasoning misses long-term consequences.Source: §1 · p. 2
  • GLAD uses environment-based MCTS to explore promising paths and dead ends, then distills the search trajectories into concise reasoning chains grounded in environmental feedback.The chains train the model to predict candidate-action outcomes and identify risks rather than rely on hallucinated dynamics.Source: §1 · p. 2
  • MC-Critic adds lightweight environment rollouts to multi-turn RL, providing low-variance value estimates to stabilize and accelerate training.It is presented as a plug-and-play auxiliary value-estimation method for aligning internalized lookahead with environmental dynamics.Source: §1 · p. 2
  • A 4B ProAct model outperforms all open-source baselines, matches the performance of state-of-the-art closed-source models, and generalizes to unseen environments.Experiments use stochastic 2048 and deterministic Sokoban as long-horizon testbeds.Source: §1 · p. 2

2 Related Work

ProAct unifies multi-turn agentic reinforcement learning, reasoning distillation, and value estimation by calibrating agent reasoning with environment-based search and stabilizing policy updates with Monte Carlo estimates.

Multi-Turn Agentic Reinforcement Learning

Reasoning Distillation from System 2 to System 1

  • ProAct uses MCTS calibrated against environmental dynamics to compress search trees into concise future-trend estimates, distilling System 2 foresight into token-efficient System 1 policies without explicit-search overhead.Unlike cloning verbose search traces or explicit world-model states, this approach aims to retain diversity and accuracy.Source: §2 · p. 3

Value Estimation in Agentic Reinforcement Learning

  • Rather than training a separate parametric critic, ProAct uses low-cost Monte Carlo rollouts for dynamic state-value estimates that provide an environment-grounded, low-variance signal to stabilize PPO and GRPO.The passage contrasts this approach with neural, hierarchical, asymmetric, and turn-level reward estimation methods.Source: §2 · p. 3

3 The ProAct Framework

ProAct trains LLM agents to reason about future outcomes through grounded search-data distillation, then refines their decisions with online reinforcement learning and a Monte-Carlo Critic. Its framework separates deliberation from action and aims to reduce simulation drift without requiring inference-time search.

3.1 Overview

  • ProAct models LLM decisions as deliberation followed by action, and addresses compounding errors from inaccurate imagined transitions with grounded lookahead training.The framework has two phases: Grounded Lookahead Distillation (GLAD) from environment-based search, followed by online reinforcement learning with the Monte-Carlo Critic.Source: §3.1 · p. 3

3.2 Grounded Lookahead Distillation

  • GLAD externalizes lookahead by exposing the model to real future outcomes, then compresses those search trajectories into reasoning that captures future trends.This is intended to replace reliance on the agent’s potentially divergent internal world model during reasoning-data generation.Source: §3.2 · p. 4

3.2.1 Constructing Reasoning via Environmental Probing

  • Environmental probing uses MCTS to sample optimal and dead-end futures, which the agent analyzes to choose an action or backtrack from a suboptimal branch.The sampled paths provide a ground-truth future map, and the probing-decision-reflection loop supports course correction.Source: §3.2.1 · pp. 4–5

3.2.2 Cognitive Compression

  • Cognitive compression turns verbose search traces into natural-language, causal reasoning that compares future trends and rejected actions while preserving trade-offs.The compressed state-reasoning-action examples are used for supervised fine-tuning, teaching future-trend reasoning without search at inference time.Source: §3.2.2 · p. 5

3.3 Online Reinforcement Learning with Monte-Carlo Critic

  • MC-Critic targets the high-variance value estimates and training instability caused by the slow interaction and sample generation of LLM agents.Unlike traditional lightweight RL policies, LLMs can require hundreds or thousands of generated tokens for a single interaction step.Source: §3.3 · pp. 5–6

3.3.1 MC-Critic Formulation

  • MC-Critic estimates state value by averaging discounted returns from parallel rollouts, using a lightweight random policy to reduce the cost of LLM-policy rollouts.In 2048, the random policy can roll out over 1,000 trajectories in less than 3 seconds.Source: §3.3.1 · p. 6

3.3.2 Multi-Turn RL Optimization

  • MC-Critic plugs into GRPO and PPO: MC-GRPO uses critic action-values for step-level advantages, while MC-PPO combines its value estimate with the turn-level critic.For MC-GRPO, when sampled actions are identical, an action-space average-value baseline avoids the zero-advantage group that would otherwise yield zero policy gradients.Source: §3.3.2 · pp. 6–8

4 Experiments

ProAct is evaluated on 2048 and Sokoban, complementary long-horizon planning benchmarks, and on environment variants to test generalization beyond the training distribution.

4 Experiments

  • The evaluation uses 2048 and Sokoban as complementary long-horizon decision-making benchmarks for lookahead reasoning and error accumulation.2048 is stochastic, with uncertainty and hundreds of turns per trajectory; Sokoban is deterministic, with shorter trajectories and sparse rewards.Source: §4 · p. 8
  • ProAct is also tested on multiple environment variants to assess generalization beyond its training distribution.Source: §4 · p. 8
  • 2048 tests planning under uncertainty across hundreds of turns in a single trajectory.Source: §4 · p. 8

4.1 Experimental Setup

The experiments use 2048 and Sokoban to assess ProAct with a shared Qwen3-4B-Instruct backbone, a two-stage GLAD-to-MC-Critic training protocol, and environment-specific evaluation metrics. Generalization is tested using modified 2048 settings and Sokoban variants.

4.1.1 Environments

  • 2048 is a stochastic 4 × 4 tile-merging puzzle, while Sokoban is deterministic and requires pushing boxes onto targets.2048 runs for at most 1,000 steps and rewards merges; Sokoban runs for at most 200 steps and rewards boxes placed on targets, penalizes boxes pushed off targets, and grants +10 for level completion.Source: §4.1.1 · pp. 8–9
  • Generalization settings include 2048 variants with a 3 × 3 grid or minimum tile value 3 (3072), plus Sokoban unseen levels, action-space changes, and map-representation changes.Source: §4.1.1 · pp. 8–9

4.1.2 Model

  • All experiments use Qwen3-4B-Instruct, fine-tuned end-to-end, with baselines and ablations sharing its architecture and parameter count.Source: §4.1.2 · p. 9

4.1.3 Training Protocol

  • Training first applies GLAD SFT to search-generated trajectories, then uses MC-Critic with PPO or GRPO to refine decisions through reinforcement learning.The SFT data contain 25K 2048 samples and 8K Sokoban samples; RL starts either from the SFT checkpoint or original model weights, with simplified Sokoban levels capped at 20 steps for the latter.Source: §4.1.3 · p. 9

4.1.4 Evaluation

  • Performance is measured by cumulative tile-merge score in 2048 and average boxes placed per Sokoban level, with mean scores reported across multiple runs.The study compares ProAct with open- and closed-source instruction models and evaluates MC-Critic against PPO and GRPO under a shared interface and temperature 0.6.Source: §4.1.4 · p. 9

4.2 Results

GLAD gives the 4B model lookahead reasoning that generalizes across 2048 and Sokoban variants; MC-Critic further improves reinforcement-learning performance and robustness, with its benefits depending on rollout settings.

4.2.1 GLAD

  • The 4B GLAD model outperforms all open-source baselines and several closed-source models, with gains across 2048 and Sokoban variants, including unseen settings.For 2048, a model trained on the standard 4×4 grid also gains on 3×3 and 3072; for Sokoban, it generalizes to unseen levels and modified action spaces and symbolic representations.Source: §4.2.1 · pp. 9–10, Table 1 · p. 10
  • After GLAD supervision, GLAD-trained reasoning is more compact and accurate than the base model’s, with explicit multi-action lookahead and justified action selection.The base model sometimes hallucinates board states and transitions, while GLAD-trained reasoning describes the current board and compares candidate actions’ tile merges and future potential.Source: §4.2.1 · p. 10, Figure 3 · p. 11

4.2.2 MC-Critic

  • On GLAD-initialized models, MC-Critic improves PPO and GRPO final scores on both 2048 and Sokoban, with gains transferring to variant environments.The reported results attribute this refinement to Monte-Carlo returns while retaining out-of-distribution generalization.Source: §4.2.2 · p. 10, Figure 4 · p. 11
  • From scratch, MC-PPO achieves the highest scores across both environments, while MC-GRPO leads other GRPO variants on 2048 and MC-Critic improves generalization to environment variants.On Sokoban training levels, MC-GRPO performs comparably to Traj-GRPO because most levels take only a few steps; on longer-horizon 2048, Traj-GRPO degrades while MC-GRPO remains stable and improves.Source: §4.2.2 · pp. 11–12, Figure 5 · p. 12, Table 3 · p. 13
  • On 2048, increasing M from 10 to 100 or 1000 reduces Monte-Carlo value-estimate variance and improves MC-GRPO over Step-GRPO by supporting multi-turn lookahead.With only 10 trajectories, estimates are high-variance in this potentially hundreds-of-turns environment; variance decreases as M rises.Source: §4.2.2 · pp. 12–13
  • On sparse-reward Sokoban, M = 10 outperforms M = 100 and 1000 because larger rollout averages dilute rare successful returns and obscure action-value differences.A smaller M has higher variance but better preserves distinctions between actions, helping the model learn the optimal policy.Source: §4.2.2 · p. 13
  • The MC-Critic recipe recommends larger M for dense-reward environments, restrained M for sparse rewards, and T near the average steps needed for success.T should not be set excessively high.Source: §4.2.2 · p. 13
  • Increasing T improves 2048 performance until it saturates around T = 100, while longer horizons slightly reduce performance; on Sokoban, T = 5 exceeds T = 10 and 20.Longer single-trajectory rollouts increase return variance, and five steps suffice to solve most Sokoban levels.Source: Figure 6 · p. 13, §4.2.2 · p. 13

5 Conclusion

ProAct trains LLM agents for accurate lookahead and stable policy optimization in long-horizon interactive environments. Its GLAD and MC-Critic methods address compounding simulation errors and high-variance value estimation, while benchmark results show strong performance and generalization.

  • A 4B-parameter ProAct model outperforms existing open-source baselines and generalizes comparably to state-of-the-art proprietary models on stochastic 2048 and deterministic Sokoban benchmarks.The conclusion reports strong generalization capabilities across these distinct benchmark types.Source: §5 · p. 14
  • GLAD distills explicit MCTS trajectories into concise, environment-grounded reasoning chains, bridging expensive inference-time search and efficient policy intuition.It is the first stage of ProAct’s two-stage training paradigm.Source: §5 · pp. 13–14
  • MC-Critic is a low-variance, plug-and-play value estimator that significantly enhances the stability of multi-turn agentic RL algorithms.It is the second stage of ProAct’s two-stage training paradigm and addresses high-variance value estimation.Source: §5 · pp. 13–14

A Appendix · A.1 Environment and Evaluation Details

The appendix specifies textual interaction protocols and evaluation procedures for 2048 and Sokoban. It also describes simplified Sokoban levels used only for RL training from scratch.

A.1 Environment and Evaluation Details

  • For both environments, agents receive textual observations and must output thought and action fields; malformed or invalid actions are penalized, and episodes end at terminal states or beyond T.The environment wrapper parses and applies the action field.Source: §A.1 · p. 17
  • 2048 performance is the cumulative merge score at episode termination; merges reward the resulting tile value, and evaluation also covers 3 × 3 and 3072 variants.Agents choose up, down, left, or right.Source: §A.1 · p. 17
  • Sokoban observations use symbolic ASCII maps, with four directional moves and push actions available when adjacent to a box.Table 4 summarizes symbol mappings for the standard and symbolic-variant environments.Source: §A.1 · p. 17, Table 4 · p. 18
  • Because SFT Sokoban levels were too challenging for sufficient RL exploration, the authors constructed simplified levels solvable within 20 steps for RL training from scratch.These levels are used only for training; evaluation remains on the original test distributions unless stated otherwise, with T = 20 for this setting.Source: §A.1 · p. 17
  • Sokoban evaluation averages boxes placed on targets per level across runs, capped at 200 steps, using each run’s peak box count and ending deadlocked episodes early.The metric is smoother than binary success because levels contain 1 to 5 boxes.Source: §A.1 · pp. 17–18

A.2 Training Config

This section documents the training configurations used in the SFT and RL experiments.

A.3 Prompt Template and Example Outputs

The section presents the 2048 prompt and contrasts a baseline response that repeatedly revises its move reasoning with a GLAD-trained response that compares alternatives and chooses up.

A.3 Prompt Template and Example Outputs

  • The prompt asks the agent to analyze the 2048 board and choose a directional move, while advising it to keep the largest tile in a corner, create merges and empty spaces, and consider chain reactions.The response format is a thought followed by a move: up, down, left, or right.Source: §A.3 · p. 19
  • The baseline Qwen3-4B-Instruct response repeatedly contradicts itself about tile movement and merges before eventually choosing right.It alternates among left, down, and right, including a mistaken claim that a right move merges vertically aligned 4s.Source: §A.3 · pp. 19, 21
  • The GLAD-trained Qwen3-4B-Instruct response compares up, left, down, and right, then selects up to merge the right-column 2s while preserving the top-left 128.It also describes a possible follow-up left move and weighs how down or right would affect alignment with the corner anchor.Source: §A.3 · pp. 19, 22

Related papers

Loading 2602.05327v1…