PaperlayerView exact revision on arXiv

Source-linked AI summary

SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents

Wanxin Tian, Shijie Zhang, Kevin Zhang, Xiaowei Chi, Chunkai Fan, Junyu Lu, Yulin Luo, Qiang Zhou, Yiming Zhao, Ning Liu, Siyu Lin, Zhiyuan Qin, Xiaozhu Ju, Shanghang Zhang, Jian Tang

arXiv:2506.21669v2cs.AI

TL;DR

Long-horizon embodied agents need intermediate learning signals and rewards that generalize beyond task-specific settings. SEEA-R1 combines MCTS-based Tree-GRPO with a multimodal generative reward model. On ALFWorld, it reports state-of-the-art scores and strong results without ground-truth rewards.

  • Problem

    Long-horizon embodied tasks make intermediate credit assignment difficult, and handcrafted reward functions may not generalize to novel tasks and environments.

    Source: §1 · p. 2

  • Method

    SEEA-R1 combines MCTS-augmented Tree-GRPO for step-wise process signals with MGRM, a multimodal reward model trained on multi-turn trajectories.

    Source: §1 · p. 2

  • Results

    On ALFWorld, SEEA-R1 reaches 85.07% textual and 46.27% multi-modal success, outperforming prior models including GPT-4o; without ground-truth rewards, it scores 80.30% textual and 44.03% multi-modal.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors conclude that SEEA-R1 has potential as a scalable framework for embodied reasoning and autonomous learning.

    Source: §5 · p. 10

    SEEA-R1 does not yet fully address highly dynamic and unpredictable real-world environments; future work may scale to more environments, samples, and larger models.

    Source: M Limitations · p. 29

Abstract

from arXiv · show

Self-evolution, the ability of agents to autonomously improve their reasoning and behavior, is essential for the embodied domain with long-horizon, real-world tasks. Despite current advancements in reinforcement fine-tuning (RFT) showing strong performance in enhancing reasoning in LLMs, its potential to enable self-evolving embodied intelligence with multi-modal interactions remains largely unexplored. Specifically, reinforcement fine-tuning faces two fundamental obstacles in embodied settings: (i) the lack of accessible intermediate rewards in multi-step reasoning tasks limits effective learning signals, and (ii) reliance on hand-crafted reward functions restricts generalization to novel tasks and environments. To address these challenges, we present Self-Evolving Embodied Agents-R1, SEEA-R1, the first RFT framework designed for enabling the self-evolving capabilities of embodied agents. Specifically, to convert sparse delayed rewards into denser intermediate signals that improve multi-step reasoning, we propose Tree-based group relative policy optimization (Tree-GRPO) integrates Monte Carlo Tree Search into GRPO. To generalize reward estimation across tasks and scenes, supporting autonomous adaptation and reward-driven self-evolution, we further introduce Multi-modal Generative Reward Model (MGRM). To holistically evaluate the effectiveness of SEEA-R1, we evaluate on the ALFWorld benchmark, surpassing state-of-the-art methods with scores of 85.07% (textual) and 46.27% (multi-modal), outperforming prior models including GPT-4o. SEEA-R1 also achieves scores of 80.3% (textual) and 44.03% (multi-modal) without ground truth reward, surpassing all open-source baselines and highlighting its scalability as a self-evolving embodied agent. Additional experiments and qualitative analysis further support the potential of SEEA-R1 for future research in scalable embodied intelligence.

1 Introduction

Embodied agents face challenges in long-horizon tasks, including sparse intermediate feedback and reward functions that generalize poorly. SEEA-R1 combines MCTS-based policy learning with a multimodal reward model and reports strong ALFWorld results, including without ground-truth rewards.

Motivation

  • Long-horizon embodied tasks require perception, planning, and decision-making, but current LLMs and MLLMs struggle with perceptual grounding, structured planning, coherence, or adaptation.Sparse delayed rewards make credit assignment difficult, while handcrafted reward functions may not generalize to novel environments.Source: §1 · p. 2

Contributions

  • Tree-GRPO integrates MCTS into GRPO to turn sparse outcome rewards into step-wise process signals, while MGRM estimates rewards across tasks from multimodal, multi-turn trajectories.The authors describe joint training in which improved MLLM reasoning enhances MGRM accuracy and refined MGRM rewards advance policy learning.Source: §1 · pp. 2–3

Results

  • 85.07% textual and 46.27% multi-modal success rates on ALFWorld make SEEA-R1 state of the art, outperforming prior models including GPT-4o.The benchmark tests mapping abstract goals to visually grounded action sequences.Source: §1 · p. 2
  • 80.30% textual and 44.03% multi-modal success rates without ground-truth rewards surpass all open-source baselines.The authors present these results as evidence of SEEA-R1’s scalability as a self-evolving embodied agent.Source: §1 · p. 3

2 Related Works

Related work covers self-evolution through experience and search, planning approaches for embodied agents, and reinforcement fine-tuning in language and vision-language models. SEEA-R1 combines MCTS-based planning with a vision-grounded reward model in a self-evolution loop.

2.1 Self-Evolution Learning

  • Prior self-evolution work uses self-play, reinforcement learning, and trajectory refinement; SEEA-R1 builds on this line with MCTS-driven rollout, trajectory filtering, and policy–reward co-refinement.Source: §2.1 · p. 3

2.2 Planning for Embodied Agent

  • Embodied planning spans training-free VLM planning and training-based methods, some of which depend on predefined modules or expert-designed strategies.Other approaches incorporate planning into reward assignment or combine tree search with preference optimization.Source: §2.2 · p. 3

2.3 Reinforcement Fine-Tuning

  • R1-style reinforcement fine-tuning has expanded from language reasoning to visual tasks, while cited embodied methods still rely on task-specific priors; SEEA-R1 combines vision-grounded rewards with MCTS-based planning.The framework overview depicts data generation followed by updates to both the policy and reward models.Source: §2.3 · p. 3, Figure 2 · p. 4

3 Methods

The methods define embodied interaction and describe SEEA-R1 as a loop that generates MCTS experience and updates policy and reward models. MCTS-derived action values provide process rewards, while Tree-GRPO and MGRM support policy and reward learning.

3.1 Preliminaries

  • An embodied agent is defined by state, action, reward, and observation; the state records interaction history, while the multi-round scenario is modeled as a POMDP.MCTS is introduced as a heuristic decision-making algorithm for large state spaces, with selection, expansion, simulation, and backup steps.Source: §3.1 · p. 4

3.2 Self-Evolving Emodied Agent-R1

  • Data Evolution uses MCTS to generate experience, and Model Evolution updates the policy and reward models; the evolved models then drive the next data-generation cycle.Source: §3.2 · p. 5

3.2.1 Data Evolution: Experience Generation via MCTS

  • MCTS-derived action values estimate expected future reward and provide step-by-step process rewards from sparse environmental signals, with estimates refined through more simulations.The simulations explore a wider range of trajectories, progressively refining the experience data.Source: §3.2.1 · p. 5

3.2.2 Model Evolution: Co-refining Policy Model and Reward Model

  • MGRM uses multimodal interaction history to predict success, continue, or failure; it is trained with ground-truth labels when available or majority-voted trajectory predictions as pseudo-ground truth otherwise.Tree-GRPO reuses GRPO loss over grouped actions at each tree node, with token-level advantages and a KL penalty to the reference policy.Source: §3.2.2 · pp. 5–6

3.2.3 Iterative Self-Evolution Loop

  • SEEA-R1 alternates Data Evolution and Model Evolution, deploying the updated agent and MGRM in each subsequent data-generation phase.The authors state that this loop progressively enhances reasoning, reward understanding, and task execution.Source: §3.2.3 · p. 6

4 Experiments

The experiments evaluate SEEA-R1 across simulated, out-of-distribution, and physical settings, comparing performance and efficiency while testing training algorithms and reward designs. Results include strong ALFWorld performance, real-world gains, and evidence of generalization and self-supervised improvement.

4.1.1 Environment, Dataset and Evaluation

  • ALFWorld provides textual and visual observations with ground-truth rewards; EmbodiedEval tests five task types across 125 realistic 3D scenes, while physical tests span three task types in six environments.Each physical environment contained 12 test cases.Source: §4.1.1 · pp. 6–7

4.1.2 Implementation Details

  • Training uses Qwen2.5-VL-7B-Instruct and consistent hyperparameters across SFT, DPO, and GRPO; each iteration is one trajectory-sampling and model-training cycle.Evaluation uses zero decoding temperature, one in-context example per task, and average success rate; MCTS uses probabilistic expansion and a path budget, with 30 iterations and maximum simulation depth 30.Source: §4.1.2 · p. 7, §4.1.1 · p. 7

4.2.1 Overall Performance Comparison

  • 46.27% success in text+visual ALFWorld and 84.3% in text-only; the visual result outperforms GPT-4o and leading open-source models, while text-only completion steps (14.75) are on par with GPT-4o.The evaluation compares text+visual (MLLM) and text-only (LLM) settings.Source: §4.2.1 · p. 8

4.2.2 Real World Results

  • 34.72% absolute improvement in overall real-world success; kitchens rose from 1/12 to 9/12 and children’s rooms from 4/12 to 10/12.SEEA-R1 was evaluated with single-trajectory execution; Figure 4 visualizes the clothes-to-washing-machine task.Source: §4.2.2 · p. 8, Figure 4 · p. 8, Table 2 · p. 9

4.2.3 Comparative Study of Training Algorithms

  • Tree-GRPO consistently achieves higher success rates and fewer average steps than MCTS + DPO and MCTS + SFT on the multi-modal ALFWorld scenario.Figure 5 compares optimization algorithms over training iterations.Source: §4.2.3 · pp. 8–9, Figure 5 · p. 9

4.2.4 Efficiency Analysis

  • SEEA-R1 combines training-stage optimization and lightweight inference to achieve higher success with lower computational cost.MCTS is used only during training; at test time, ReAct-style single-path sampling achieves 3.1× higher success rate with 38.8% fewer tokens.Source: §4.2.4 · p. 9

4.2.5 Ablation Study: Self-Evolution with Different Rewards

  • Self-trained MGRM improves policies in both paradigms: supervised MGRM approaches the GT-reward upper bound, while self-supervised MGRM eventually exceeds the GT baseline.Figure 6 compares supervised and self-supervised reward configurations.Source: §4.2.5 · p. 9, Figure 6 · p. 10

4.2.6 Generalization on Out-of-Distribution Embodied Benchmark

  • 19.88% overall success on EmbodiedEval, versus 18.29% for base Qwen2.5-VL-7B-Instruct and 16.77% each for SFT and DPO.EmbodiedEval is an out-of-distribution multimodal embodied benchmark.Source: §4.2.6 · p. 10, Table 3 · p. 10

5 Conclusion

The conclusion presents SEEA-R1 as an RFT framework for self-evolving embodied agents, combining Tree-GRPO and MGRM to address sparse rewards and generalization. The supplementary material outlines its training loop, embodied-agent formulation, MCTS procedure, experiments, and limitations.

5 Conclusion

  • SEEA-R1 combines MCTS-based Tree-GRPO and a multi-modal reward model to address sparse rewards and generalization, with strong ALFWorld performance and self-learning generalization.The conclusion describes the framework as a scalable approach for embodied reasoning and autonomous learning.Source: §5 · p. 10

A Appendix Overview

  • The appendix overview covers algorithm details, agent formulation, MCTS, world-model feedback, datasets, extended experiments, case studies, reasoning traces, reward accuracy, training costs, and limitations.It also lists real-world task details and prompts for reproducibility.Source: A Appendix Overview · pp. 15–16

B Algorithm Details

  • Each training iteration generates MCTS-tree experience, extracts state-action-reward triples, updates the reward model and policy with GRPO, then carries the updated policy forward.The loop returns the optimized embodied agent after repeated iterations.Source: B Algorithm Details · p. 16

C Embodied Agent Formulation

  • The agent is formulated as a POMDP with state, action, transition, reward, observation, observation-probability, and discount components.Its observations may be text or images, while its state retains interaction history for multi-turn reasoning.Source: C Embodied Agent Formulation · pp. 16–17

D MCTS

  • MCTS initializes the root at the initial state, selects actions by UCT, expands nodes with environment observations and sampled actions, simulates rollouts, and backs up returns and action values.The backup updates visit counts, returns, and action values along the trajectory from terminal nodes upward.Source: D MCTS · p. 17

D.1 Training and Efficiency Analysis

Training-time MCTS is paired with pruning and a lightweight inference path; the reported efficiency comparison favors SEEA-R1 on success and token use.

D.1 Training and Efficiency Analysis

  • MCTS action values Q(s_t, a_t) are computed as the average rollout return for an action, providing the value estimate used in search.The displayed expression averages returns across the action's visit count.Source: §D.1 · p. 17
  • 36 hours on 8×A100 GPUs is the reported SEEA-R1 training cost, versus 24 hours for Qwen2.5 fine-tuning and 48 hours for PPO under similar hardware settings.The paper characterizes this as a modest budget relative to its performance against proprietary models.Source: §D.1 · p. 17
  • O(pLk) complexity is reported for the pruning strategy, which expands k actions per node with 50% probability and limits full expansions to L = 5 per path.The passage contrasts this with standard MCTS scaling as O(bd).Source: §D.1 · p. 17
  • 3.1× higher success and 38.8% fewer tokens are reported for SEEA-R1 using single-path inference without MCTS search overhead.MCTS is used during training, while inference uses lightweight ReAct-style reasoning.Source: §D.1 · p. 17, Table 4 · p. 18

E Exploration of World Model as environmental feedback in Self-evolving Agent

The world-model experiment examines whether a learned model can provide environmental feedback for self-evolving agents. The authors report trajectory reliability, instruction-following, and generation-speed limitations that hinder this use.

E Exploration of World Model as environmental feedback in Self-evolving Agent

  • A single accurate and complete trajectory was extremely difficult to obtain when the agent relied on a world model for feedback.The authors conclude that current world models are not yet fully reliable world simulators.Source: E Exploration of World Model as… · p. 18
  • World models often produce visual hallucinations, including impossible dynamics, warped or vanishing objects, and artifacts, making their videos unreliable as simulator feedback.The authors report that repeated tests could not reliably prevent these hallucinations, even with detailed prompts.Source: E Exploration of World Model as… · p. 18
  • World models may generate coherent scenes that misinterpret unfamiliar instructions or fail to represent plausible outcomes of physically unsound plans.The authors identify generalization gaps and planning errors as distinct instruction-understanding failure modes.Source: E Exploration of World Model as… · p. 18
  • 10–30 minutes to generate a 5–10-second clip is reported for large visual diffusion models, slowing data collection and increasing training time and compute.When an output is unusable, data collection must restart and wait again.Source: E Exploration of World Model as… · p. 19

F Dataset Details

The experiments use ALFWorld’s unseen test split to assess planning and generalization, alongside broader multimodal benchmark evaluations. Further analyses compare training configurations, algorithms, long-term learning, and iterative self-evolution.

F Dataset Details

  • ALFWorld evaluation used the test-unseen split, which contains 134 games, to assess generalization to novel tasks and environments.The dataset also has 3,321 training games and 140 test-seen games.Source: F Dataset Details · p. 19

G.1 Evaluation on the Genenal Multi-Modal Benchmark

  • The multimodal evaluation describes MMBench’s reasoning and perception dimensions and identifies additional benchmarks covering multimodal reasoning, diagrams, hallucinations, and OCR.The supplied passages identify Tables 5 and 6 as performance comparisons but do not provide their results.Source: §G.1 · pp. 19–20, Table 5 · p. 20, Table 6 · p. 21

G.2 Impact of Sample Size and Batch Size on Training Stability

  • Sample=1024, Batch=512 achieved a final ALFWorld unseen-test success rate of 41.79%, outperforming smaller configurations.The medium and smallest configurations peaked earlier at 32.46% and 25.75%, respectively, and had lower final performance.Source: §G.2 · p. 20, Figure 9 · p. 20

G.3 Comparison with Different Algorithms

  • Tree-GRPO’s peak success rate was 32.46%, versus 25.37% for MCTS+DPO and 16.79% for MCTS+SFT on ALFWorld unseen tasks.The comparison spans training iterations.Source: §G.3 · p. 21, Figure 10 · p. 21

G.4 Long-Term Performance of Tree-GRPO

  • After 30 iterations, Tree-GRPO’s ALFWorld unseen-test success rate rose from 11.57% to 39.48%.The authors report minor fluctuations but a sustained upward trend.Source: §G.4 · p. 22, Figure 11 · p. 22

G.5 Impact of Iterative Self-Evolution on MCTS Performance and Data Quality

  • At 15 MCTS search steps, iter_12 reached approximately 0.8 average reward, compared with approximately 0.7 for iter_1.Later self-evolution iterations achieved higher average cumulative maximum rewards, with the gap tending to widen as search steps increased.Source: §G.5 · p. 22, Figure 12 · p. 23

G.6 Evaluating Data Quality from Iterative MCTS + GRPO via Supervised Fine-Tuning

The auxiliary SFT experiment tests whether data from later Iterative MCTS + GRPO rounds benefits a separate pretrained model. The section also presents ALFWorld task examples, including single- and multi-object placement.

G.6 Evaluating Data Quality from Iterative MCTS + GRPO via Supervised Fine-Tuning

  • The experiment filtered policy-generated samples to Advantage values greater than zero, then used each iteration’s data to SFT Qwen2.5-VL-7B-Instruct.This tested whether later-iteration data would improve a separate pretrained model.Source: §G.6 · p. 23
  • SFT success rates did not increase monotonically with source-data iteration, peaking at approximately 0.268 for iteration 7 before declining in later iterations.For example, the rate was approximately 0.247 for iteration 3, then approximately 0.164 and 0.179 for iterations 4 and 5.Source: §G.6 · p. 23, Figure 13 · p. 24
  • Data from later policy iterations may become tailored to the generating policy and less beneficial for fine-tuning a different model.The authors contrast the fixed-model SFT trend with the Iterative MCTS + GRPO policy’s own improving success rate.Source: §G.6 · pp. 23–24

H Case Studies on ALFWorld Tasks

  • The case studies present representative ALFWorld executions to illustrate instruction understanding, goal-directed planning, and environment interaction.The examples cover varied task categories.Source: H Case Studies on ALFWorld Tasks · p. 24

H.1 ALFWorld: Pick and Place Task; H.2 ALFWorld: Multi-Object Placement Task

  • The pick-and-place examples require locating and manipulating an object, with the two-CD safe task adding multi-object tracking and compound goals.The examples describe placing a pencil on a shelf and placing two CDs in a safe.Source: §H.1 · p. 25, §H.2 · p. 25

H.3 ALFWorld: Look Task

The qualitative examples cover perception and manipulation tasks, while reasoning traces compare a baseline with SEEA-R1 after five and ten training iterations. The later traces describe more structured plans and corrective reasoning.

H.3 ALFWorld: Look Task

  • The Look task requires locating an object by spatial reference and performing an observation action.The example asks the agent to look at a CD under the desklamp.Source: §H.3 · p. 25

H.4 ALFWorld: Heat and Cool Tasks; H.5 ALFWorld: Clean Task

  • Heat and cool tasks require object-state changes, subgoal ordering, and interaction with appliances; cleaning tasks test recognition of required preprocessing.Examples include heating a mug before placing it in a coffee machine and cleaning an egg before microwaving it.Source: §H.4 · p. 25, §H.5 · p. 25

H.6 ALFWorld: Composite Pick & Look Task

  • The composite Pick & Look task combines object manipulation with inspecting surroundings, requiring coordination of multiple sub-skills.Source: §H.6 · p. 25

H.7 Visualization of Policy and Reward Model Interactions

  • The policy-reward visualization shows visual state, reward-model reasoning, and the policy’s selected action in each panel.Source: §H.7 · p. 25

I.0.1 Baseline Model (Qwen-2.5-VL-7B-Instruct)

  • The baseline repeatedly attempted to open drawer 3 more than ten times without changing strategy after failure.The analysis characterizes this as exhaustive search with little reflection on failure causes.Source: §I.0.1 · p. 26

I.0.2 SEEA-R1 (Iteration 5)

  • After five iterations, SEEA-R1 tried alternatives after a failed placement action, eventually succeeding through strategic adjustment.Its reasoning considered the safe’s state, although its initial understanding was flawed.Source: §I.0.2 · p. 26

I.0.3 SEEA-R1 (Iteration 10)

  • After 10 iterations, SEEA-R1’s trace lays out a stepwise plan for finding and placing two CDs, then uses corrective reasoning when an action fails.The analysis also notes fewer redundant thoughts and more actionable logic.Source: §I.0.3 · p. 27

J Detailed Real-World Tasks

K Reward Model Accuracy Analysis

The reward-model analysis contrasts an untrained model’s task-state judgments with a GRPO-trained model. The following section describes the training-efficiency analysis, computing environment, and per-iteration timing measures.

K Reward Model Accuracy Analysis

  • GRPO training raised reward-model prediction accuracy to 91.67%, compared with 51.59% for the frozen model.The section identifies Table 8 as the accuracy comparison.Source: K Reward Model Accuracy Analysis · p. 27, Table 9 · p. 28
  • The frozen reward model sometimes labeled an ongoing task as Failure rather than Continue, which the authors say could disrupt learning.They report that such feedback could lead the agent to halt prematurely despite valid progress.Source: K Reward Model Accuracy Analysis · p. 27

L.1 Experimental Environment

  • Experiments ran on a cluster with 8 NVIDIA A100 80GB GPUs, using MS-Swift for distributed training and vLLM for inference evaluation.Source: §L.1 · p. 28

L.2 Training Efficiency Comparison

  • The training-efficiency comparison reports average per-iteration sampling, policy-training, reward-model-training, and total training times in minutes.Table 10 compares different reward configurations; the supplied passages do not give its cell values.Source: §L.2 · p. 29, Table 10 · p. 29

L.3 Key Observations

The self-supervised reward configuration avoids ground-truth rewards but adds training time; its sampling is faster, while policy-training time is unchanged.

L.3 Key Observations

  • 248 minutes of total training time for self-supervised MGRM training, including 244 minutes for reward-model training.The supervised configuration takes less total training time.Source: §L.3 · p. 29
  • 173.50 minutes of average sampling time for the self-supervised configuration, versus 190.12 minutes for the supervised configuration.Source: §L.3 · p. 29
  • 4 minutes of policy-training time in both configurations, with the reward configuration not significantly affecting this process.Source: §L.3 · p. 29
  • The self-supervised approach requires no ground-truth rewards but is computationally more expensive, a trade-off relevant where labeled data is scarce or unavailable.The findings highlight the computational considerations of deployment and trade-offs between training efficiency and ground-truth supervision.Source: §L.3 · p. 29

M Limitations

The limitations section identifies dynamic real-world environments as an unresolved challenge; the accompanying prompts specify interaction formats for benchmark, reward-model, and real-world tasks.

M Limitations

  • SEEA-R1 does not yet address the complexity of highly dynamic, unpredictable real-world environments; future work could scale to more environments, samples, and larger models.Source: M Limitations · p. 29

N.1 Prompt for the ALFWorld Dataset

  • The ALFWorld prompt lets the agent alternate between thought and action, choose from a fixed household-action set, and use immediate environment feedback.Invalid actions elicit “Nothing happened,” prompting the agent to try other options.Source: §N.1 · p. 30

N.2 Prompt for the EmbodiedEval Dataset

  • The EmbodiedEval prompt provides egocentric images, action history, and available choices for a 3D vision-language agent, with a step limit and constraints on object interaction.It also instructs the agent to reflect on failed actions and avoid repeated motion.Source: §N.2 · p. 31

N.3 Prompt for the real-world experiment.

  • The real-world prompt gives the agent a household goal, a fixed action set, and immediate feedback after each turn to guide subsequent steps.“Nothing happens” indicates an invalid action, after which the agent should try other options.Source: §N.3 · p. 32

N.4 Prompt for the Multi-modal Generative Reward Model.

  • The MGRM prompt labels a state as success, failure, or continue; failure means success is no longer reachable, while continue means it remains reachable.Source: §N.4 · p. 32

Related papers

Loading 2506.21669v2…