Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation

Yunhai Feng; Jiaming Han; Zhuoran Yang; Xiangyu Yue; Sergey Levine; Jianlan Luo

Source-linked AI summary

Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation

Yunhai Feng, Jiaming Han, Zhuoran Yang, Xiangyu Yue, Sergey Levine, Jianlan Luo

arXiv:2502.16707v1Feb 23, 2025cs.ROcs.AIcs.LG

TL;DR

Long-horizon manipulation challenges VLMs because they require precise physical reasoning and planning over extended sequences. ReflectVLM uses diffusion-generated future visual states to support VLM reflection and action refinement, and it significantly outperforms commercial VLMs and MCTS on the evaluated tasks.

Problem
VLMs lack the nuanced physical reasoning and long-horizon planning needed for complex multi-stage manipulation.
Method
ReflectVLM combines diffusion-based future-state imagination with reflection that critiques and refines VLM action plans, supported by targeted post-training.
Results
ReflectVLM significantly outperforms commercial state-of-the-art VLMs and MCTS on the evaluated manipulation tasks.
Takeaways & Limitations
Structured test-time reasoning can improve VLM performance on physically grounded, long-horizon manipulation tasks while maintaining computational efficiency.
Takeaways & Limitations
The current implementation reflects primarily on final outcomes, while richer intermediate feedback and multiple reflection rounds remain future directions.

Abstract

from arXiv · show

Solving complex long-horizon robotic manipulation problems requires sophisticated high-level planning capabilities, the ability to reason about the physical world, and reactively choose appropriate motor skills. Vision-language models (VLMs) pretrained on Internet data could in principle offer a framework for tackling such problems. However, in their current form, VLMs lack both the nuanced understanding of intricate physics required for robotic manipulation and the ability to reason over long horizons to address error compounding issues. In this paper, we introduce a novel test-time computation framework that enhances VLMs' physical reasoning capabilities for multi-stage manipulation tasks. At its core, our approach iteratively improves a pretrained VLM with a "reflection" mechanism - it uses a generative model to imagine future world states, leverages these predictions to guide action selection, and critically reflects on potential suboptimalities to refine its reasoning. Experimental results demonstrate that our method significantly outperforms several state-of-the-art commercial VLMs as well as other post-training approaches such as Monte Carlo Tree Search (MCTS). Videos are available at https://reflect-vlm.github.io.

1. Introduction

The paper targets multi-stage manipulation tasks requiring physical reasoning, long-horizon planning, adaptation, and generalization. ReflectVLM combines visual future-state prediction with reflection to improve VLM planning and outperforms commercial VLMs and MCTS.

1. Introduction: Complex manipulation tasks require reasoning about physical constraints, long-term action consequences, execution outcomes, and novel scenarios.
1. Introduction: VLMs are promising for visually grounded planning but still struggle with precise physical reasoning and long-horizon planning.
1. Introduction: ReflectVLM combines a diffusion-based dynamics model that predicts future visual states with VLM reflection that critiques and refines planned actions.The framework uses test-time computation and targeted post-training to improve physical reasoning without extensive retraining.
1. Introduction: The method significantly outperforms commercial state-of-the-art VLMs and MCTS on sequential interlocking-object manipulation tasks.It also surpasses supervised fine-tuning with the same labeled-data quantity while maintaining computational efficiency.
1. Introduction: The framework is designed to extend beyond manipulation to domains requiring visual understanding and sequential decision-making.

2. Related Work

Prior reflection methods mainly address language or visual comprehension, while robotic planners often lack visual imagination of future consequences. This work combines diffusion-based future-state prediction with VLM reflection for robust long-horizon planning.

2. Related Work: ReflectVLM extends reflection to robotics by grounding plan revision in diffusion-generated imagined future visual states.This differs from language-only self-critique and symbolic reasoning by providing concrete visual predictions.
2. Related Work: Many VLM robotics approaches rely on symbolic states or make single-step decisions from current observations without explicitly reasoning about future consequences.
2. Related Work: ReplanVLM and GameVLM use execution feedback but retain symbolic state representations, whereas diffusion-based prediction methods may omit plan reflection or accumulate prediction errors.
2. Related Work: Traditional TAMP combines symbolic and motion planning but struggles with real-world physical interactions and visual inputs, while learning-based methods often require task-specific engineering.
2. Related Work: The proposed approach leverages VLM knowledge while adding structured physical reasoning through visual imagination and reflection without extensive engineering or large training datasets.

3. Preliminaries and Problem Statement

The task is formulated as a partially observable Markov decision process in which a VLM maps current and goal images to high-level manipulation actions. Training assumes access to interactive expert supervision.

3. Preliminaries and Problem Statement: The environment is modeled as a POMDP with physical states, manipulation actions, transition dynamics, RGB observations, and an observation model.The action space includes picking up, inserting, reorienting, and putting down objects, with primitive failure rate ϵ.
3. Preliminaries and Problem Statement: The policy seeks a sequence of actions reaching a goal state while observing only images, using the current image and goal image as inputs.The VLM outputs high-level manipulation primitives in text form.
3. Preliminaries and Problem Statement: Post-training follows interactive imitation learning and assumes an expert policy supplies near-optimal actions for encountered training states.

4. Reflective Planning with Vision Language Models

The framework combines interactive VLM post-training with a diffusion dynamics model and inference-time reflective planning. The VLM imagines future observations, evaluates proposed action sequences, and revises its executed action using those predictions.

4.1. Interactive VLM Policy Post-Training: Interactive learning trains the VLM from expert-supervised trajectories while exposing it to learner actions and physical interaction outcomes.Training mixes learner and expert actions during rollouts, adds expert-labeled state-action pairs, and fine-tunes the policy iteratively.
4.1. Interactive VLM Policy Post-Training: Reflection training relabels terminated trajectories with a future observation so the VLM learns to revise action sequences while preserving the expert action target.The future image provides feedback about the consequences of the proposed sequence and supports a second reflection question-answering task.
4.2. Diffusion Dynamics Model: The diffusion dynamics model predicts the next visual observation from the current observation and action, enabling hypothetical future-state evaluation without executing actions.It formulates forward dynamics as image-to-image translation and builds on a pretrained InstructPix2Pix model rather than training from scratch.
4.2. Diffusion Dynamics Model: The diffusion model encodes observations and action conditions, denoises a latent transition with sampled noise, and decodes the predicted latent into a future observation.Its latent encoder and text encoder remain frozen, while the diffusion UNet and latent decoder are fine-tuned on task data.
4.3. Reflective Planning: Reflective planning feeds imagined future images and proposed action sequences back into the VLM, which outputs the final action to execute.The imagination rollout repeatedly proposes actions and generates future images before reflection conditions on the current observation, goal, imagined future, and plan.

5. Multi-Stage Robotic Manipulation Planning Tasks

The benchmark uses procedurally generated assembly tasks in which pieces must be inserted into a board over long horizons. The tasks require high-level sequencing over interlocking objects, with primitive actions executed by rule-based controllers.

5. Multi-Stage Robotic Manipulation Planning Tasks: The tasks require assembling randomly placed pieces into a target board configuration by inserting them one at a time.Initial and goal configurations are generated procedurally, with many tasks containing interlocking pieces.
5. Multi-Stage Robotic Manipulation Planning Tasks: Dependency graphs encode assembly order: each predecessor object must be assembled before its directed successor.The graph is shown for the blue assembly board alongside initial and goal configurations.
5. Multi-Stage Robotic Manipulation Planning Tasks: High-level plans use pick up, insert, reorient, and put down primitives over named objects, implemented by rule-based script controllers.The framework focuses on planning while allowing alternative low-level controllers, including learning-based policies.

6. Experiments

Experiments compare ReflectVLM with zero-shot VLMs, MCTS, and reflection variants on procedurally generated tasks, measuring success, generalization, and computation. ReflectVLM achieves the strongest reported performance while using less inference computation than MCTS.

Experimental setup: Evaluation tasks use previously unseen object configurations, colors, and spatial arrangements to test generalization under challenging physical-reasoning and multi-step-planning conditions.The evaluation set is separate from the training set and emphasizes varied configurations and spatial arrangements.
Quantitative results: ReflectVLM outperforms commercial zero-shot VLMs and MCTS on the evaluated multi-stage manipulation tasks.The comparisons include leading VLMs, a VLM-based MCTS policy, and the full ReflectVLM variants.
VLM zero-shot: 15 out of 100 tasks were solved by GPT-o1, primarily on simpler cases lacking sophisticated interlocking-mechanism reasoning.Gemini-2.0-flash-thinking and GPT-o1 performed marginally better than other tested zero-shot models, but remained insufficient for the complex tasks.
MCTS: 47.8% success was achieved by the pretrained VLM, whereas adding MCTS reduced performance to 24.0%.MCTS sometimes overrode valid base-policy plans and was difficult to tune because of value-function sensitivity and nuanced physical reasoning requirements.
ReflectVLM ablations: The full ReflectVLM variant with simulator inference performs best, while diffusion-model inference is slightly worse and removing reflection performs worst among variants.All variants improve with additional post-training iterations, and reflection during training can improve implicit reasoning even without inference-time reflection.
Computation efficiency: ReflectVLM requires only a fraction of MCTS’s inference time per step while achieving substantially higher performance.The computation-cost comparison averages results over 100 tasks, with MCTS using one seed and other methods using five seeds on one A100 GPU.

7. Discussion

The discussion presents reflection as a post-training strategy that improves long-horizon VLM manipulation planning with less computation than MCTS. It identifies expanded context, physically constrained diffusion models, and multiple reflection rounds as future improvements.

Discussion: ReflectVLM demonstrates superior long-horizon planning with significantly less compute than traditional approaches such as MCTS.The discussion frames reflection as a post-training strategy for improving VLM policies on long-horizon manipulation tasks.
Future directions: Future architectures could use expanded context windows to incorporate richer intermediate feedback during reflection.The current implementation uses final outcomes because of VLM context constraints.
Future directions: The diffusion model could be improved with physical constraints and better architectures to stabilize predictions over longer horizons.The paper also proposes extending single-round reflection to multiple rounds while maintaining computational efficiency.

A. Task generation

The task-generation procedure discretizes assembly boards into voxels and incrementally adds interlocking pieces while resolving shared critical voxels. This produces boards composed of multiple interlocking components.

Board generation: Assembly boards are represented as 3D voxel arrays whose values indicate the piece occupying each voxel.The empty board begins with all voxels set to 0 before pieces are added.
Board generation: The generator first samples a base-board size and then iteratively adds pieces to the board.The example begins with a base board of size (12, 12, 3).
Intersection handling: Critical voxels at intersections between pieces are assigned to one piece to resolve overlaps and preserve an interlocking board structure.Figure 7 illustrates assigning intersection voxels to the red brick and then showing the resulting three-piece board in explosion view.

B. Samples of generated tasks

Generated tasks span simple peg insertion through complex multi-piece assemblies, with initial and goal configurations specified visually. The expert policy selects manipulation actions from symbolic object states and dependency statuses.

Samples of generated tasks: Generated tasks range from simple peg insertion to complex assemblies containing multiple interlocking pieces.Samples display initial configurations on top and goal configurations on the bottom.
Expert policy: The expert policy uses simulator object states and dependency information to classify pieces as DONE, READY, BAD, or BLOCKED.Statuses encode insertion, readiness, blocking, orientation, and predecessor or successor dependencies.
Expert policy: The policy chooses among picking up, inserting, reorienting, and putting down objects according to hand occupancy, predecessor completion, and global task status.Algorithm 3 specifies actions for held objects and for global READY or BAD B states.

D. Training details

The VLM combines a shared vision encoder with an LLM for multimodal reasoning over interleaved image-text inputs. Efficient task adaptation is performed through added LoRA layers.

Architecture: The implementation initializes the model with CLIP ViT-L/14 at 336 pixels and Vicuna-13B-v1.5 using LLaVA-v1.5 weights.
Architecture: The architecture uses a vision encoder and an LLM, with interleaved image-text inputs concatenated for multimodal reasoning.Images are encoded into latent embeddings before being combined with text embeddings.
Training parameters: Only newly added LoRA layers are fine-tuned to adapt the VLM efficiently.The default LoRA rank is 128.

D.2. Diffusion Dynamics Model

The diffusion dynamics model section describes training data generation and prompts that support visual future-state reasoning for puzzle manipulation. The reflection prompt evaluates an initial multi-step plan while executing only its first action.

Data generation: Training data comprises about 1M transitions from 50K trajectories collected on 10K boards with sub-optimal policies.Trajectories have a maximum length of 50 and terminate upon success; 50K transitions are reserved for evaluation.
Training parameters: The Diffusion Dynamics Model is initialized from pretrained Instructpix2pix weights.
Reflection prompt: The reflection prompt supplies a five-step plan and predicted future state, then asks the model to choose or revise the immediate next action.Only the first action is executed, and the initial plan may be modified when its predicted future is undesirable.

F.1. Zero-shot VLMs

The zero-shot evaluation tests multiple commercial and open-source VLMs under standardized prompting and image sizing. The reported materials include detailed results and visual examples of both successes and failures.

Evaluation setup: The evaluation compares zero-shot VLMs using standardized 336×336 images, zero temperature, and a maximum planning horizon of 50 steps.The evaluated models include LLaVA-Onevision, Gemini-2.0 variants, GPT-4o, and GPT-o1.
Evaluation prompt: The evaluation prompt requires the model to output only a single robot action, such as picking up a named colored object.The prompt presents the robot as observing and interacting with desktop objects.
Results: LLaVA-Onevision produced no valid extractable actions because of limited instruction-following capability, while the other closed-source models received detailed evaluation results.Success and failure cases for the closed-source models are visualized separately.
MCTS comparison: The MCTS comparison uses a VLM policy for action proposals, a heuristic value estimator, and tree search with selection, expansion, and backup phases.The search runs for 50 iterations and replans at each timestep.