PaperlayerView exact revision on arXiv

Source-linked AI summary

Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection

Zhen Liu, Xinyu Ning, Zhe Hu, Xinxin Xie, Weize Li, Zhipeng Tang, Chongyu Wang, Zejun Yang, Hanlin Wang, Yitong Liu, Zhongzhu Pu

arXiv:2604.13942v1cs.RO

TL;DR

Long-horizon manipulation requires persistent context and reasoning beyond the limited observation windows of many existing VLA systems. The paper separates semantic planning from visuomotor execution with structured memory, verification, and reflective recovery. On five representative RMBench tasks, it reports 32.4% average success rate versus 9.8% for the strongest baseline.

  • Problem

    Many VLA systems use limited observation windows, while long-horizon tasks require persistent task context, decomposition, verification, and adaptation when execution deviates.

    Source: §1 · pp. 1–2, §2.1 · p. 3

  • Method

    A VLM planner maintains structured memory, decomposes goals, verifies outcomes, and reflects on failures, while a VLA executor performs sub-tasks in a closed-loop framework.

    Source: §1 · p. 2, §6 · p. 12

  • Results

    32.4% average success rate versus 9.8% for the strongest baseline on five representative RMBench tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Effective long-horizon manipulation requires coordination among persistent task memory, semantic reasoning, and adaptive recovery, not only stronger action generation.

    Source: §6 · p. 13

    Performance is not uniform: on Observe and Pick Up, the method reaches 8%, slightly below the best baseline value of 9%.

    Source: §5.2 · p. 11

Abstract

from arXiv · show

Recent vision-language-action (VLA) systems have demonstrated strong capabilities in embodied manipulation. However, most existing VLA policies rely on limited observation windows and end-to-end action prediction, which makes them brittle in long-horizon, memory-dependent tasks with partial observability, occlusions, and multi-stage dependencies. Such tasks require not only precise visuomotor control, but also persistent memory, adaptive task decomposition, and explicit recovery from execution failures. To address these limitations, we propose a dual-system framework for long-horizon embodied manipulation. Our framework explicitly separates high-level semantic reasoning from low-level motor execution. A high-level planner, implemented as a VLM-based agentic module, maintains structured task memory and performs goal decomposition, outcome verification, and error-driven correction. A low-level executor, instantiated as a VLA-based visuomotor controller, carries out each sub-task through diffusion-based action generation conditioned on geometry-preserving filtered observations. Together, the two systems form a closed loop between planning and execution, enabling memory-aware reasoning, adaptive replanning, and robust online recovery. Experiments on representative RMBench tasks show that the proposed framework substantially outperforms representative baselines, achieving a 32.4% average success rate compared with 9.8% for the strongest baseline. Ablation studies further confirm the importance of structured memory and closed-loop recovery for long-horizon manipulation.

1 Introduction

Existing VLA systems are often limited by short observation windows and weak support for memory, verification, and correction across long-horizon tasks. The paper proposes a dual-system planner and executor with structured memory and reflective recovery, and reports higher average success than the strongest baseline.

1 Introduction

  • Most existing VLA systems rely on recent observations, while long-horizon tasks require persistent context, sub-task dependencies, intermediate verification, and adaptation to execution deviations.Occlusions, distractors, constrained interaction spaces, and changing scenes further complicate these tasks.Source: §1 · p. 1
  • The proposed framework separates semantic planning from visuomotor execution, using a VLM planner to decompose goals and a VLA executor to carry out sub-tasks.The planner evaluates post-conditions and can adjust a sub-task or revise the remaining plan when it detects discrepancies.Source: §1 · pp. 2–3
  • Structured memory records progress, outcomes, and failure signals to guide what the planner executes next and whether it should replan.The executor uses geometry-preserving filtered observations based on planner-predicted spatial constraints to suppress distractors.Source: §1 · p. 2
  • The framework closes the loop by evaluating sub-task post-conditions, identifying possible failure causes, and retrying with updated parameters or dynamically replanning.Ablations identify structured memory as a major source of improvement on memory-sensitive tasks and verification with reflection as beneficial on failure-prone tasks.Source: §1 · pp. 2–3
  • On five representative RMBench tasks, the framework achieves 32.4% average success rate versus 9.8% for the strongest baseline.The advantage is especially pronounced on memory-intensive M(n) tasks, where it reaches 38.7% compared with 9.0% for the best competing approach.Source: §1 · p. 3

2 Related Work

Related work covers VLA systems and memory-augmented approaches for long-horizon manipulation. The section identifies continued challenges in temporal context, active use of memory for recovery, and separation of high-level reasoning from motor generation.

2.1 Vision-Language-Action Foundations

  • VLA systems such as RT-2 and OpenVLA (Kim et al., 2024) use large-scale pretraining and multi-robot finetuning, but long-horizon manipulation still demands persistent context and stable control.Other advances include continuous action generation, diffusion-based control, and cross-embodiment training.Source: §2.1 · p. 3

2.2 Memory-Augmented VLA Systems

  • Memory-augmented VLA work preserves longer-range context through structured representations such as Multi-Scale Embodied Memory (Torne et al., 2026), MemoryVLA (Shi et al., 2025), and ReMem-VLA (Li et al., 2026a).MAP-VLA (Li et al., 2025b) and Meta-Memory (Mao et al., 2025) retrieve demonstrations or semantic-spatial cues during inference.Source: §2.2 · p. 3
  • MindExplore uses memory-based feedback between reasoning and execution, while NVIDIA ReMEmbR (Anwar et al., 2025) combines VLMs, LLMs, and retrieval-augmented generation for persistent embodied deployment.These approaches frame memory as relevant to reasoning about progress and environmental change, not only context preservation.Source: §2.2 · pp. 3–4
  • The section identifies two remaining gaps: memory less often supports verification, diagnosis, and correction, and memory-aware reasoning is often not separated from low-level motor generation.The proposed direction is modular decision making in which structured memory supports closed-loop reasoning while execution remains specialized.Source: §2.2 · p. 4

3 Problem Formulation

The formulation represents a long-horizon manipulation task as goal-conditioned control over adaptive sub-tasks. A high-level planner selects structured sub-tasks using observations and memory, while a low-level executor produces continuous actions within a verification-driven loop.

3 Problem Formulation

  • The robot receives an observation containing visual input and a proprioceptive state, and acts in a continuous space of commands for two 6-DoF arms and their grippers.The task is specified by a natural-language goal.Source: §3 · p. 4
  • The objective is to maximize the probability of achieving the goal by maximizing expected cumulative binary sub-task success over an adaptive number of sub-tasks.The number of sub-tasks K is adaptive, and each sub-task reward is binary.Source: §3 · p. 4
  • Decision making is factorized into a high-level planner that selects a structured sub-task from the observation, goal, and memory, and a low-level executor that turns it into continuous control.Each sub-task specifies an instruction, pre- and post-conditions, a maximum horizon, spatial constraints, and a primitive skill index.Source: §3 · p. 4
  • The low-level policy acts on geometry-preserving filtered observations that suppress distractor regions specified by the planner's spatial constraints.The high-level module sets objectives and constraints; the low-level module determines how to execute the motor behavior.Source: §3 · pp. 4–5
  • A high-level verifier evaluates each sub-task's completion signal to determine whether to continue, retry, or trigger replanning.The resulting hierarchy coordinates memory-guided planning, geometry-oriented perception, and low-level action generation.Source: §3 · p. 5

4 System Architecture

The architecture pairs a VLM-based agentic planner with a VLA executor. The planner handles planning, memory management, and reflective reasoning; the executor translates sub-tasks into continuous motor commands.

4 System Architecture

  • The dual-system architecture assigns planning, memory management, and reflective reasoning to a VLM-based agent, while a VLA model converts individual sub-tasks into continuous motor commands.Figure 2 depicts the closed-loop interaction linking semantic planning with visuomotor execution.Source: §4 · p. 5

4.1 Overall Architecture

The framework uses a two-scale closed-loop hierarchy: a high-level planner manages goals, memory, plans, and outcomes, while a low-level executor controls the current sub-task. A sub-task token and completion signal couple long-horizon reasoning to visuomotor execution.

4.1 Overall Architecture

  • The high-level planner maintains memory, generates or revises the plan, checks completion or failure, and triggers reflection when needed.The low-level executor receives the current sub-task and sensor stream, filters observations, and generates motor actions until termination is signaled.Source: §4.1 · p. 5
  • A sub-task token and planner-produced completion signal mediate the modules’ interaction, coupling semantic objectives with closed-loop visuomotor control.The planner defines objectives and execution conditions; the executor determines how to carry them out.Source: §4.1 · p. 6

4.2 High-Level Planner

The high-level planner combines task planning, memory management, and reflection to organize long-horizon decisions. It decomposes goals into revisable sub-tasks, tracks progress and failures, and uses diagnoses to select recovery actions.

4.2 High-Level Planner

  • The planner comprises a task planner, memory manager, and reflection engine for long-horizon decision making.The planner is built on a pre-trained VLM.Source: §4.2 · p. 6

4.2.1 Task Planning and Decomposition

  • The VLM converts a global goal into an ordered sequence of sub-tasks with stage-specific preconditions and postconditions.Plans can be revised during decision cycles to accommodate environmental changes, and spatial constraints identify task-irrelevant regions for scene filtering.Source: §4.2.1 · p. 6

4.2.2 Memory Management

  • Structured memory combines episodic history, working memory, and an error register to retain actions, observations, task progress, and failures.Episodic history records observations, active sub-tasks, executor actions, and completion signals; working memory is a compact natural-language world-state summary updated after sub-task transitions.Source: §4.2.2 · p. 6
  • A VLM-based verifier checks each sub-task’s postcondition against visual observations and working memory, providing semantic assessment beyond detecting action termination.The planner conditions decisions on memory; episodic history is compressed by retaining recent entries verbatim and absorbing earlier entries into working memory.Source: §4.2.2 · pp. 6–7

4.2.3 Reflection Engine

  • When execution fails, the reflection engine diagnoses a cause and records a recovery recommendation; local parameter adjustment can preserve the current plan, while retries are bounded by a threshold before replanning.A revised plan may substitute, skip, or insert sub-tasks to bypass obstructions.Source: §4.2.3 · p. 7, §4.2.2 · p. 7

4.3 Low-Level Executor

The low-level executor converts assigned sub-tasks into motor commands through geometry-oriented perception, diffusion-based skills, and local monitoring. Its action chunks return to the high-level planner for semantic verification, which coordinates retries or replanning.

4.3 Low-Level Executor

  • The executor is a hierarchical VLA stack with geometry-oriented perception, a diffusion-based skill library, and a local execution monitor.It turns assigned sub-tasks into continuous control commands through closed-loop visual feedback.Source: §4.3 · p. 7

4.3.1 Geometry-Oriented Perception via Masked Filtering

  • Planner-identified distractor regions are converted into geometry-preserving filtered observations that suppress distractors while retaining task-relevant scene geometry.The mask is initialized from distractor constraints and propagated through time.Source: §4.3.1 · p. 8

4.3.2 Diffusion-Based Skill Library and Action Generation

  • A selected diffusion-based primitive generates action chunks conditioned on the filtered image, proprioceptive state, and sub-task instruction.The executor uses the filtered observation rather than the raw visual input.Source: §4.3.2 · p. 8

4.3.3 Local Execution Monitoring and Interface to the High-Level Planner

  • After each action chunk or checkpoint, the executor returns an updated observation for planner verification; it reports local progress but does not assess global task completion.The planner determines whether the postcondition is satisfied and whether to retry or revise the plan.Source: §4.3.3 · p. 8
  • The decision cycle updates working memory after execution, advances to the next sub-task on success, and otherwise reflects before retrying, adjusting parameters, or replanning.Figure 3 distinguishes standard execution, locally adjusted retries, and replanning after repeated failures or timeouts invalidate the current plan.Source: Figure 3 · p. 7, §4.3.3 · pp. 8–9

5 Experiments

The experiments compare the framework with baselines on five RMBench tasks and analyze memory and recovery components. The framework’s largest benchmark advantage is on more memory-intensive tasks, while ablations examine how structured memory and explicit recovery relate to performance.

5.1 Experimental Setup

  • Evaluation covers five RMBench tasks, with 50 expert demonstrations per task and 100 rollout episodes; the method is trained for 30k optimization steps.Results use the final checkpoint; baseline values in the main comparison follow the same evaluation protocol.Source: §5.1 · p. 9

5.2 Comparison with Baselines

  • 32.4% total average versus the strongest baseline’s 9.8%; on M(n), the framework reaches 38.7% versus 9.0%.The comparison covers five representative RMBench tasks.Source: Table 1 · p. 10, §5.2 · p. 10
  • Figure 4 shows key frames from one rollout for each of five tasks, illustrating progression through different sequential interaction requirements.The rows show Observe and Pick Up, Rearrange Blocks, Battery Try, Blocks Ranking Try, and Press Button.Source: Figure 4 · p. 10, §5.2 · p. 11
  • 8% on Observe and Pick Up trails the best baseline’s 9%, so the framework’s gains are not uniform across tasks.The larger gains occur on tasks more tied to persistent memory, multi-stage reasoning, and long-horizon coordination.Source: §5.2 · p. 11

5.3 Targeted Ablations

  • Episodic history lifts memory-ablation average from 6.7% to 27.7%; the full model reaches 35.3%, while working memory’s effects vary by task.On Blocks Ranking Try, episodic history raises success from 6% to 54%; working memory improves Rearrange Blocks from 21% to 36% but lowers Blocks Ranking Try from 54% to 42%.Source: §5.3 · p. 11, Table 2 · p. 12
  • Recovery-ablation average success rises from 8.0% for the base model to 17.5% with verification, 24.0% with reflection, and 28.0% for the full model.The ablation evaluates tasks sensitive to failed execution and corrective retry.Source: §5.3 · pp. 11–12, Table 3 · p. 12

6 Conclusion

The framework separates semantic planning from visuomotor execution and links them through structured memory, verification, and reflection-driven recovery. Experiments show gains over representative baselines, especially on memory-intensive tasks.

6 Conclusion

  • A VLM-based planner with structured memory, verification, and reflection-driven recovery is paired with a VLA executor for geometry-oriented action generation on filtered observations.This separates high-level semantic planning from low-level visuomotor execution.Source: §6 · p. 12
  • The framework substantially outperforms representative baselines across five RMBench tasks, with especially large gains on memory-intensive M(n) tasks.Ablations identify structured memory as a major source of improvement and verification with reflection as improving robustness on failure-prone tasks.Source: §6 · p. 12
  • The results suggest long-horizon manipulation requires coordination among persistent task memory, semantic reasoning, and adaptive recovery, not stronger action generation alone.Source: §6 · p. 13

Related papers

Loading 2604.13942v1…