PaperlayerView exact revision on arXiv

Source-linked AI summary

From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation

Tingwen Du, Haoyu Guo, Hanxue Zhang, Xuanxu Wang, Junting Dong, Chao Gao, Mingfei Han, Tao Lu, Mulin Yu, Chunhua Shen, Xiaojun Chang, Zhihui Li

arXiv:2610.29560v1cs.RO

TL;DR

Long-horizon manipulation can require information about task progress and past outcomes that is unavailable from the current observation. SERA organizes a high-level VLM in an editable harness around a fixed VLA executor, managing context and revising the harness through rollout feedback. On RoboMME, autonomous SERA achieves 51.38% overall success, compared with 44.38% without self-evolution.

  • Problem

    History-dependent manipulation decisions can require task progress and past interaction outcomes that the current observation does not reveal.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    SERA uses an editable harness to manage task context around a high-level VLM and revise planning guidance, assessment criteria, and control rules from rollout feedback while keeping a fixed VLA executor.

    Source: §1 · pp. 2–3, §3.2 · p. 4, §3.3 · p. 6

  • Results

    51.38% overall success on RoboMME for autonomous SERA, compared with 44.38% without self-evolution.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results support further investigation into how organization and adaptation of the high-level planner can contribute to VLA-based control over long execution horizons.

    Source: §5 · p. 9

    Repeated VLM inference over observations and task context can add substantial planning latency, and the trade-off between reasoning quality and responsiveness remains to be systematically evaluated.

    Source: §5 · p. 9

Abstract

from arXiv · show

Vision-language-action (VLA) models achieve strong performance across embodied manipulation tasks by unifying visual understanding, language conditioning, and action generation. However, long-horizon tasks can involve non-Markovian dependencies: successful decisions may require retaining task progress and past interaction outcomes, while partial observability and execution errors compound over time. Existing work has begun to address these challenges by augmenting foundation-model policies with memory and by introducing hierarchical architectures that separate high-level planning from low-level execution. We organize the high-level VLM as a robotic agent through an editable harness to maintain task context and improve planning through execution experience. We instantiate this approach as SERA (Self-Evolving Robotic Agent), with a fixed VLA executor providing low-level control. The harness maintains and updates task context using observations and execution feedback to guide subsequent planning. Self-evolution uses environment rollouts to propose and evaluate harness revisions, with the selected configuration fixed before held-out testing. On RoboMME, SERA achieves 51.38% overall success with autonomous selection, compared with 44.38% without self-evolution. Our exploration suggests that agentic planning is a promising direction for history-dependent robotic manipulation.

1 INTRODUCTION

Long-horizon manipulation can require tracking task progress and execution outcomes that are not recoverable from the current observation. SERA uses an editable harness to manage context and revise high-level planning from rollout experience; on RoboMME, autonomous SERA outperforms the configuration without self-evolution.

1 INTRODUCTION

  • Long-horizon decisions may depend on task progress and past interaction outcomes that similar current observations do not reveal.The harness addresses the need to interpret execution feedback because issuing a command alone does not establish that an operation succeeded.Source: §1 · pp. 1–2
  • Prior work provides foundations in memory-augmented policies, hierarchical planning, and agentic feedback loops for studying context management and execution experience.These approaches motivate examining how a high-level planner can manage task context and use execution experience.Source: §1 · pp. 1–2
  • SERA organizes a high-level VLM within an editable harness around a fixed VLA executor, managing task context and using rollout feedback to revise planning guidance, assessment criteria, and control rules.The selected harness is fixed before held-out testing, while task context continues to update within each episode.Source: §1 · pp. 2–3
  • 51.38% overall success on RoboMME for autonomous SERA, compared with 44.38% without self-evolution.Self-evolution improves success by 7.00 percentage points over the initial configuration.Source: §1 · p. 3

2 RELATED WORK

Related work spans memory mechanisms for VLA policies, hierarchical interfaces between reasoning and execution, and agentic systems that use feedback or adapt harnesses. SERA explores execution-driven harness revision for high-level planning in history-dependent manipulation.

2.1 MEMORY-AUGMENTED POLICIES

  • Memory-augmented policies integrate historical information into action prediction through temporal representations, retrieval, or recurrent updates, with RoboMME evaluating history-dependent behavior.Examples include HAMLET, MemoryVLA, ContextVLA, WeaveLA, ECHO, and ReMem-VLA.Source: §2.1 · p. 3

2.2 HIERARCHICAL ROBOTIC PLANNING

  • Hierarchical approaches connect high-level reasoning to robot behavior through intermediate decisions, including language-model planning, feedback-based replanning, and structured execution interfaces.MemER and Goal2Skill combine memory with a VLM–VLA hierarchy, using historical keyframes or structured task memory with verification and replanning.Source: §2.2 · pp. 3–4

2.3 AGENTIC EMBODIED SYSTEMS AND SELF-EVOLUTION

  • Agentic systems use reasoning, actions, observations, and feedback for iterative decision-making or experience reuse, though evaluation settings may differ from robotic manipulation.Examples include ReAct, Reflexion, and Voyager.Source: §2.3 · p. 4
  • Embodied systems place task memory, verification, planning, or recovery around learned executors, including frozen VLA or WAM systems.Examples include Harness VLA, HarnessWAM, RoboMemory, HELM, and Cortex.Source: §2.3 · p. 4
  • SERA explores an editable high-level planning harness that manages context within episodes and is revised from development-stage execution experience before the selected version is fixed for testing.Related adaptation work includes ETA, SHAPER, and Zetta.Source: §2.3 · p. 4

3 METHOD

SERA formulates manipulation as history-dependent and organizes a VLM within an editable harness around a fixed VLA executor. The harness maintains task context and execution feedback, while rollout-based self-evolution revises and selects harness configurations before held-out testing.

3.1 PROBLEM FORMULATION

  • Similar observations can require different actions when earlier interactions or initial context change the task state, making the decision problem non-Markovian.The formulation represents interaction history h_t and predicts an action chunk A_t over horizon K, with task completion bounded by an interaction budget.Source: §3.1 · p. 4

3.2 AGENTIC HARNESS FOR PLANNING

  • The editable harness maintains task context from observations and execution feedback to guide a VLM’s commands to a fixed VLA executor.Context retains task progress, decision-relevant facts, outcomes, and uncertainty beyond the recent observation window.Source: §3.2 · pp. 4–5

3.3 EXECUTION-DRIVEN SELF-EVOLUTION

  • Rollout-driven self-evolution diagnoses execution failures and revises planning guidance, assessment criteria, or control rules without additional planner fine-tuning.A verifier compares candidates with cached baselines, supports further revision or rejection, and selected configurations remain fixed during held-out testing.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments assess SERA on history-dependent manipulation tasks, comparing agentic planning and execution-driven harness revision while illustrating successes and remaining execution-reliability challenges.

4.1 EXPERIMENTAL SETUP

  • RoboMME contains 16 history-dependent tasks across four suites, evaluated with 50 episodes per task and equally averaged success rates.The evaluated baselines include direct VLA, memory-augmented, and hierarchical policies.Source: §4.1 · p. 7

4.2 AGENTIC PLANNING

  • 51.38% overall success for SERA exceeds MemER’s 42.38% and FrameSamp+Modul’s 44.51%; SERA scores highest on Reference and Permanence.RoboMME results use task-averaged success rates.Source: §4.2 · p. 7, Table 1 · p. 8
  • In VideoPlaceOrder, the agent uses demonstration visit order and retained context to place a cube at the intended target after a swap.The demonstration is not resupplied at each planning round; context tracks whether grasping succeeded.Source: §4.2 · p. 7

4.3 EXECUTION-DRIVEN SELF-EVOLUTION

  • 44.38% to 51.38%: harness self-evolution gains 7.00 percentage points, with revisions addressing spatial interpretation, stage checks, and missing action candidates.The revisions improve performance on tasks such as SwingXtimes and PatternLock.Source: §4.3 · p. 8
  • PatternLock failure analysis identifies a direction-to-command mismatch and an omitted action; revised guidance and candidate restrictions complete the sequence using existing executor actions.The figure presents this as a self-evolution repair example.Source: §4.3 · p. 8, Figure 3 · p. 9
  • On the six-episode PatternLock revision panel, success rises from two initially to five after verifier feedback prompts a second revision.The second revision makes all eight existing directional commands available; these are pre-test results, not an isolated comparison of repair components.Source: §4.3 · p. 8

4.4 FURTHER IMPROVEMENTS AND DISCUSSION

  • A RoboMemArena pouring example motivates adding interventions for execution deviations when replanning is insufficient.The planner requests a second pour, but the executor places the container in a rack; subsequent regrasp attempts fail despite retained task context.Source: §4.4 · p. 8
  • Exploratory gripper interventions sometimes complete tasks that matched runs without intervention fail, but another run remains unsuccessful.The interventions fall outside the current planner-side harness, and reliable recovery remains open.Source: §4.4 · p. 9, Figure 3 · p. 9

5 CONCLUSION

The conclusion presents SERA as a harness-based approach to history-dependent manipulation and reports improved RoboMME success with self-evolution. It identifies planning latency as a limitation and describes the authors’ use of AI tools.

5 CONCLUSION

  • On RoboMME, autonomous SERA achieves 51.38% success, compared with 44.38% without self-evolution.The selected harness is fixed before held-out testing.Source: §5 · p. 9
  • Repeated VLM inference over observations and task context can add substantial latency during long-horizon execution, creating an unevaluated trade-off between reasoning quality and responsiveness.Future work will investigate more efficient context processing and reasoning strategies.Source: §5 · p. 9
  • The authors used language models to assist literature collection, code writing, experiment organization, and manuscript drafting and editing, with author review and final decisions.The authors state that they developed the core research ideas and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A.1 METRIC AGGREGATION AND EVALUATION SCOPE

The appendix describes the RoboMME success-rate aggregation and how autonomous intervention selection is handled for testing.

A.1 METRIC AGGREGATION AND EVALUATION SCOPE

  • Success rates are task-averaged across 16 tasks, with suite scores using the same rule restricted to each suite’s four tasks.Each task has 50 episodes, so task averaging and episode pooling coincide.Source: §A.1 · p. 15
  • Autonomous SERA’s test aggregate retains candidates selected through development and fresh-validation checks or allowed terminal development fallbacks, without replacing them based on test outcomes.Tasks without a selected intervention retain the base result.Source: §A.1 · p. 15

A.2 ROBOMME PER-TASK RESULTS

The appendix reports per-task RoboMME results as successful episode counts for a local MemER reproduction and autonomously selected SERA.

A.2 ROBOMME PER-TASK RESULTS

  • Table 2 reports successful episodes out of 50 per task, comparing SERA with a local reproduction of MemER using released checkpoints.The MemER counts are distinct from the public leaderboard results in Table 1; bold marks the higher count per row, including ties.Source: Table 2 · p. 15, §A.2 · p. 15

B.1 ROBOMME COMMAND INTERFACE

The RoboMME command interface retrieves and validates catalog templates, with grounding points tied to front-camera frames and normalized image coordinates. The VLA retains its native goal and observation interface, while execution evidence is handled in a later planning round.

B.1 ROBOMME COMMAND INTERFACE

  • The base retrieval scheme ranks known-task entries using the preceding skill, recorded successors, and skill frequency, returning up to five candidates without semantic query similarity.It combines the rankings with reciprocal-rank fusion.Source: §B.1 · p. 16
  • Each selected catalog template requires its specified number of grounding points, each with an object description, a front-camera frame, and normalized integer image coordinates.The harness checks template, point count, coordinate bounds, and frame validity before rendering the command.Source: §B.1 · p. 16
  • The VLA keeps its native task-goal and observation interface, including current front/wrist images, proprioceptive state, and an executor buffer distinct from the planner’s recent visual window.The command interface does not establish physical success; execution evidence is processed in the next planning round.Source: §B.1 · p. 16

B.2 CONTEXT STATE AND RUNTIME CONTROL

RoboMME context initialization uses benchmark task-specification priors and synchronized camera observations, while runtime summaries and control state are updated during execution. Development selection permits specified terminal development fallbacks, and deployed revisions must use task identity and runtime-observable activation conditions.

B.2 CONTEXT STATE AND RUNTIME CONTROL

  • Episode initialization uses benchmark task-specification priors, with synchronized front- and wrist-camera observations supplying scene-level and local interaction evidence during execution.The initialization processes the instruction and initial context.Source: §B.2 · p. 16
  • The base planner rewrites a progress summary of up to 1,600 characters covering phase, completed steps, task and physical state, execution state, and uncertainty.Structured fields include estimated held object, object support, and gripper state; initialization guidance stays fixed while runtime summary and control state update and reset between episodes.Source: §B.2 · p. 16
  • Execution status distinguishes progressing, completed, uncertain, stagnant, and regressed outcomes, while counters track repeated lack of progress and stagnation to guide planning mode.The counters are not verified task-completion counts, and the action-specific visual check is advisory rather than an independent Python-side veto.Source: §B.2 · p. 16

B.3 DEVELOPMENT SELECTION AND REVISION SCOPE

  • Offline failure analysis may use benchmark traces or expert demonstrations, but deployed revision activation conditions must rely on task identity and runtime-observable information.The additional diagnostic materials are not themselves a basis for unavailable-at-runtime activation conditions.Source: §B.3 · p. 16
  • Autonomous selection can retain terminal development fallbacks after search-budget exhaustion when they add one successful episode without observed regression; tasks without a selected intervention retain base results.The aggregate includes these fallbacks and is not limited to candidates passing fresh validation; baseline-failure-only panels cannot establish retention of prior successes.Source: §B.3 · p. 16
  • The checked runtime loads manifest guidance, criterion overlays, and control rules into corresponding harness operations, but rejects unregistered Python intervention plugins.Broader extension types in the manifest schema do not show that they were executed or establish unrestricted online code modification.Source: §B.3 · pp. 16–17

Related papers

Loading 2610.29560v1…