Source-linked AI summary
Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memory
Bingxin Xu, Yuzhang Shang, Emilio Ferrara
TL;DR
Long-horizon manipulation remains limited by compounding errors and missing transition conditions between contact-rich subtasks. BATON explores subtasks separately and composes them with transition-aware memory, improving task success by 11.6 points and cumulative success by 14.9 points over the current SoTA on RoboMemArena.
Problem
Long-horizon manipulation lacks effective composition across contact-rich subtasks and explicit conditions governing transitions between them.
Method
BATON uses an LLM agent to explore subtasks independently and compose them with language memory encoding invocation, handoff, and lookahead conditions around a frozen VLA.
Results
11.6 points in task success and 14.9 points in cumulative success over the current SoTA on RoboMemArena’s thousand-step household tasks.
Takeaways & Limitations
BATON makes transitions explicit as language-level contracts while keeping exploration cost additive and failures attributable to named subtasks or boundaries.
Takeaways & Limitations
The approach assumes a contact-rich policy backend that can be called through the same interface, including a VLA or WAM.
Abstract
from arXiv · showhide
Long-horizon robot manipulation chains many contact-rich skills into one multi-stage task. Vision-language-action (VLA) models increasingly master the individual skills, yet the chain still fails: errors compound beyond the policy's ability to correct, and one subtask silently constrains the next. A promising recipe freezes the VLA and puts an LLM agent in charge: it plans in language, moves in free space with analytic primitives, invokes the VLA only for contact-rich segments, and writes adaptation into language memory. Applied to long horizons, it breaks twice. (1) Competence comes from whole-task exploration at test time, whose cost is multiplicative in stages: if one stage needs T episodes, a K-stage task needs about T^K, and a failure does not reveal which stage caused it. (2) It has no representation of transitions: the VLA primitive carries an exit but no entry condition, so a subtask can succeed in a form its successor cannot use. We present BATON. Against (1), BATON makes the subtask the unit of exploration: each is explored in the cheap short-horizon regime and its solution stored in memory; a long-horizon trajectory is then composed from these solutions rather than discovered whole. Cost becomes additive (T*K) and every failure is attributed to a single stage. Against (2), BATON equips exploration with a transition-aware memory. Within a subtask, a verifier agent governs the invocation transition: the VLA is called only after the wrist view confirms the scene is ready. Across subtasks, a handoff transition restores an entry state disturbed by the predecessor's residue, and a lookahead transition selects the strategy whose outcome the successor can inherit. No parameters are updated. On the long-horizon benchmark RoboMemArena, BATON improves task success by 11.6% and cumulative success by 14.9% over the SoTA.
1 INTRODUCTION
BATON addresses the compounding errors, costly whole-task exploration, and fragile transitions that prevent contact-rich skills from composing over long-horizon manipulation tasks. It explores and stores verified subtasks, then composes them with transition-aware memory around a frozen VLA without parameter updates.
- Challenges: Long-horizon manipulation fails because errors compound across contact-rich stages and neighboring subtasks are coupled through the states execution leaves behind.VLAs may generalize individual skills but can replay motions from unseen intermediate states without recognizing or correcting accumulated drift.
- Challenges: Whole-task test-time exploration costs about T^K episodes for K stages and reveals only that the task failed, not which stage broke.The agent must trial staging orders, pre-contact poses, invocation timings, and termination thresholds across the full chain.
- Agentic subtask exploration: BATON makes each subtask the exploration unit, stores successful executions in abstract memory, and composes long-horizon trajectories from those verified pieces at additive cost.Subtasks are explored in the cheaper short-horizon regime, changing exploration cost from multiplicative to additive across stages.
- Transition-aware memory: Transition-aware memory governs invocation, handoff, and lookahead transitions so each VLA call starts from a verified scene and each successor inherits a usable entry state.A verifier confirms invocation readiness, restores disturbed successor entry conditions, and selects strategies whose outcomes the successor can inherit.
2 RELATED WORK
Prior work improves long-horizon manipulation by training boundary-aware policies, enriching memory, or placing agents above frozen VLAs. BATON instead combines these lines around transitions between subtasks, which it explores, verifies at runtime, and stores in memory.
- BATON’s position: BATON’s novelty lies in representing transitions between subtasks rather than introducing any component in isolation.It explores transitions one at a time, checks them at runtime, and stores them as memory edges.
- Long-horizon robot manipulation: 34–67% of chained-task failures are attributed to observation-space shifts at subtask boundaries, which additional demonstrations do not remove.Existing fixes therefore train for boundary compatibility, including configuration selection and downstream-success-shaped terminal states.
- Long-horizon robot manipulation: 77–100% isolated VLA skill success rates can collapse under composition, while similar first-subtask success produces 0%–83% success on the following subtask.Nine grasping policies with about 95% first-subtask success span this wide range on the next subtask.
- Memory, skill libraries, and task-aware grasping: Existing manipulation memories retrieve by instruction, observations, or execution history, recording what was seen rather than which strategy was selected.Memory-augmented VLAs either encode episodic context in the policy or retrieve past keyframes from an experience buffer.
- Agentic control of frozen VLAs: Agentic control can revise programs from execution feedback without training, and frozen VLAs can serve as retryable primitives for contact-rich manipulation.Related approaches also include code-based agentic skill discovery and reasoning agents operating above the motor level.
3 METHOD
BATON freezes the VLA and uses an LLM coding agent to explore and compose verified subtasks, making long-horizon search additive and failures attributable to individual boundaries. Its transition-aware memory governs VLA invocation, restores disturbed handoffs, and preserves successor-compatible strategies without updating parameters.
- Hierarchical exploration and composition: BATON scopes exploration to one subtask at a time, stores verified solutions, and composes them outward boundary by boundary instead of searching whole tasks.This makes exploration additive across subtasks and exposes failures at the newly examined boundary rather than inside verified compositions.
- Method overview: BATON operates over a harnessed frozen VLA with analytic staging primitives, assigns all roles to one LLM agent, and updates no parameters.The learned backend is VLA ACT, although any contact-rich policy exposing the same call could serve instead.
- Hierarchical exploration and composition: BATON verifies compositions level by level, chaining related neighbors first and promoting successful pairs into larger trusted units.Each handoff boundary is examined when its two sides first meet, while memory grows only when composition introduces a boundary-specific constraint.
- Transition-aware memory: Within each subtask, a verifier permits VLA ACT to fire only when a wrist-camera readiness condition confirms the scene is prepared.If readiness fails, the agent re-stages analytically before retrying the contact-rich primitive.
- Transition-aware memory: Across subtasks, the handoff transition restores an entry state moved by the predecessor, preventing successor failure caused by inherited residue.For example, a gripper left horizontally rotated after opening a microwave door can disrupt the following can-pickup skill despite both skills being individually correct.
4 EXPERIMENTS
Experiments evaluate BATON on RoboMemArena, a long-horizon benchmark designed around state carried across stages, using held-out scene configurations. BATON achieves the strongest reported composite-task performance and shows that subtask-level exploration outperforms whole-task exploration under the same budget.
- Setup: Exploration uses reference configuration seed 50, while evaluation averages two held-out configurations with seeds 51 and 52.The protocol separates exploration from evaluation by scene configuration.
- Benchmark: RoboMemArena contains 26 household tasks averaging over 1,000 steps and 3–9 verification stages, with 68.9% of subtasks memory-dependent.Tasks use partially observable kitchen scenes, and stage-level predicates enable TSR and CSR measurement.
- Results: 57.7% TSR and 78.8% CSR make BATON the best average performer in Table 1.These are BATON’s reported average task-success and cumulative-success results.
- Results: 11.6 and 14.9 points are BATON’s margins over FrameSamp+Modul [6] on TSR and CSR, respectively.BATON also surpasses the ground-truth oracle averages of 46.1 TSR and 64.8 CSR.
- Failure analysis: 64.8 CSR becomes 46.1 TSR for the oracle, while π0.5 converts 38.7% CSR into 21.5% TSR, locating difficulty between stages.Memory-augmented executors help most on counting and sequence, whereas occlusion remains hardest; the oracle reaches only 33.6 TSR there.
- Exploration comparison: 57.7% TSR from BATON’s subtask-by-subtask exploration exceeds whole-task Harness VLA’s 26.9% average under the same exploration budget.Whole-task exploration performs strongly on sequence at 82.0 TSR but never solves an occlusion task; BATON reaches 50.0 versus 39.1 on occlusion against the strongest reported system.
5 CONCLUSION
The conclusion frames long-horizon manipulation primarily as a problem of transitions rather than better skills, planners, or policies. BATON makes transitions explicit as a language-level object.
- Conclusion: Long-horizon manipulation is argued to be, to a first approximation, a problem of transitions.The paper contrasts this framing with approaches focused on better skills, planners, or policies.
- Conclusion: Whole-task exploration fails when it cannot identify which stage broke.This prevents failures from being attributed to a specific transition or subtask.
- Conclusion: Unconditioned VLA invocation and successor-blocking strategies expose transition failures that prior systems do not represent.The passage identifies missing entry conditions and incompatible subtask outcomes as examples of this unrepresented object.