Source-linked AI summary
Long-Horizon Manipulation via Trace-Conditioned VLA Planning
Isabella Liu, An-Chieh Cheng, Rui Yan, Geng Chen, Ri-Zhao Qiu, Xueyan Zou, Sha Yi, Hongxu Yin, Xiaolong Wang, Sifei Liu
TL;DR
Long-horizon manipulation is difficult because VLA policies must manage many dependent steps while handling progress changes and execution errors. LoHo-Manip uses a decoupled task-management VLM that predicts remaining subtasks and visual traces for a short-horizon executor, with repeated updates forming a closed loop. Across embodied, simulated, and real-world evaluations, it reports improved success, generalization, and task-planning accuracy, while its 2D trace remains limited for complex contact-rich behaviors.
Problem
Long-horizon manipulation requires progress tracking, state-dependent decisions, and recovery across many interdependent steps, exposing limitations of direct long-horizon action prediction.
Method
LoHo-Manip decouples a vision-language task manager that predicts remaining subtasks and 2D visual traces from a VLA executor adapted to follow those traces.
Results
LoHo-Manip improves success rates, generalization, and task-planning accuracy across embodied reasoning, simulation, and real-world benchmarks compared with monolithic VLA baselines.
Takeaways & Limitations
Repeated remaining-plan prediction provides implicit progress tracking, replanning, and failure recovery while the trace interface supports reusable task management across executor policies.
Takeaways & Limitations
The 2D visual trace may not fully express highly precise manipulation or contact-rich interaction patterns, and performance depends on accurate subtask grounding and trace prediction.
Abstract
from arXiv · showhide
Long-horizon manipulation remains challenging for vision-language-action (VLA) policies: real tasks are multi-step, progress-dependent, and brittle to compounding execution errors. We present LoHo-Manip, a modular framework that scales short-horizon VLA execution to long-horizon instruction following via a dedicated task-management VLM. The manager is decoupled from the executor and is invoked in a receding-horizon manner: given the current observation, it predicts a progress-aware remaining plan that combines (i) a subtask sequence with an explicit done + remaining split as lightweight language memory, and (ii) a visual trace -- a compact 2D keypoint trajectory prompt specifying where to go and what to approach next. The executor VLA is adapted to condition on the rendered trace, thereby turning long-horizon decision-making into repeated local control by following the trace. Crucially, predicting the remaining plan at each step yields an implicit closed loop: failed steps persist in subsequent outputs, and traces update accordingly, enabling automatic continuation and replanning without hand-crafted recovery logic or brittle visual-history buffers. Extensive experiments spanning embodied planning, long-horizon reasoning, trajectory prediction, and end-to-end manipulation in simulation and on a real Franka robot demonstrate strong gains in long-horizon success, robustness, and out-of-distribution generalization. Project page: https://www.liuisabella.com/LoHoManip
1 Introduction
LoHo-Manip addresses long-horizon manipulation by separating task management from short-horizon execution and repeatedly updating a remaining plan with an actionable visual trace. This closed-loop design supports progress tracking, replanning, and recovery without hand-crafted failure logic.
- 1 Introduction: The framework targets tasks with many interdependent subtasks, state-dependent decisions, and recovery requirements that make long-horizon manipulation difficult.Examples include locating, grasping, filling, navigating, opening, pouring, and closing during a single household instruction.
- 1 Introduction: The visual trace specifies where to move and which objects or regions to approach next, while conditioning the executor as an actionable prompt.This converts long-horizon planning into a sequence of short-horizon control problems.
- 1 Introduction: Receding-horizon remaining-plan prediction keeps failed subtasks in subsequent outputs, enabling implicit progress tracking, replanning, and recovery.The manager is repeatedly invoked from the current observation rather than planning only at the first frame.
- 1 Introduction: LoHo-Manip separates a task-management VLM from a short-horizon VLA executor to support modular long-horizon manipulation.The manager predicts remaining subtasks and traces, while the executor performs local control.
2 Method
LoHo-Manip separates progress-aware task management from short-horizon VLA execution, conditioning the executor on visual traces and repeatedly updating plans from current observations. The framework constructs completed/remaining language memory and future end-effector trajectories to support replanning and recovery.
- System Overview: The system separates a high-level task manager that reasons about progress from a low-level executor that performs short-horizon control.The manager is designed to interface with different VLA backbones.
- System Overview: The manager predicts a natural-language subtask and a 2D keypoint trace that specifies the associated spatial trajectory or interaction target.The trace is rendered into the executor’s observation as a spatial conditioning signal.
- Sub-Task and Trace Construction: Progress-aware plans explicitly split completed subtasks from remaining subtasks, providing compact language memory for tracking what has been done and what remains.The completed prefix is represented by C_t^star and the remaining suffix by R_t^star.
- Sub-Task and Trace Construction: The remaining visual trace is the future portion of the robot end-effector trajectory, represented as 2D pixel coordinates through the final primitive.This trace encodes spatial intent for the remaining plan.
- Training: Training decouples manager prediction from executor control, reusing one manager across VLA backbones while fine-tuning each executor to follow the trace interface.Supervision is constructed from grounded interaction events, subtasks, end-effector positions, and paired instruction-trace examples.
- Inference Loop: At inference, the manager repeatedly updates the completed/remaining plan and trace from the current observation while the executor acts for a short horizon.This receding-horizon loop updates textual memory from the latest manager output rather than relying on long visual histories.
3 Experiments
LoHo-Manip is evaluated across reasoning, trajectory prediction, embodied planning, simulation, and real-robot manipulation. The experiments test both broad benchmark performance and robustness under novel objects, arrangements, and language combinations.
- Evaluation scope: The evaluation combines four capability dimensions: reasoning and trajectory prediction, embodied planning, simulated closed-loop manipulation, and real-world execution.This design separates assessment of the task manager from end-to-end control performance.
- Embodied reasoning and trajectory prediction: LoHo-Manip achieves state-of-the-art results on RoboVQA and EgoPlan-Bench2, outperforming proprietary, open-source, and embodied foundation-model baselines.RoboVQA is evaluated with BLEU, while EgoPlan-Bench2 uses Accuracy (%).
- Embodied agent planning: The method achieves leading performance on both EB-Alfred and EB-Habitat, especially for long-horizon instruction interpretation and subtask planning.EmbodiedBench evaluates instruction understanding, commonsense reasoning, and long-horizon planning through executable atomic commands.
- VLA in simulation: LoHo-Manip consistently outperforms the base VLA π0.5 on VLABench, including tasks requiring complex semantic reasoning and visual grounding.The evaluation spans simulation benchmarks designed to test long-horizon manipulation and generalization.
- VLA in real-world: In real-robot experiments, LoHo-Manip significantly outperforms π0.5 in OOD settings involving novel instructions and objects.Both methods use the same 100-sample fine-tuning dataset; the OOD tests include novel objects, spatial arrangements, and language combinations.
4 Related Work
Related work addresses long-horizon manipulation through planning, memory, intermediate guidance, and multimodal reasoning. LoHo-Manip contributes a complementary decomposition that separates long-horizon task management from short-horizon VLA control.
- Foundation-model planning: Prior work uses foundation models for grounded skill selection, embodied planning, language-state feedback, and structured world-model-based planning.These approaches instantiate high-level planning for open-ended instructions and embodied agents.
- Long-horizon VLA control: Generalist VLAs provide strong short-horizon control and transfer, but long-horizon performance remains limited by drift, partial observability, and execution errors.Recent methods therefore target long-horizon training, evaluation, and execution-time correction.
- Memory and decomposition: Other approaches integrate observations over time or add multi-scale memory, whereas LoHo-Manip separates task management from short-horizon control.This positions LoHo-Manip as a complementary alternative to scaling an end-to-end VLA or embedding all reasoning within it.
- Intermediate guidance: Intermediate representations such as visual-text prompts, code, value maps, waypoints, and trajectory traces connect semantic decisions to executable behavior.LoHo-Manip follows this line by using explicit traces as guidance between planning and control.
5 Conclusion
LoHo-Manip decouples high-level symbolic planning from low-level sensorimotor execution through joint subtask and visual-trace prediction. Across embodied QA, simulation, and real-world benchmarks, it improves success rates, generalization, and task-planning accuracy over monolithic VLA baselines.
- Conclusion: LoHo-Manip predicts subtask sequences and canonical visual traces, reformulating long-horizon reasoning as manageable short-horizon control problems.The hierarchical design separates symbolic task planning from sensorimotor execution.
- Conclusion: Experiments across embodied QA, simulation, and real-world benchmarks report improved success rates, generalization, and task-planning accuracy compared with monolithic VLA baselines.The architecture is presented as a scalable pathway for combining future VLM and robot-control advances.
A More Qualitative Results on VLABench
Qualitative VLABench results cover five tracks spanning in-domain learning, category transfer, commonsense reasoning, semantic instruction understanding, and texture robustness. The method generalizes across in-distribution and out-of-distribution scenes and tasks while producing coherent subtasks and actions.
- Evaluation tracks: VLABench evaluates five tracks: in-distribution, cross-category, commonsense, semantic-instruction, and unseen-texture robustness.Each track targets a distinct form of task learning or generalization.
- Overall qualitative results: LoHo-Manip generalizes well across both in-distribution and out-of-distribution VLABench scenes and tasks.The qualitative results indicate robustness across varied object categories, instructions, and visual conditions.
- Reasoning tracks: On commonsense and semantic-instruction tracks, the method shows strong reasoning and generalization for open-ended instructions.These tracks require identifying targets or actions from complex semantic or contextual information.
- Unseen-texture track: Under unseen background and table textures, the method remains robust while continuing to generate reasonable subtasks and actions.This track directly tests robustness to visual appearance changes.
B More Qualitative Results on Real Evaluation
Real-robot experiments show LoHo-Manip decomposing long-horizon instructions into executable subtasks and spatial traces that guide stable, recoverable manipulation across scenes and object configurations.
- Real-robot trials consistently completed multi-step manipulation sequences across different scenes and object configurations.The system maintained task progress while handling long-horizon instructions with top-down and wrist-mounted camera inputs.
- The task manager predicted each next subtask with a corresponding visual trace, which the VLA executor used to generate low-level robot actions.The traces served as an intermediate representation for targeting the correct interaction locations.
- Qualitative evaluations covered different long-horizon manipulation tasks and multiple VLABench tracks using predicted subtasks and trajectories.
C.1 Quality of Sub-Task and Trace Curation
Subtask and trace curation improves trajectory prediction, while the modular manager transfers across executor policies with low additional inference overhead.
- The same task manager can be reused with different VLA executor architectures because the spatial trace provides a general interface.The planning component remains unchanged while each executor is adapted to follow the trace interface.
- Curated subtask and trace data consistently improved DFD, HD, and RMSE trajectory prediction on ShareRobot-T and VABench-V.Table 6 reports gains across all three metrics on both benchmarks.
- Integrating the manager with StarVLA improved performance across all VLABench evaluation tracks.The manager supplied StarVLA with predicted subtasks and visual traces as execution guidance.
- The planner runs once every 100 low-level executor steps, keeping additional inference overhead minimal and overall latency comparable to executor-only inference.This hierarchical schedule preserves intermittent high-level reasoning alongside continuous low-level control.
D Details for Sub-Task and Trace Extraction
Trace supervision is extracted from robot demonstrations by localizing the Franka end-effector in each video frame and converting those locations into normalized 2D coordinates.
- A VLM extracts the robot end-effector trajectory from demonstration videos as normalized 2D pixel coordinates.The process queries each frame to localize the Franka gripper and wrist assembly while excluding upstream arm links.
E Limitations
The framework remains limited by task-manager grounding, the expressiveness of 2D traces for complex contact interactions, and evaluation focused mainly on tabletop single-arm manipulation.
- The approach depends on accurate subtask grounding and trace prediction, leaving perception and spatial-grounding improvements as important stability targets.
- A 2D trajectory may not fully express highly precise or contact-rich interaction patterns.Richer spatial or temporal guidance could extend the representation’s applicability.
- Experiments primarily cover tabletop manipulation with a single robotic arm, so broader embodiments, tasks, and dynamic environments remain to be evaluated.