Source-linked AI summary
Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation
Vivek Chavan, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger
TL;DR
Long-horizon VLA manipulation remains challenged by persistent state, dependencies, conditional decisions, and grounding. This paper combines task graphs, procedural memory, continuous perception, and demonstration-derived saliency, then studies prompt and regularization interfaces in an initial grounding evaluation. The results support complementary roles for symbolic structure and visual guidance, while broader end-to-end and generalization evaluation remains open.
Problem
Long-horizon VLA procedures require persistent progress, valid dependencies, conditional branches, and reliable object and destination grounding.
Method
The paper combines task graphs, event memory, continuous fixed-base and wrist-camera perception, a fine-tuned VLA, and pseudo-gaze procedural saliency.
Results
Both guided variants complete every initial instrument-handling trial, while removing the expected cue reduces ambiguous container routing from 10/10 to 5/10.
Takeaways & Limitations
The initial evidence positions symbolic procedural structure and demonstration-derived visual guidance as complementary mechanisms for long-horizon VLA manipulation.
Takeaways & Limitations
The broader pipeline remains unevaluated end to end, and the current study does not establish saliency transfer to unseen coordinates or wrist coordinates.
Abstract
from arXiv · showhide
Vision-language-action (VLA) models can execute short manipulation skills, but remain brittle in long-horizon procedures requiring persistent task state, dependency-aware reasoning, conditional decisions, and reliable grounding. We investigate a neuro-symbolic framework that combines learned VLA control with explicit task graphs and multimodal procedural memory. Task graphs encode action dependencies, valid transitions, and branch conditions, while memory maintains the active step, completed actions, textual context, and task-relevant visual evidence. Together, these structures guide object selection, destination grounding, subgoal dispatch, and verification of expected state transitions. Human demonstrations provide additional spatial and temporal guidance through gaze or saliency cues. To isolate their effect on policy learning, our initial study bypasses cross-view gaze transfer and directly annotates pseudo-gaze in robot-view teleoperation videos. The resulting guidance is used during VLA fine-tuning and inference. We study two long-horizon manipulation domains, workspace clearing and surgical-instrument handling, which require ordered execution, visually grounded decisions, and conditional branching. We evaluate correct-object and destination selection, subtask completion, task progress, step-order consistency, complete-task success, and procedural or execution mistakes. This work positions structured symbolic reasoning and demonstration-derived visual guidance as complementary mechanisms for reliable long-horizon VLA manipulation.
1 Introduction
Long-horizon VLA manipulation requires persistent state, dependency-aware decisions, conditional branches, and reliable grounding. The paper combines task graphs, event memory, continuous perception, fine-tuned control, and procedural saliency to address these requirements.
- Long procedures require persistent progress, valid dependencies, conditional branches, and reliable object and destination grounding.
- The pipeline uses task graphs and event memory for high-level decisions, continuous perception for transition verification, and a fine-tuned VLA for control.
- The observe–ground–remember–choose–execute–verify loop links procedural reasoning with robot action and state verification.
- The architecture connects graph-and-memory reasoning, multiple executor roles, and a shared saliency interface with explicit and internalized realizations.
2 Neuro-Symbolic Procedural Pipeline
The procedural pipeline separates what action is valid next from how the VLA executes it, using graph structure, event memory, multimodal perception, and sparse procedural saliency.
- Task Structure, Memory, and Collaboration: A directed task graph encodes instructions, executor roles, visual conditions, status, dependencies, preconditions, and valid transitions.
- Task Structure, Memory, and Collaboration: Append-only event memory records decisions, observed transitions, and visual evidence from fixed global and moving wrist views.
- Task Structure, Memory, and Collaboration: Verified collaborator actions become external events that update memory and can release subsequent robot nodes.
- Task Structure, Memory, and Collaboration: The graph determines the next valid action, while the VLA executes the dispatched subgoal and perception links execution to symbolic state.
- Sparse Procedural Saliency: Pseudo-gaze annotations use visual correspondence and optical-flow stabilization before serving as RGB prompts or attention-regularization targets.
- Sparse Procedural Saliency: At inference, the regularized policy receives clean images, but the study does not establish redirection to unseen coordinates.
3 Initial Grounding Study
The initial real-robot study fine-tunes π0.5 on teleoperated instrument-handling episodes and compares prompt-based and regularized saliency guidance across spatially ambiguous subgoals.
- Study Design: 231 teleoperated episodes support fine-tuning π0.5 for a seven-step instrument-handling procedure observed with fixed-base and wrist cameras.
- Study Design: 75 trials evaluate five subgoals under prompt-finetuned, visual-prompt, and clean-input regularized conditions.
- Results: Both guided variants complete every trial, while removing the expected cue reduces ambiguous container routing from 10/10 to 5/10.
- Results: The five routing failures without the expected cue are wrong-destination placements, while prompting and regularization tie behaviorally at the current ceiling.
4 Discussion and Outlook
The discussion distinguishes explicit prompting from internalized regularization while identifying unresolved questions about generalization, view transfer, and end-to-end procedural evaluation.
- Discussion: Prompting retains a visible deployment cue, whereas regularization internalizes supervision and removes the overlay.
- Outlook: The current saturated score makes generalization the next question, including reversed geometries, independently moved objects and containers, held-out scenes, and camera perturbations.
- Outlook: The broader pipeline remains unevaluated end to end, including graph-valid ordering, human-dependency recognition, transition verification, recovery, and full-task success.