See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation

• SPR decomposes tasks into concrete 2D spatial subgoals, enabling verifiable progress tracking without auxiliary models.Each waypoint provides perceptual anchoring, trajectory-planning guidance, and an actionable completion target.Source: §1 · p. 2
• SPR unifies Seeing, Planning, and Rewinding in a closed loop that generates next-subgoal trajectories and resumes execution after detected failures.The framework uses explicit progress awareness rather than abstract linguistic signals or separate recovery models.Source: Figure 1 · p. 1, §1 · p. 2
• SPR surpasses MolmoAct by 3.8% on LIBERO and shows an 18.8% average performance drop on LIBERO-Plus versus 27.0% for OpenVLA-OFT and 37.5% for UniVLA.The evaluation covers unseen variation types and also includes real-robot manipulation tasks.Source: §1 · p. 2
• On challenging real-robot tidying and pushing tasks, baseline methods fail entirely while SPR maintains consistent performance.Source: §1 · p. 2
• Progress monitoring becomes an executable recovery policy that detects anomalies and restores the robot to in-distribution states.Repeated failed grasps and stagnant trajectories trigger Rewind to recover from unproductive execution states.Source: §1 · p. 2

• Existing VLA systems extend pretrained multimodal models to robot control, while hierarchical and spatial approaches improve long-horizon task reasoning.Examples include RT-2, OpenVLA, π0, π0.5, ECOT, MolmoAct, and SeqVLA.Source: §2 · p. 2
• Many progress-monitoring methods rely on prompt heuristics or coarse signals that lack explicit grounding in dynamic physical task states.This limits reliability under visual ambiguity, occlusion, or long-horizon execution.Source: §2 · pp. 2–3
• Failure-recovery methods use human supervision, language-model reasoning, or targeted failure data, but these approaches face scalability, adaptability, or data-collection costs.The related work motivates recovery behaviors synthesized from successful demonstrations instead of extensive failure-specific data.Source: §2 · p. 3

• SPR sees the current state and remaining subtask, plans a trajectory to the next 2D waypoint, and rewinds when anomalies require a recoverable state.Its data-generation pipeline constructs supervision for subtask boundaries, spatial coordinates, and rewind trajectories without additional human annotation or auxiliary models.Source: §3 · p. 3

3.1 Fine-Grained Spatial Subtask Planning

• Fine-grained spatial subtask planning decomposes manipulation into intermediate milestones and trajectories toward concrete waypoints.The design mirrors continuous human-like monitoring of progress during manipulation.Source: §3.1 · p. 3

3.1.1 Spatial Subtask See and Plan

• The See phase predicts remaining subtask count and semantic 2D subtask coordinates, while Plan generates up to five waypoints toward the next subtask goal.Next-subtask planning is intended to improve guidance when final-goal planning is misleading in long-horizon tasks.Source: §3.1.1 · p. 3

3.1.2 Action Reasoning with Subtask Awareness

• SPR factorizes generation into depth tokens, remaining count, subtask specifications, trajectory tokens, and action tokens conditioned on the observation and instruction.Source: §3.1.2 · p. 3
• The model represents the joint output distribution as p(a, d, n, s, τ|I, T) = p(d) · p(n) · p(s) · p(τ) · p(a).Each factor conditions on the observation, instruction, and previously generated autoregressive tokens.Source: §3.1.2 · p. 3

3.2 See-Plan Data Curation

• The pipeline extracts subtask boundaries, counts, and descriptions from demonstrations without additional human annotation or auxiliary vision-language models during training or inference.Pick-and-place boundaries come from gripper state transitions, while other manipulation tasks use annotated segment boundaries and semantic descriptions.Source: §3.2 · p. 3, §3.2.1 · p. 3

3.2.1 Subtasks Segmentation and Description.

• Subtask boundaries for pick-and-place tasks use gripper opening and closing, whereas other manipulation tasks use Gemini-3 annotations with segment frames and semantic descriptions.For LIBERO tasks, gripper-based segmentation is paired with DeepSeek-R1-generated segment descriptions.Source: §3.2.1 · p. 3

3.2.2 Gripper Trajectory Extraction.

• DINOv3 and SAM jointly localize the gripper, combining SAM’s horizontal precision with DINOv3’s vertical endpoint accuracy to obtain 2D coordinates.Coordinates are discretized to [0, 255], then trajectories are smoothed while preserving subtask boundaries.Source: §3.2.2 · p. 4
• The framework uses detected boundary-frame gripper positions as subtask waypoints and samples 1-5 intermediate positions for trajectory planning.Outlier interpolation and median filtering reduce abrupt trajectory deviations before waypoint extraction.Source: §3.2.2 · p. 4, Figure 2 · p. 4

3.3.1 Rewind Data Construction

• Reverse rewind trajectories are constructed by temporally reversing successful demonstrations, negating action tokens, and changing the instruction to return to the initial position.Subtask boundaries, waypoint coordinates, and trajectories are inherited from the existing data-curation pipeline.Source: §3.3.1 · p. 4

3.3.2 Progress-Based Anomaly Detection

• The state recorder tracks subtask counts over 4 timesteps and planned trajectories over 8 timesteps to identify regressions and stalled motion.Anomalies correspond to increasing subtask counts across consecutive windows or identical trajectories across all 8 recorded timesteps.Source: §3.3.2 · p. 4
• SPR achieves a 71.8% average success rate on LIBERO-Plus with the smallest average degradation of 18.8% across five perturbation types.The table evaluates robustness using success rates and degradation relative to the original LIBERO test sets.Source: Table 1 · p. 5
• The LIBERO results table compares separately trained and jointly trained SPR models across the benchmark’s training configurations.The supplied table passage specifies the configurations and ranking notation but does not provide the individual cell values.Source: Table 2 · p. 5
• Both anomaly criteria require persistence across multiple timesteps, filtering transient prediction noise before triggering recovery.Repeated identical plans can indicate collisions, misalignment, or other out-of-distribution configurations that prevent meaningful progress.Source: §3.3.2 · pp. 4–5

3.3.3 Rewind Execution Strategy

• Rewind substitutes the task instruction with a temporary return-to-initial-position command after anomaly detection, then resumes the original task from the new state.The empirically selected rewind duration is N = 3, balancing operational clearance against camera-view and pose risks.Source: §3.3 · p. 4, §3.3.3 · p. 5

4.2 Environment Setup

• The experiments use LIBERO, LIBERO-Plus, and three real-robot tasks to test standard performance, distribution shifts, long-horizon manipulation, and continuous-contact control.LIBERO-Plus includes unseen backgrounds, robot states, language, layouts, and lighting; real-robot evaluation uses 10 trials per task configuration.Source: §4.2 · pp. 6–7

4.3 Simulation Results

• 91.8% success on LIBERO, +5.0% over MolmoAct, with joint training also improving the challenging Long benchmark by 8.2%.Separately trained models reach 90.6%, +3.8% over MolmoAct; joint training achieves 91.8%, +5.0%.Source: §4.3 · p. 7
• SPR has the smallest overall performance drop on LIBERO-Plus and leads robustness under Language (-12.1%), Light (-5.6%), and Robot (-42.9%) shifts.The results are attributed to geometry-grounded planning, semantics-based progress awareness, and rewind-based recovery.Source: §4.3 · p. 7

4.4 Real-Robot Results

• SPR outperforms MolmoAct on all three real-robot tasks: 70% versus 50% for Pick up the Object, 30% versus 0% for Tidy up the Table, and 40% versus 0% for Push-T.The tasks cover single-object grasping, long-horizon multi-object tidying, and continuous-contact pushing.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 Ablation Study

• See and Plan alone improve LIBERO performance by 4.0% over MolmoAct, while adding Rewind yields a further 1% overall gain and stronger Language and Robot robustness.Spatial coordinates and semantic descriptions are complementary: coordinates provide precise grounding, while semantics support higher-level task understanding.Source: §4.5 · p. 8, Table 6 · p. 9
• With longer episode limits, SPR variants continue improving after MolmoAct plateaus, and full SPR completes tasks faster than the no-Rewind model.The Rewind mechanism reduces the time needed to re-execute failed subtasks, especially on LIBERO-Long and LIBERO-Goal.Source: Figure 4 · p. 7, §4.5 · p. 9

4.6 Visualization

• SPR maintains finer-grained subtask decomposition and recovery across sequential manipulation, layout shifts, relative positioning, failed grasps, perturbations, and poor initial poses.Its monitoring detects unchanged subtask progress, relocalizes moved objects, updates spatial coordinates, and rewinds to enable retries.Source: Figure 9 · p. 9, Figure 8 · p. 9, §4.6 · p. 10

4.7 Failure Analysis

• Failure analysis identifies three remaining limits: discrete action tokens reduce placement precision, physical constraints can defeat Rewind, and executed actions may diverge from correct planned waypoints.These failures affect careful placement, physically stuck robots, and cases where spatial planning succeeds but action execution remains misaligned.Source: §4.7 · pp. 10–11

• 5% higher performance than MolmoAct on LIBERO demonstrates SPR’s benchmark advantage.Source: §5 · p. 11
• SPR achieves state-of-the-art robustness on LIBERO-Plus, with the smallest average performance degradation of –18.8% under unseen instructions and initial-state perturbations.Source: §5 · p. 11
• The framework’s future scope includes diverse simulations, real-world scenarios, and noisy or suboptimal training demonstrations.Source: §5 · p. 11