PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicted Futures Are Not Enough: Learning Executable Goals for Robot Manipulation

Tzu-Yu Chuang, Ching-Hsiang Chang, Yi-Hsiu Lee, Yi-Ting Chen, Min Sun, YuanFu Yang

arXiv:2610.09309v1cs.RO

TL;DR

Future predictions do not by themselves expose the compact task variables needed for control, and prediction training alone does not directly optimize terminal-goal accuracy. The paper learns an explicit goal readout from a 3D trace world model and pairs it with a fixed-goal executor using online pose feedback. The pipeline reaches 79.69% mean success across five tasks, while the authors identify limits of single-pose goals and one-time planning.

  • Problem

    Rich future predictions do not directly expose compact control variables, and future-prediction supervision alone does not directly optimize terminal-goal accuracy.

    Source: §I · p. 1

  • Method

    Entity-Level Goal Readout combines object-centric pose prediction with depth-grounded position to expose an executable SE(3) goal, which a shared executor uses with online pose feedback.

    Source: §I · p. 1, [1] INPUT · p. 2

  • Results

    79.69% mean success for the final pipeline, compared with 54.45% for K = 1 and 59.87% for K = 4 under shared executor weights.

    Source: B. Closed-Loop Execution · p. 5

  • Takeaways & Limitations

    The reported results support treating the prediction-to-execution interface as an explicit learned component rather than incidental control-pipeline post-processing.

    Source: §VI · pp. 7–8

    A single terminal pose does not encode trajectory constraints, contact sequences, or articulated and deformable-object states; planning once per episode cannot revise an invalidated goal.

    Source: §V · p. 7

Abstract

from arXiv · show

Generative world models provide rich predictions of how manipulation scenes may evolve toward task objectives, yet those futures do not directly expose the compact task variables required by control. When training supervises future prediction alone, terminal goal accuracy is not an explicit learning objective, even when geometric recovery is available. We present Entity-Level Goal Readout, a learned prediction-to-execution interface that makes the executable terminal goal an explicit output of a 3D trace world model. It combines object-centric pose prediction with translation grounded in observed depth to produce a compact goal in SE(3). A shared Pose-Native Executor consumes this fixed goal with online object-pose feedback for closed-loop control without rerunning the world model. Across five manipulation tasks, the pipeline achieves a mean success rate of 79.69%. Goal diagnostics directly measure terminal goal accuracy, while controlled translation perturbations characterize how execution degrades under goal error. Zero-shot deployment on a Franka arm achieves 73.33% success on nominal StackCube, 66.67% with distractors, and 75.00% on PickPlate with a target unseen during policy training. These results support treating the prediction-to-execution interface as an explicit learned component of world-model planning rather than incidental post-processing in the control pipeline itself. Project page: https://claire0730.github.io/executable-goals/

I. INTRODUCTION

Future predictions can be rich without exposing the compact task variables control needs, and converged prediction objectives do not guarantee accurate goals. The paper makes terminal goals explicit learned outputs and connects them to closed-loop manipulation.

I. INTRODUCTION

  • Future prediction alone does not optimize recovered terminal-goal accuracy, so prediction errors can still produce inaccurate control targets.The introduction frames the question as whether control-consumed task variables are explicitly learned and exposed.Source: §I · p. 1

[1] INPUT

  • Entity-Level Goal Readout exposes an executable object-relative goal from a 3D trace world model for a Pose-Native Executor.The StackCube example illustrates the handoff from predicted futures to closed-loop manipulation.Source: Figure 1 · p. 1, [1] INPUT · p. 1
  • A shared executor uses pose feedback with the fixed goal to correct execution without rerunning the world model.This separates episodic planning from continuous control.Source: [1] INPUT · p. 2
  • The evaluation measures goal accuracy, manipulation success, and sensitivity to goal error across five tasks, alongside real-robot trials.The trials include distractors and a target unseen during policy training.Source: [1] INPUT · p. 2
  • The readout combines object-centric pose prediction with translation grounded in observed depth to produce a compact executable goal.This makes control-relevant terminal variables explicit learning targets.Source: [1] INPUT · p. 2
  • The shared Pose-Native Executor is trained under measured planner goal errors using privileged teachers, but receives no privileged guidance during execution.The paper presents this as execution training for imperfect goals.Source: [1] INPUT · p. 2

II. RELATED WORK

Prior work spans world models that generate different forms of future motion, geometric methods that recover control targets, and goal-conditioned policies. This paper focuses on how predicted futures become task goals consumed by control.

Generative World Models

  • World-model planners represent future motion as generated observations, point trajectories, or scene geometry, with methods predicting 2D or 3D motion and geometry.Dream2Flow [4] and TesserAct recover motion or object goals from generated observations; Track2Act recovers geometric poses from 2D point trajectories.Source: §II · p. 2

From Predicted Futures to Executable Goals

  • Geometric readouts recover control targets through pose estimation or rigid registration, but future-prediction supervision alone does not directly optimize terminal-goal accuracy.The paper makes terminal goal quality an explicit learning target and evaluates it in Table I.Source: §II · p. 2

Goal-Conditioned Control

  • Goal-conditioned policies map rich observations to actions, whereas the Pose-Native Executor receives one fixed SE(3) pose goal per episode.Its distillation strategy is motivated by dataset aggregation and privileged-teacher methods,,.Source: §II · p. 2

III. METHOD

The system predicts one object-relative SE(3) goal once per episode, then executes it with online pose feedback. The planner hands off a compact goal rather than the full predicted future.

A. Overview and Problem Statement

  • From a single RGB-D observation and task instruction, the world model predicts future scene motion and the interface explicitly predicts one executable goal relative to the initial object frame.Planning occurs once per episode; execution then runs independently at 20 Hz.Source: A. Overview and Problem Statement · p. 2
  • The executor maps current pose, fixed goal, proprioception, and previous action to seven arm-joint increments and a gripper target.Its command is eight-dimensional and produced at 20 Hz.Source: A. Overview and Problem Statement · p. 2, Figure 3 · p. 3
  • The readout exposes one object-relative goal anchored at t = 0 and fixed for the episode, while online tracking and pose estimation update object state.These updates close the loop during execution.Source: Figure 3 · p. 3
  • The planner communicates a compact seven-parameter SE(3) goal rather than the full predicted future.Task success is evaluated separately from goal accuracy.Source: A. Overview and Problem Statement · pp. 2–3

B. Trace Planner and Rigid Readout

The method compares geometric rigid fitting with a learned readout that predicts control-relevant quantities from the shared planner representation. The learned readout combines an entity-derived rotation with depth-grounded position to form the executable goal.

B. Trace Planner and Rigid Readout

  • TraceGen predicts sparse 3D point trajectories, and the task-adapted planner fine-tunes planner layers while keeping visual and language encoders frozen.Its decoder predicts trajectories for 400 query points over 32 future steps.Source: B. Trace Planner and Rigid Readout · p. 3
  • Rigid readout estimates a terminal transform from predicted object tracks using weighted Kabsch inside RANSAC, without directly training the predictor for terminal-goal accuracy.For K = 4, traces are averaged before rigid fitting.Source: B. Trace Planner and Rigid Readout · p. 3

C. Entity-Level Goal Readout

  • The entity branch pools motion-selected vision tokens and uses cross-attention to predict terminal object pose rotation; keyframe poses and contact quantities are auxiliary supervision only.Those auxiliary predictions are not passed to the executor.Source: C. Entity-Level Goal Readout · p. 3, Figure 4 · p. 4
  • The map branch predicts a spatial goal distribution, masks it by valid observed depth, and unprojects its depth-weighted centroid to obtain world-frame position.This readout also supports targets defined by regions without a unique rigid transform.Source: C. Entity-Level Goal Readout · p. 3, Figure 4 · p. 4
  • Goal composition combines map position and entity rotation in the world frame, then expresses the pose relative to the initial object frame.The readout is jointly trained with the trace objective, with losses directly supervising terminal pose, auxiliary interaction quantities, and the spatial goal distribution.Source: C. Entity-Level Goal Readout · pp. 3–4

D. Executor and Distillation

The executor keeps one predicted goal fixed while online object-pose feedback supports closed-loop correction. Its student policy is trained with dataset aggregation on states visited under imperfect goals, while privileged guidance is withheld from the student.

D. Executor and Distillation

  • The executor reconstructs a fixed world-frame goal from the initial pose and updates execution against online pose estimates without rerunning the world model.The executable goal remains fixed while feedback updates the current state.Source: D. Executor and Distillation · p. 4
  • Online perception tracks image correspondences and uses depth unprojection followed by Kabsch to recover the current object pose.The estimate is updated while the executable goal remains fixed.Source: D. Executor and Distillation · p. 4
  • At 20 Hz, the shared 0.8M-parameter policy maps pose, goal, robot state, and previous action to seven joint increments and a gripper target.The same student policy is used across all five tasks.Source: D. Executor and Distillation · p. 4
  • DAgger trains the student on visited states with privileged teacher labels, while goal perturbations sampled from planner errors expose it to imperfect goals.Privileged guidance is available to the teacher but not to the student during training or deployment.Source: D. Executor and Distillation · p. 4

IV. EXPERIMENTS

The evaluation measures terminal-goal quality, closed-loop success, and response to imposed goal error across five ManiSkill3 tasks, with real-robot trials extending evaluation beyond simulation. The protocols distinguish position-focused tasks, success-only tasks, and separate evaluation banks.

IV. EXPERIMENTS

  • Goal quality, shared-executor success, and controller sensitivity to imposed goal error are evaluated separately, and real-robot trials extend the evaluation beyond simulation.Archived rigid and Oracle results pool three seeds and 768 episodes per task and goal source; perturbation results use 512 episodes per point.Source: §IV · p. 4
  • Five ManiSkill3 tasks are evaluated, with position comparisons on PickCube, PegInsertionSide, and StackCube and success evaluations also including LiftPegUpright and PushCube.LiftPegUpright admits multiple upright positions, while PushCube accepts a broad target region.Source: §IV · p. 4
  • Terminal-goal position error is the 3D Euclidean distance to reference goal positions, reported in millimeters as mean ± standard deviation over valid readouts.Evaluation banks and validity conventions differ across the listed planner groups and stored-goal rows.Source: Table I · p. 5, §IV · p. 5
  • Table I groups planners by prediction representation, while its final three rows compare rigid recovery with learned goal readout on the same 3D point-trajectory representation.Table II separates native-executor evaluations from shared-executor comparisons; Oracle Goal rollouts use designated correct goals.Source: §IV · p. 5

A. Goal Quality Across Planners

Entity-Level Goal Readout has the lowest mean position error among the reported readouts sharing a 3D trace representation. The results also show that converged future-prediction losses do not ensure accurate recovered terminal goals.

A. Goal Quality Across Planners

  • Training losses converge, but recovered terminal goals retain substantial errors; training data and optimization budgets differ across the compared methods.The baseline adaptations use different training datasets and adaptation approaches.Source: A. Goal Quality Across Planners · p. 5
  • 15.3%, 27.9%, and 49.6% lower mean position error than rigid K = 4 are reported for Entity-Level Goal Readout on PickCube, PegInsertionSide, and StackCube, respectively.Within the shared 3D trace representation, it has the lowest mean position error on all three reported tasks.Source: A. Goal Quality Across Planners · p. 5
  • Structured-4D reconstructs geometry at roughly centimeter scale yet retains substantial PickCube and PegInsertionSide goal errors; adapted TesserAct also retains substantial errors on its reported tasks.Adapted TesserAct has lower position error on PegInsertionSide than on PickCube or StackCube.Source: A. Goal Quality Across Planners · p. 5

B. Closed-Loop Execution

Under shared executor weights, the final pipeline achieves higher mean success than both rigid readouts, while diagnostic results characterize goal errors and their relation to execution. The comparisons have scope limits, and the geometry and rotation probes do not establish rotation-mediated execution gains.

B. Closed-Loop Execution

  • Shared-executor comparisons support pipeline comparison, not isolated readout attribution; Oracle Goal measurements are empirical references rather than theoretical upper bounds.The comparisons use different checkpoints and samples, one policy-training seed, and noise matched to final-pipeline errors.Source: B. Closed-Loop Execution · p. 5
  • Additional trace averaging improves rigid readout on most tasks but does not close the mean success gap.The final pipeline's largest gains over K = 4 are on PickCube and StackCube.Source: B. Closed-Loop Execution · p. 5
  • Adapted Structured-4D reports 89.00% StackCube success versus 86.54%, using its native executor and weights trained with 1,000 demonstrations per task.Its 1.1B planner takes 4.45 s per plan versus 1.27 s for the 677.6M planner; the latter's executor is shared across five tasks.Source: B. Closed-Loop Execution · p. 5

C. Readout Diagnostics

  • The final pipeline reduces typical and tail position error on StackCube and shows a clearer tail reduction on PegInsertionSide.Fig. 5 shows logarithmic error distributions; Fig. 6 reports median and 90th-percentile position errors in millimeters.Source: C. Readout Diagnostics · p. 5, Figure 5 · p. 6, Figure 6 · p. 7
  • A single threshold does not fully characterize goal quality: the 25 mm StackCube proxy improves, while the 20 mm PegInsertionSide proxy changes little.Quantiles exclude nonfinite estimates, but execution denominators retain all scenes.Source: C. Readout Diagnostics · p. 5
  • PegInsertionSide raw quaternion errors are lower for the final entity-branch bank, but those errors are not symmetry adjusted and do not establish rotation-mediated execution gains.No learned-rotation gain is claimed for LiftPegUpright or StackCube because their task metrics require task-aware treatment.Source: C. Readout Diagnostics · pp. 5–6

B. Closed-Loop Execution

  • 79.69% mean success versus 59.87% for K = 4 and 54.45% for K = 1 is reported for the final pipeline under shared executor weights.It exceeds both rigid variants on four tasks; PushCube remains near saturation at 99.20%, slightly below both rigid variants.Source: B. Closed-Loop Execution · p. 5, Table II · p. 6

C. Readout Diagnostics

  • Removing geometry from the entity-position head increases error in the test-time probe, which is not a full training-loss ablation.The probe compares median and 90th-percentile errors; its segments connect quantiles, not confidence bounds.Source: C. Readout Diagnostics · p. 6, Figure 6 · p. 7

D. Position Sensitivity

The position-sensitivity intervention tests how a fixed executor responds to imposed translation error, separately from the goal errors delivered by a planner. Success declines as offsets grow, with a sharper StackCube drop between 30 and 50 mm.

D. Position Sensitivity

  • Offsets are sampled isotropically in the world frame, including vertical offsets, and stay fixed per episode across paired scenes and directions.The two tasks span 8,192 rollouts over eight settings; only each environment’s initial episode is scored.Source: D. Position Sensitivity · p. 7
  • Success declines with larger offsets on both tasks, with a sharper StackCube drop from 30 to 50 mm.The curves retain their measured zero-offset baselines rather than the pipeline Oracle reference.Source: D. Position Sensitivity · p. 7
  • Goal distributions measure planner-delivered errors, while this intervention measures the fixed executor’s response to imposed errors.Task success reflects both sources of error; error bars are pointwise 95% Wilson intervals, not paired-effect or policy-training-variability intervals.Source: D. Position Sensitivity · p. 7

E. Computational Cost and Real Robot Deployment

The deployment audit reports low executor inference time alongside Franka performance across three real-robot settings. PickPlate tests a target unseen during policy training, but offers a larger valid target region.

E. Computational Cost and Real Robot Deployment

  • The executor’s approximately 1.4 ms inference fits inside the 20 Hz loop, while planning runs only once per episode.The readout adds less than 0.5% to planner parameters.Source: E. Computational Cost and Real Robot… · p. 7
  • 73.33% success on nominal StackCube, 66.67% with distractors, and 75.00% on PickPlate were achieved in Franka trials.PickPlate was unseen during policy training and has a larger valid target region, so it provides evidence of target transfer rather than greater manipulation difficulty.Source: E. Computational Cost and Real Robot… · p. 7
  • Real-robot evaluation extends beyond nominal stacking to distractors and PickPlate.PickPlate places the cube in an orange plate and uses a target unseen during policy training.Source: E. Computational Cost and Real Robot… · p. 7

V. LIMITATIONS

The paper frames explicit terminal-goal learning as a prediction-to-execution interface, while noting that a single terminal pose and fixed episodic planning constrain what the system can represent and revise. The conclusion identifies richer goals, goal revision, and uncertainty-aware perception as directions for broader deployment.

V. LIMITATIONS

  • A single terminal pose omits trajectory constraints, contact sequences, and articulated or deformable-object states.Planning occurs once per episode, so pose feedback can correct execution but cannot revise an invalidated goal.Source: §V · p. 7
  • The system depends on reliable tracking, depth, and pose estimates, and new goal semantics may require additional task-relevant supervision.Source: §V · p. 7

VI. CONCLUSION

  • The conclusion presents Entity-Level Goal Readout as an interface that makes terminal task variables explicit learning targets through object-centric pose prediction and depth-grounded translation.The reported pipelines improve terminal goal quality and mean closed-loop success, while controlled perturbations show that positional goal error matters to a fixed executor.Source: §VI · p. 7
  • Richer goal representations, goal revision during execution, and uncertainty-aware perception remain directions for broader deployment.The conclusion states that direct goal prediction may suffice for visually observable objectives, so the supported claim is task-aligned readout rather than a necessity for future prediction.Source: §VI · p. 8

Related papers

Loading 2610.09309v1…