PaperlayerView exact revision on arXiv

Source-linked AI summary

Revision, Not Restart: Revisable Visual Plans for Closed-Loop World-Action Models

Pengyiang Liu, Junbo Niu, Wenhao Zheng, Xinchen Chen, Canyu Li, Zhongyue Shi, Jiahao Xie, Si Liu

arXiv:2609.35439v1cs.ROcs.CV

TL;DR

Execution feedback can invalidate parts of a predicted visual future while leaving useful task structure, so world–action models need a way to update their visual action condition. RTP revises saved generation states under current observations and uses the resulting plan to condition control. It achieves the highest observed task-averaged success among compared methods on both RoboMME and RMBench.

  • Problem

    Execution feedback can disagree with a predicted future even when its remaining task structure is useful, raising the question of how to revise that visual condition for the next action.

    Source: §1 · p. 2

  • Method

    RTP maintains a persistent visual action condition and learns to adapt saved intermediate generation states to feedback using visual and action supervision.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    RTP has the highest observed task-averaged success among compared methods on RoboMME and RMBench, improving over fresh replanning by 8.2 and 4.9 percentage points, respectively.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The results connect feedback-driven visual-plan revision to closed-loop task performance, with learned-residual comparisons providing the clearest component evidence.

    Source: §5 · p. 9

    Because predicted-future errors can influence executed actions, downstream applications require task-specific safety evaluation, operational safeguards, and human oversight suited to deployment.

    Source: Ethics Statement · p. 10

Abstract

from arXiv · show

World-action models use predicted visual futures to condition robot actions, yet execution feedback can invalidate parts of a prediction while leaving its task structure useful. We propose Revisable Temporal Planning (RTP), which maintains the visual future as a persistent action condition and revises it after feedback. Its central mechanism is a learned revision bridge: it resumes an intermediate state saved during visual generation and adapts its continuation to current observations. Visual and action supervision connect this revision to subsequent control. Time-aware history supplies observed evidence, and an adaptive policy selects retention, bridge revision, or fresh replanning from new noise before decoding the next action. On RoboMME and RMBench, RTP achieves task-averaged success rates of 48.6% and 84.8%, respectively. Matched comparisons support learned continuation; estimated checkpoint-source and action-prefix effects are positive but less precisely resolved. These results connect feedback-driven visual-plan revision to closed-loop task performance. Project Page: https://PLACEHOLDER.github.io/RTP/

1 Introduction

Execution feedback can invalidate a predicted visual future while leaving its task structure useful, making reuse unchanged or generation from scratch imperfect responses. RTP keeps the future as a persistent action condition and learns to revise it under feedback.

1 Introduction

  • Execution can shift an event such as contact while leaving the planned approach and subsequent manipulation useful.Reusing the prediction unchanged can preserve timing errors, while restarting from noise repeats visual generation.Source: §1 · p. 2
  • The motivating question is how a world–action model can revise an already generated visual future under feedback and use it to guide the next action.Prior work addresses persistent observation memory, efficient representations, feedback-aware execution, and diffusion-plan repair or continuation.Source: §1 · p. 2
  • RTP’s central insight is to maintain the visual future as a persistent action condition that evolves with execution feedback.Saved intermediate denoising states are adapted to current observations through visual and action supervision so the revised future can guide the next action.Source: §1 · p. 2
  • The method combines time-aware observed history, a learned revision bridge, and adaptive selection among retention, revision, and fresh replanning.The accepted plan and current observations condition the next action; RTP is evaluated on RoboMME and RMBench using task-adapted LingBot-VA.Source: §1 · p. 2
  • The stated contributions include closed-loop comparisons of learned correction, checkpoint source, and action conditioning, plus success–cost comparisons of complete update policies.The paper also identifies a persistent visual action condition and a separate time-aware record of observed history.Source: §1 · p. 2

2 Related Work

Related work covers visual futures for world–action models, feedback-driven revision of generated plans, and memory for long-horizon video and robotics. RTP distinguishes itself by revising a visual condition under feedback while retaining its generation state and observed history.

2.1 World–Action Models and Future Conditioning

  • World–action models use predicted visual futures to condition actions, while RTP targets how that visual condition changes after feedback.Related approaches also develop visual/action representations, efficient generation, and asynchronous execution.Source: §2.1 · p. 2

2.2 Feedback Verification and Generative-Plan Revision

  • Prior plan-revision methods include endpoint re-noising, selective re-noising, likelihood-based retention or repair, and streaming updates to partially denoised action buffers.RTP resumes intermediate states saved during generation of an existing visual plan, learns their continuation under new observations, and uses the revised prefix to condition action.Source: §2.2 · p. 3

2.3 Long-Horizon World Memory

  • Long-horizon memory work maintains scene evidence or robotic context, whereas RTP keeps observed history and a separately revised predicted future for control.Its records provide multiscale evidence addressed by environment time.Source: §2.3 · p. 3

3 Method

RTP maintains observed history separately from an unexecuted visual plan, updates both at feedback boundaries, and decodes each next action from the accepted plan and current facts. Its training proceeds from time-aware model adaptation to feedback-conditioned revision and discrepancy-estimator fitting.

3 Method

  • Training adapts LingBot-VA to time-aware history, learns feedback-conditioned revision with the adapted model frozen, then fits a discrepancy estimator with preceding modules fixed.At each feedback boundary, RTP updates history, selects a plan update, and decodes an action under current observations.Source: §3 · p. 3

3.1 Time-Aware History and Persistent Plans

  • The history sampler retains original environment timestamps, combining recent detail with older evidence under a fixed group budget.It reserves a reset anchor and recent-group quota, then adds pyramid candidates and fills remaining capacity with the newest omitted groups.Source: §3.1 · p. 3
  • A persistent plan record stores the clean prediction, intermediate checkpoints, root origin, consumed index, and generation provenance.After each executed action block, consumed groups advance and new facts arrive; reuse requires enough unconsumed groups for the next action.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Feedback-Conditioned Plan Revision

  • The revision bridge resumes a saved generation state under new observations and adds a learned correction to the frozen visual velocity field.Bridge-5 resumes at x15 for five intervals and Bridge-10 at x10 for ten, within a 20-interval generation.Source: §3.2 · p. 5
  • Bridge training fits the feedback encoder and velocity correction to observed continuations and behavior actions, with fresh-generation consistency regularization.Archived fresh roots are paired with later feedback after one, two, or three consumed groups; the adapted model remains fixed.Source: §3.2 · p. 5

3.3 Adaptive Updates and Action Generation

  • The selector evaluates legal retention and bridge modes, choosing the first passing option in the priority order retention, Bridge-5, then Bridge-10.If none passes, it triggers fresh replanning; every option uses current facts to decode a new action.Source: §3.3 · pp. 5–6
  • The recurrent update loop appends observed and applied records, rebuilds facts, runs the selected visual update, saves the accepted record, and decodes from its next prefix.Fresh replanning establishes a new root, while reuse keeps the existing root.Source: §3.3 · p. 6

4 Experiments

RTP has the highest observed task-averaged success on both benchmarks, and matched comparisons examine plan updating, history, learned correction, computation, and controlled feedback mismatch.

4.1 Experimental Setup

  • Evaluation covers 16 RoboMME tasks and nine RMBench tasks, with equal task weights and paired intervals resampled over reset keys.Within-run update policies share adapted weights, history, normalization, and action interfaces; intervals characterize reset variation conditional on the fitted model, not independent training runs.Source: §4.1 · pp. 6–7

4.2 Main Results

  • RTP has the highest observed task-averaged success on RoboMME and RMBench, exceeding fresh replanning by 8.2 and 4.9 percentage points, respectively.Task-level comparisons include a tie and a regression on RoboMME, and several intervals include zero.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 7

4.3 Ablation Studies

  • Learned correction improves success over zero-residual Fixed bridge-10 by 5.8 points; checkpoint-source and action-prefix effects are less precisely resolved.Fresh-20+C gains 3.1 points over Fresh-20, and RTP gains a further 5.1 points; the prefix effect under current facts is 2.5 points with interval [0.0, 5.0]. The nominal saved-checkpoint advantage remains imprecise, and RTP improves both evaluated history configurations, with a larger gain under time-aware history.Source: §4.3 · p. 8, Table 5 · p. 8, Table 4 · p. 8

4.4 Computation and Recurrent Behavior

  • RTP averages 7.78 visual intervals per noninitial call versus 12.33 for Fixed bridge-10, a 36.9% reduction, while recurrent fitting lowers discrepancy without a resolved success gain.Mean full-call times are 1.051 and 1.115 seconds, respectively; P95 times are 1.294 versus 1.268 seconds.Source: §4.4 · pp. 8–9, Table 3 · p. 8, Figure 4 · p. 9

4.5 Controlled Feedback Mismatch

  • Under actuator holds, an eight-sample hold causes small success decreases; the saved-checkpoint advantage over root-noise reconstruction is 2.8 points nominally and 2.9 under the longer hold.This setting probes recovery from an early interruption with substantial time remaining for feedback correction; further activation-delay and target-shift tests keep nominal parameters frozen.Source: §4.5 · p. 9, Table 6 · p. 9

5 Conclusion

RTP revises saved visual-generation states under execution feedback while retaining the visual future as an action condition. Component evidence most clearly supports learned residual correction; checkpoint-source and action-prefix effects are less precisely resolved.

5 Conclusion

  • RTP maintains a visual future as a persistent action condition and revises saved generation states under execution feedback.The learned-residual comparison provides the clearest component evidence; checkpoint-source and prefix effects are less precisely resolved.Source: §5 · p. 9
  • Removing action supervision improves visual error while lowering success, separating prediction accuracy from control utility.Recurrent fitting improves discrepancy control without establishing a task-success gain over the matched refit.Source: §5 · p. 9
  • Evaluation uses LingBot-VA, finite prediction windows, synchronous execution, and controlled delays; longer interactions are identified as a direction for future work.Source: §5 · p. 9

AI Use Statement

The authors report generative-AI assistance in language editing and literature identification, while taking responsibility for the final manuscript.

AI Use Statement

Reproducibility Statement

The paper describes where implementation and evaluation details appear and states that code and model checkpoints will be released.

Reproducibility Statement

Ethics Statement

The ethics statement calls for license-aware resource use and context-appropriate safeguards; A.1 specifies how bounded, timestamped history and reference context are selected.

Ethics Statement

  • Deployment guidance calls for task-specific safety evaluation, operational safeguards, and human oversight suited to the context.The work uses existing benchmarks and pretrained models, whose licenses and access conditions should be respected.Source: Ethics Statement · p. 10

A.1 Temporal Foundation Configuration

  • History groups package causal latents, preceding actions, proprioception, and original sample endpoints; RoboMME groups represent 4 native samples and RMBench groups 16.The reset image is group zero, and camera identity and spatial positions remain distinct from environment time.Source: §A.1 · p. 14
  • History is capped at 60 groups for RoboMME and 72 for RMBench, with recent groups and fixed-phase pyramid samples prioritized.The selector preserves a reset anchor and sorts selected records by physical endpoint; a short record uses every available group.Source: §A.1 · p. 14, Table 7 · p. 14
  • RoboMME reference clips are separate context, sampled into at most 32 groups; RMBench uses no visual reference groups.Tasks without a RoboMME reference also use zero reference groups, and language instructions are budgeted separately.Source: §A.1 · p. 14
  • Training targets the next four contiguous groups, while deployment uses the specified pyramid selector and factual and query positions are rebuilt for each consumer.The causal observation encoder does not read predicted frames, and generated consumed positions are not reclassified as evidence.Source: §A.1 · pp. 14–15

A.2 Data, Supervision and Fitting

A.2 describes adaptation and bridge fitting on separated trajectory data; A.3 defines the discrepancy and correction modules, and A.4 specifies checkpoint restoration and state invariants.

A.2 Data, Supervision and Fitting

  • Stage A adapts the video–action model with frozen pretrained observation and language encoders, using clean future groups to teacher-condition aligned action samples.The action branch uses the generated prefix at deployment, preserving the pretrained video–action interface.Source: §A.2 · p. 15
  • Bridge tuples are split by trajectory into fitting and calibration data; fitting samples 800 tuples per task, while calibration uses 4,096 tuples per benchmark.Fresh-parent trajectories are collected on disjoint reset keys, and calibration tuples estimate no bridge or backbone parameters.Source: §A.2 · pp. 15–16

A.3 Distances and Trainable Modules

  • The distance and supervision setup compares clean latent predictions and continuous denormalized actions, with action loss applied before clipping or gripper conversion.Stage-B scales come only from fitting targets, are floored at 10^-6, and are then frozen.Source: §A.3 · p. 16
  • DiffEncode combines prediction, observation, their difference, applied controls, proprioception, and task summaries to condition a learned visual velocity correction.The correction is applied to the visual solver, while the action field itself has no trainable RTP residual.Source: §A.3 · pp. 16–17
  • The estimator uses saved-state statistics, including source age measured from the checkpoint’s immutable creation boundary, to estimate discrepancy and its error.Disjoint calibration data sets fresh-relative discrepancy tolerances, while later recurrent states are evaluated separately.Source: §A.3 · p. 17

A.4 Native Windows and Checkpoints

  • Checkpoint restoration rebuilds the factual interface under current observations, resumes from saved solver states, and preserves checkpoint provenance and source versions.Retain preserves the plan; a bridge keeps its starting checkpoint and adds newly visited checkpoints. Fresh resets the consumed index, and an empty legal set falls back to fresh.Source: §A.4 · pp. 17–18

B.1 Execution and Statistical Units

The experiments define paired reset-key evaluation and compare matched fresh, reconstructed, action-input, and policy-selection controls. The reported pooled RMBench difference is 4.889 percentage points.

B.1 Execution and Statistical Units

  • Evaluation alternates complete controller calls with execution, pairs policies on reset keys, and reports bootstrap intervals conditional on fitted weights.Repeated seeds are averaged within each key; comparisons without paired intervals are descriptive, and unreported intervals establish neither significance nor equivalence.Source: §B.1 · p. 18
  • 4.889 percentage points is the reported pooled RMBench difference.The per-task RMBench comparison uses 100 paired keys per task; the table caption does not provide cell values here.Source: Table 9 · p. 19, §B.1 · p. 19

B.2 Matched Fresh and Reconstructed Controllers

  • Fresh-k+C matches the bridge’s residual architecture, supervision, and optimizer updates but starts from new Gaussian noise rather than a saved continuation state.The matching does not equalize differentiable visual intervals; reconstructed-state comparisons test native versus reconstructed starts at matched continuation depth.Source: §B.2 · pp. 20–21, Table 11 · p. 20

C.1 Action Inputs

  • The action-input factorial cells vary whether the decoder reads previous or current facts and a retained or revised prefix, while all update the visual record with Bridge-10.The action noise is common across interventions, and subsequent actions and feedback are policy-specific.Source: §C.1 · p. 21

C.2 Selection Controls

  • The evaluated selection set is retention, Bridge-5, Bridge-10, and fresh-20; fixed-depth and binary controls are compared with adaptive updates.The candidate budgets are interface-defined, not asserted to be optimal; policy comparisons include exhaustion refreshes and each policy’s stopping time.Source: §C.2 · p. 21, Table 12 · p. 21

C.3 Recurrent Fitting and Diagnostics

The section evaluates recurrent fitting and diagnostic measures, then specifies probes and intervention settings used to assess visual-plan updates across feedback conditions.

C.3 Recurrent Fitting and Diagnostics

  • Recurrent fitting uses accepted records from a frozen default policy and matched feedback-boundary quotas, with fresh behavior continuations supplying visual and action targets.The matched parent refit uses identical tuple counts, quotas, epochs, and optimizer settings; the age/length ablation removes both lifecycle coordinates.Source: §C.3 · pp. 21–22
  • -1.04 percentage points is the nominal recurrent-fitting change in tolerance-exceedance rate, while recurrent-minus-matched success is 0.125 percentage points and does not establish a task-success gain.The exceedance comparison is conditional on each policy’s selected reuse; whole-policy outcomes use every reset.Source: §C.3 · p. 22, Table 14 · p. 22

D.1 Common-Reference and Own-Action Probes

  • Common-reference and own-action probes use disjoint source pools; own-action candidates execute separate action-conditioned continuations and are evaluated against their own observations.The probes compare endpoints 16, 20, and 24, using contact-time error, event-order accuracy, and latent error for own-action metrics.Source: §D.1 · p. 23, Table 16 · p. 23

D.2 Eligibility, Annotation and Losses

  • Eligibility rules preserve primary resets but restrict complete-window metrics to common eligible cases; blinded annotators label target contact and task-script event order.Fixed bridge-10 loss controls isolate learned correction or loss terms, while success and visual error remain separate endpoints.Source: §D.2 · p. 24, Table 17 · p. 24, Table 18 · p. 24

E.1 Holds and Source Comparisons

  • Actuator holds begin at sample 8, while source controls share operation depth but can achieve different refresh fractions when episode lengths differ.Cost accounting uses each policy’s own counts and times, so equal depth does not imply equal whole-policy cost.Source: §E.1 · pp. 24–25, Table 19 · p. 25, Table 20 · p. 25

E.2 Activation Delay

  • Activation-delay comparisons impose the same capture delay across policies and test delays of 0, 1, or 2 samples; delays of 4 or more miss the action block.Intervening observations are inaccessible to the in-flight call, and the original prediction is compared at the shared endpoint.Source: §E.2 · p. 25, Table 22 · p. 25

E.3 Spatial Shift and Temporal Configuration

  • Spatial-shift tests translate the free target by 2 cm when clearance permits, while horizon and sample-spacing settings refit each comparator under the changed interface.The H=6, J=4 setting admits five reusable updates; H=4, J=8 changes sample spacing and action-head alignment.Source: §E.3 · p. 26, Table 23 · p. 26, Table 24 · p. 26

F Controller and Resource Accounting

This section accounts for controller-call timing, episode cost, solver work, and model storage and parameter counts.

F Controller and Resource Accounting

  • Controller-call durations are measured on an NVIDIA H200 with batch-one bfloat16 inference; episode time includes initialization and all noninitial calls.Full-call timing covers factual encoding, scoring, selected visual generation, and action decoding; P50 and P95 are call-duration percentiles.Source: F Controller and Resource Accounting · p. 26, Table 25 · p. 27, Table 26 · p. 27
  • RTP solver-interval cost sums 5 intervals per Bridge-5 call, 10 per Bridge-10 call, and 20 per fresh call; each selected branch executes one next-action decode.Backbone forward counts include guidance multiplicity and are distinct from solver-interval counts.Source: F Controller and Resource Accounting · p. 26
  • For H=4, stored first-order Euler checkpoints and the clean endpoint require 589,824/552,960 bytes, while Bridge-5 requires 393,216/368,640 bytes across the two stated patch sizes.The residual has 7,374,528 parameters; Bridge+DiffEncode totals 17,126,592/17,507,520, and the discrepancy estimator has 20,743,240.Source: F Controller and Resource Accounting · p. 27

Related papers

Loading 2609.35439v1…