PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Improving Loops for Visual Robotic Planning

Calvin Luo, Zilai Zeng, Mingxi Jia, Yilun Du, Chen Sun

arXiv:2506.06658v3cs.ROcs.AI

TL;DR

Visual planners trained offline struggle to generalize and cannot improve beyond their pre-collected data. SILVR iteratively adapts an in-domain video model using self-collected trajectories, optionally incorporating web-scale video priors. Experiments on MetaWorld and real robot arms show improvements on unseen tasks, robustness to weaker feedback and demonstrations, and favorable sample efficiency versus alternative online approaches.

  • Problem

    Offline-trained visual planners have limited generalization to unseen robotic tasks and cannot learn beyond pre-collected data.

    Source: §1 · p. 1

  • Method

    SILVR iteratively finetunes an in-domain video model on self-produced trajectories, optionally integrating an internet-pretrained video model as a prior.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4

  • Results

    SILVR improves performance over multiple iterations on novel MetaWorld and real-robot tasks and outperforms equivalent behavior-cloning and reinforcement-learning alternatives in self-improvement and sample efficiency.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    SILVR supports sample-efficient self-improvement without requiring human-defined ground-truth rewards or expert-quality demonstrations, while enabling final policy distillation for fast deployment.

    Source: §1 · p. 2

    SILVR assumes the initial model can achieve enough success on a novel task to collect useful online experience, which may fail for overly challenging tasks.

    Source: §5 · p. 10

Abstract

from arXiv · show

Video generative models trained on expert demonstrations have been utilized as performant text-conditioned visual planners for solving robotic tasks. However, generalization to unseen tasks remains a challenge. Whereas improved generalization may be facilitated by leveraging learned prior knowledge from additional pre-collected offline data sources, such as web-scale video datasets, in the era of experience we aim to design agents that can continuously improve in an online manner from self-collected behaviors. In this work we thus propose the Self-Improving Loops for Visual Robotic Planning (SILVR), where an in-domain video model iteratively updates itself on self-produced trajectories, and steadily improves its performance for a specified task of interest. We apply SILVR to a diverse suite of MetaWorld tasks, as well as two manipulation tasks on a real robot arm, and find that performance improvements continuously emerge over multiple iterations for novel tasks unseen during initial in-domain video model training. We demonstrate that SILVR is robust in the absence of human-provided ground-truth reward functions or expert-quality demonstrations, and is preferable to alternate approaches that utilize online experience in terms of performance and sample efficiency.

1 INTRODUCTION

SILVR addresses limited generalization of offline-trained visual planners by iteratively refining an in-domain video model with self-collected online experience. Across simulated and real robotic tasks, it improves unseen-task performance while reducing reliance on human rewards and expert demonstrations.

  • SILVR combines visual planning with online self-improvement to move beyond the fixed generalization limits of purely offline video data.Prior web-scale video models improve offline generalization, but do not themselves learn from deployment experience.Source: §1 · p. 1
  • SILVR iteratively finetunes an in-domain video model on trajectories collected by agents following their own generated visual plans.The loop can also adapt the inverse dynamics model with environment-grounded data.Source: §1 · p. 2, Figure 1 · p. 2
  • SILVR outperforms direct behavior cloning and is more sample efficient than reinforcement-learning finetuning while supporting later policy distillation for fast inference.The framework retains visual planning for learning and can distill the final planner into a lightweight policy.Source: §1 · p. 2
  • SILVR remains effective without human-defined ground-truth rewards by using a pretrained VLM to score trajectories from task descriptions.The method is presented as robust to sparse-reward quality and suboptimal initial data.Source: §1 · p. 2
  • 285% improvement in success rate is observed over 10 SILVR iterations on novel MetaWorld tasks.The method is also evaluated on colored-object selection and pushing or drawer-opening tasks with a real robot arm.Source: §1 · p. 2

2 RELATED WORK

Related work uses video models as decision-making components and studies self-improvement or online finetuning separately. SILVR combines visual planning with iterative adaptation from online experience, including web-scale priors and comparisons with reinforcement-learning finetuning.

  • Video models have served as reward functions, dynamics models, and pixel-based planners, with visual plans translated into actions through inverse dynamics.SILVR extends this planning paradigm by learning from online environment interactions.Source: §2 · p. 3
  • Web-scale video priors can improve text-conditioned and in-domain visual generalization, but such approaches remain limited to offline data.SILVR instead uses self-collected experience to continue adapting after deployment.Source: §2 · p. 3
  • Prior self-improving generative-model work includes self-generated outputs, VLM feedback, and successful-rollout finetuning, whereas SILVR centers on self-adaptation with internet-scale video priors.The approach also targets suboptimal initial data and relaxed filtering requirements.Source: §2 · p. 3
  • SILVR is positioned against reinforcement-learning finetuning methods for behavior-cloning policies and is reported to achieve faster iterative improvement with greater sample efficiency.The comparison concerns online experience used to improve pretrained action policies.Source: §2 · p. 3

3 METHOD

SILVR uses text-conditioned video planning, optional inverse probabilistic adaptation with a web-scale prior, and iterative finetuning on self-collected trajectories. The resulting planner can be distilled into a lightweight policy for faster deployment.

3.1 VIDEO MODELS AS VISUAL PLANNERS

  • Text-conditioned video models synthesize future visual frames, while an inverse dynamics model converts consecutive frames into executable robotic actions.Replanning frequency trades computational cost against plan-following accuracy and error compounding.Source: §3.1 · pp. 3–4

3.2 INVERSE PROBABILISTIC ADAPTATION

  • Inverse Probabilistic Adaptation composes scores from a small in-domain video model and a frozen web-scale model during sampling.The web-scale model supplies a text-conditioned generalization prior while the in-domain model preserves environment-specific behavior; γ controls prior strength and α controls text guidance.Source: §3.2 · p. 4

3.3 SELF-IMPROVING LOOPS FOR VISUAL ROBOTIC PLANNING

  • SILVR starts from an in-domain video model and repeatedly collects filtered rollouts, accumulates them with initial data, and finetunes the model.Each iteration adapts the planner for a specified task, with optional inverse-dynamics-model finetuning and final policy distillation.Source: §3 · p. 3, §3.1 · p. 4, §3.3 · p. 5
  • SILVR bootstraps the planner by using trajectories from novel-task interaction as further finetuning data, allowing performance to improve for that task over time.The method combines offline demonstrations, optional internet-scale priors, and online experience in a self-adapting cycle.Source: §3.3 · p. 5
  • Visual planning is more sample efficient than reinforcement-learning finetuning of behavior-cloning models and can replace human-defined rewards with an automated VLM success signal.The framework also handles suboptimal initial data quality, reducing dependence on human feedback and expert demonstrations.Source: §3.3 · p. 5
  • The final video planner can be distilled into a lightweight policy that preserves strong performance while enabling reactive, faster inference.Experiments report that the distilled policy outperforms starting self-improvement directly from such a policy and can slightly exceed its teacher's final performance.Source: §3.3 · p. 5

4 EXPERIMENTS

SILVR is evaluated on unseen MetaWorld tasks and real-robot manipulation tasks, where iterative self-improvement, filtering robustness, internet video priors, and distillation are assessed. Across these settings, SILVR improves performance over iterations, including with suboptimal data and without ground-truth filtering signals, while saturation limits later gains.

4.1 EXPERIMENTAL SETUP AND EVALUATION

  • SILVR is evaluated on 12 unseen MetaWorld tasks and two Panda-arm tasks involving pushing unseen-colored cups and opening a novel-colored drawer.MetaWorld uses 25 demonstrations from eight tasks and 30 trajectories per iteration; the real-robot experiments use human-teleoperated demonstrations and self-collected trajectories.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 SILVR VIA FILTERED FINETUNING

  • SILVR outperforms all baselines on 12 unseen MetaWorld tasks, while SILVR-Distilled DP achieves the best overall performance.The comparison uses the same initial in-domain data and aggregates results over three seeds.Source: Table 1 · p. 6, §4.2 · p. 8

4.1 EXPERIMENTAL SETUP AND EVALUATION

  • After two SILVR iterations, visual plans become clearer and demonstrate successful task completion in the same initial layout despite initially blurry, unsuccessful plans.The improvement is shown qualitatively with slight color drift in the generated plans.Source: Figure 2 · p. 6, §4.3 · p. 8

4.3 SILVR WITH INTERNET-SCALE VIDEO PRIOR

  • SILVR improves real-robot visual planning with AnimateDiff, whereas both real-world tasks struggle to improve or deteriorate without the internet video prior.In MetaWorld, self-improvement occurs with or without internet priors, suggesting their benefit is especially important under real-world visual complexity and task difficulty.Source: §4.3 · p. 8

4.4 SILVR SATURATION AND DISTILLATION

  • SILVR performance increases monotonically through 10 iterations but reaches effective saturation after iteration 5, while distillation can further improve performance and enables lightweight deployment.The saturation may reflect local minima from repeatedly collecting similar experiences; the distilled policy at Iteration 9 outperforms BCIL.Source: Figure 4 · p. 7, §4.4 · p. 9

4.5 IMPACT OF DATA FILTERING SIGNALS ON SILVR

  • VLM-based filtering enables self-improvement, and Panda-arm SILVR continues improving without any filtering despite the absence of ground-truth success signals.Gemini achieves the best performance among the evaluated VLM filters on MetaWorld, while unfiltered Panda-arm adaptation benefits from the internet video prior.Source: Figure 5 · p. 8, §4.5 · p. 9

4.6 SILVR WITH SUBOPTIMAL DATA

  • SILVR continues improving when initialized with suboptimal demonstrations, although task-specific outcomes differ between expert and random-action initialization.The suboptimal trajectories use random actions 70% of the time and expert actions 30% of the time; some tasks benefit more from expert demonstrations, while others benefit from random exploration.Source: §4.6 · pp. 9–10, Figure 6 · p. 10

5 CONCLUSION AND FUTURE WORK

SILVR iteratively improves visual planning for novel robotic tasks using self-collected online experience, outperforming comparable behavior-cloning and reinforcement-learning setups in self-improvement and sample efficiency. Its scope includes both synthetic environments and real-world robot arms, but it depends on sufficient initial task success and faces a cold-start challenge.

Conclusion

  • SILVR iteratively updates an in-domain visual planner for a novel task using self-collected online experience.The loop begins with an in-domain video model and repeatedly adapts it from accumulated experience.Source: §5 · p. 10
  • Compared with equivalent behavior cloning and reinforcement-learning setups, SILVR achieves superior self-improvement capabilities and sample efficiency.The conclusion reports this comparison across the evaluated settings.Source: §5 · p. 10
  • SILVR is applied successfully in both synthetic environments and real-world robot-arm deployments.The paper reports real-world application in addition to synthetic evaluation.Source: §5 · p. 10

Limitations and Future Work

  • SILVR assumes the initial in-domain model has enough success on the novel task to collect online experience and improve.This assumption may fail when the novel task is too challenging.Source: §5 · p. 10
  • The cold-start problem can challenge SILVR when the novel task lacks sufficient initial success.Improving exploration is identified as a future direction for addressing this boundary.Source: §5 · p. 10

A TASKS AND TEXT PROMPTS

The appendix lists task-prompt pairs used for in-domain training and evaluation, marking tasks seen during initial training with an asterisk and including prompts for internet-model adaptation.

A TASKS AND TEXT PROMPTS

B IMPLEMENTATION DETAILS

The implementation combines visual planning, inverse dynamics, optional internet-video adaptation, iterative policy updates, and optional distillation, with control-loop choices balancing execution quality and speed. Comparisons use matched experience and update budgets to assess behavior cloning and reinforcement-learning alternatives.

B IMPLEMENTATION DETAILS

  • The MLP inverse dynamics model predicts actions from embeddings of two video frames, with frame spacing differing between Panda Arm and MetaWorld experiments.For Panda Arm, it predicts the final-frame end-effector position for inverse-kinematics execution; MetaWorld uses consecutive frames.Source: B IMPLEMENTATION DETAILS · p. 14
  • SILVR uses an in-domain text-conditioned video model, inverse dynamics models, and optional internet-pretrained video adaptation.The in-domain model conditions on language and an initial frame, while AnimateDiff supplies a frozen internet-scale prior for adaptation.Source: B IMPLEMENTATION DETAILS · pp. 14–15
  • Visual plans contain one current observation and eight future frames, which translate into eight actions through the inverse dynamics model.Open-loop, closed-loop, and semi-open-loop control trade interaction efficiency against replanning and state-feedback reliability.Source: B IMPLEMENTATION DETAILS · p. 16
  • The distilled diffusion policy captures the slower visual planner’s current performance while enabling lightweight, fast inference, despite weaker self-improvement.Distillation therefore separates training-time autonomous improvement from deployment-time execution speed.Source: B IMPLEMENTATION DETAILS · pp. 16–17
  • With matched experience and a sufficiently large update budget, DSRL does not match SILVR’s iterative improvements, indicating experience collection—not gradient updates—is its bottleneck.A light ablation likewise finds no significant improvement-trend difference across update rates, while SILVR shows better sample efficiency.Source: B IMPLEMENTATION DETAILS · p. 17, Table A7 · p. 17

C METAWORLD TASK PERFORMANCE DECOMPOSITION

The MetaWorld decomposition evaluates task performance on 12 tasks unseen during initial in-domain training and compares visual planning with action-predictive behavior cloning. The supplied passages emphasize visual planning’s generalization mechanism: coherent pixel-level motions can remain useful on novel objects even when object details are blurred.

C METAWORLD TASK PERFORMANCE DECOMPOSITION

  • The MetaWorld evaluation reports mean success rates and standard deviations across three seeds and ten iterations for 12 initially unseen tasks.Source: Table A8 · p. 17

D GENERALIZATION CAPABILITIES BETWEEN VISUAL PLANNING AND ACTION-PREDICTIVE BC

  • Visual planning is reported to have default generalization and self-improvement benefits over direct action-prediction behavior-cloning policies.The comparison concerns decision-making on the MetaWorld task suite.Source: D GENERALIZATION CAPABILITIES BETWEEN… · pp. 17–18
  • Pixel-level visual-motion modeling may provide more training signal than directly modeling low-dimensional action sequences.The passage presents this as a hypothesis explaining the observed advantage.Source: D GENERALIZATION CAPABILITIES BETWEEN… · p. 18
  • On an unseen-object task, the visual planner can generate coherent arm motions even when the specific object interaction is initially blurred or incorrect.The inverse dynamics model can translate those coherent motions into meaningful robot actions.Source: D GENERALIZATION CAPABILITIES BETWEEN… · p. 18
  • Action-predictive behavior cloning may produce suboptimal actions on novel objects after overfitting to its training set.Unlike visual planning, it does not model visual details explicitly.Source: D GENERALIZATION CAPABILITIES BETWEEN… · p. 18

E FULL METAWORLD SUBOPTIMAL RESULTS

With suboptimal initial in-domain data, SILVR generally improves across 12 unseen MetaWorld tasks, while BCIL benefits more from this initialization than expected.

E FULL METAWORLD SUBOPTIMAL RESULTS

F METAWORLD COMPONENT ABLATIONS

Component ablations show that MetaWorld tasks with substantial novelty require updating both the video model and IDM, while the appendix visualizes filtered, unfiltered, and internet-prior variants.

F METAWORLD COMPONENT ABLATIONS

  • Updating both the video model and IDM is critical for MetaWorld tasks involving heavy novelty across objects and motions.Freezing the IDM yields only modest gains, while updating only the IDM produces gains that saturate quickly.Source: F METAWORLD COMPONENT ABLATIONS · p. 19, Table A11 · p. 19

G ADDITIONAL PLAN VISUALIZATIONS

G.1 SILVR WITH GROUND-TRUTH FILTERING

G.2 SILVR WITHOUT FILTERING

G.3 CORRECTIVE INFLUENCE OF INTERNET-SCALE VIDEO MODEL ADAPTATION

G.4 LONG-HORIZON TASK EVALUATION

SILVR’s iterative improvement on atomic cup-pushing skills extends to sequential long-horizon tasks when a visual-language model selects subtasks and detects completion.

G.4 LONG-HORIZON TASK EVALUATION

  • A VLM determines which color-specific cup-pushing skill to execute and when to switch after querying whether the current skill succeeded.Success requires pushing cups in the exact natural-language-specified order.Source: §G.4 · p. 27
  • Long-horizon success improves over successive iterations even though SILVR is applied only to individual atomic tasks.The authors suggest combining SILVR with efficient subtask segmentation for complex sequential compositions.Source: §G.4 · p. 27
  • For red-orange-green cups, only the final-iteration planner completes the long-horizon task, while the first two iterations fail.Source: Figure A16 · p. 27
  • For blue-purple-green cups, the last two iterations complete the task, while the first iteration fails.Source: Figure A17 · p. 28

G.5 VISUALIZING FAILURE CASES OF FINAL-ITERATION VISUAL PLANNERS

Final-iteration failures arise from semantic planning errors in real-world tasks and from both execution and semantic errors in simulation.

G.5 VISUALIZING FAILURE CASES OF FINAL-ITERATION VISUAL PLANNERS

Loading 2506.06658v3…