PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Improving Embodied Foundation Models

Seyed Kamyar Seyed Ghasemipour, Ayzaan Wahid, Jonathan Tompson, Pannag Sanketi, Igor Mordatch

arXiv:2509.15155v1cs.LGcs.RO

TL;DR

Low-level control training for embodied foundation models has largely been limited to behavioral cloning, while reward engineering remains a practical challenge in robotics. The paper combines supervised fine-tuning with online Self-Improvement driven by predicted steps-to-go; experiments report greater sample efficiency and skill acquisition beyond imitation data.

  • Problem

    Manual reward design requires trial-and-error, and measuring real-world rewards demands significant engineering, making it difficult to scale robot training across broader tasks.

    Source: §2 · p. 3

  • Method

    The method fine-tunes pretrained models with behavioral cloning and steps-to-go prediction, then uses predicted rewards and success detection for online reinforcement learning.

    Source: §2.2 · p. 4, §7 · p. 14

  • Results

    10% additional robot time for Self-Improvement raises LanguageTable success from 45% to 75%, compared with 45% to 60% after 8× more imitation data; the authors report broader sample-efficiency gains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that combining web-scale pretraining with online Self-Improvement enables autonomous acquisition of skills generalizing beyond imitation-training tasks.

    Source: §7 · p. 15

    The method uses on-policy REINFORCE without data reuse, foregoing the data-reuse benefits of modern off-policy algorithms.

    Source: §6 · p. 14

Abstract

from arXiv · show

Foundation models trained on web-scale data have revolutionized robotics, but their application to low-level control remains largely limited to behavioral cloning. Drawing inspiration from the success of the reinforcement learning stage in fine-tuning large language models, we propose a two-stage post-training approach for robotics. The first stage, Supervised Fine-Tuning (SFT), fine-tunes pretrained foundation models using both: a) behavioral cloning, and b) steps-to-go prediction objectives. In the second stage, Self-Improvement, steps-to-go prediction enables the extraction of a well-shaped reward function and a robust success detector, enabling a fleet of robots to autonomously practice downstream tasks with minimal human supervision. Through extensive experiments on real-world and simulated robot embodiments, our novel post-training recipe unveils significant results on Embodied Foundation Models. First, we demonstrate that the combination of SFT and Self-Improvement is significantly more sample-efficient than scaling imitation data collection for supervised learning, and that it leads to policies with significantly higher success rates. Further ablations highlight that the combination of web-scale pretraining and Self-Improvement is the key to this sample-efficiency. Next, we demonstrate that our proposed combination uniquely unlocks a capability that current methods cannot achieve: autonomously practicing and acquiring novel skills that generalize far beyond the behaviors observed in the imitation learning datasets used during training. These findings highlight the transformative potential of combining pretrained foundation models with online Self-Improvement to enable autonomous skill acquisition in robotics. Our project website can be found at https://self-improving-efms.github.io .

1 INTRODUCTION

Embodied foundation models have largely been trained for low-level control through behavioral cloning, despite evidence that reinforcement-learning post-training improves language-model performance beyond supervised fine-tuning. This work introduces SFT followed by Self-Improvement and reports improved sample efficiency, policy success, and acquisition of skills beyond imitation data.

1 INTRODUCTION

  • Prior EFM training has largely been behavioral cloning, while LLM post-training commonly pairs supervised fine-tuning with reinforcement learning.The cited LLM literature reports that RL tuning can improve downstream performance beyond SFT.Source: §1 · p. 1
  • SFT combines behavioral cloning with steps-to-go prediction, then Self-Improvement uses model-predicted rewards and success detection for autonomous downstream practice.The framework draws on pretrained multimodal foundation models and aims to let one operator monitor multiple robots.Source: §1 · p. 2, Figure 1 · p. 2
  • 10% additional robot time for Self-Improvement raises LanguageTable success from 45% to 75%, versus 45% to 60% after 8× more imitation data.The authors also report that SFT plus Self-Improvement improves performance beyond behavioral cloning and is more sample-efficient than scaling imitation data alone.Source: §1 · p. 2
  • The authors report that online Self-Improvement combined with web-scale pretraining enables robots to acquire skills that generalize beyond Stage 1 imitation data.They distinguish this behavioral generalization from prior semantic generalization, such as performing the same pick-and-place motions in new contexts [Brohan et al., 2023].Source: §1 · p. 2

2 METHODOLOGY

The methodology addresses the engineering burden of manually designing and measuring robot rewards by deriving rewards and success detection from a pretrained model's steps-to-go predictions. It combines supervised objectives with online reinforcement learning for downstream practice.

2 METHODOLOGY

  • The approach replaces task-specific ground-truth rewards with data-driven rewards that inherit robustness and generalization properties from pretrained foundation models.Manual reward design and real-world reward measurement require repeated trial-and-error and significant engineering effort.Source: §2 · p. 3

2.1 STAGE 1: SUPERVISED FINE-TUNING (SFT)

  • SFT initializes an EFM from a pretrained foundation model and trains on imitation tuples without assuming the dataset trajectories are optimal.Its objectives include goal-conditioned behavioral cloning and prediction of the timesteps remaining until the intended goal.Source: §2.1 · p. 3

2.2 STAGE 2: SELF-IMPROVEMENT

  • The Stage 2 downstream task may differ significantly from the Stage 1 imitation dataset tasks.The paper identifies this as a possible setting for online fine-tuning, rather than requiring downstream tasks to match the training dataset.Source: §2.2 · p. 3
  • Steps-to-go prediction supplies the progress estimate used to derive the online RL reward, which reflects how much closer an action moves the robot to its goal.The reward is based on changes in the model-predicted steps-to-go; the expected value is chosen for simplicity and alignment with an RL value function.Source: §2.2 · pp. 3–4
  • When ground-truth success detection is unavailable, the method derives a success indicator from the model's steps-to-go prediction and a small threshold.The authors report that this detector is robust in low-data regimes and more reliable than adding a binary success-classification objective during SFT.Source: §2.2 · p. 4
  • Self-Improvement collects trajectories until predicted success, a maximum episode length, or human termination, then computes returns and updates the policy with REINFORCE.The procedure uses a frozen Stage 1 checkpoint for reward and success computation; it uses on-policy RL without data reuse for stability, foregoing off-policy data-reuse benefits.Source: §2.2 · pp. 4–5

3 INTUITION ON REWARD FUNCTION

The section explains how steps-to-go values produce a shaped reward that favors states where the dataset policy performs well. This reward can improve goal efficiency while implicitly regularizing policies toward regions where that policy is proficient.

Mathematical Intuition

  • The value-difference reward favors states where dataset policy µ performs well, encouraging more efficient goal achievement while implicitly regularizing toward regions where µ is proficient.The paper describes the reward as combining a core reward and reward shaping.Source: §3 · p. 5

Visual Intuition

  • During Aloha insertion, predicted steps-to-go distributions widen when the peg or socket slips, then narrow during recovery; after the socket fully slips, immediate-save probability disappears.The socket slippage is barely visible from the left wrist camera and not visible in the other camera views.Source: Figure 3 · p. 6

Mathematical Intuition

  • As γ approaches 0, the return resembles single-step improvement for the goal-satisfaction reward; as γ approaches 1, it favors trajectories through states valued highly by dataset policy µ.The baseline Vµ(ot, g) lowers variance for REINFORCE estimates.Source: §3 · p. 6

Pointmass Navigation Domain

  • Without ground-truth rewards, Self-Improvement rapidly brings pointmass policies close to optimality, while Stage 1 behavioral cloning mimics the dataset’s waypoint-based suboptimalities.The imitation dataset follows five waypoints before heading to the goal.Source: §3 · pp. 6–7

4 EXPERIMENTS

Experiments compare Self-Improvement with behavioral cloning across simulated and real LanguageTable and Aloha settings, then test pretraining’s role and generalization to domain transfer and novel skills. Across these settings, results highlight sample-efficient gains and behavioral generalization beyond the Stage 1 imitation data.

4 EXPERIMENTS

  • The experiments use LanguageTable and Aloha in simulation and the real world, testing policy gains, sample efficiency, reliability, pretraining, and generalization beyond imitation data.The study uses PaLI as its pretrained foundation model and evaluates Stage 2 Self-Improvement against Stage 1 behavioral cloning.Source: §4 · p. 7

4.1.1 SIMULATED LANGUAGETABLE

  • Across simulated LanguageTable dataset sizes, Self-Improvement raised success rates by at least 1.5× with less than 2% additional episodes; 1% extra episodes on 10% data outperformed behavioral cloning on 20% and 80% data.Results are reported relative to Stage 1 behavioral-cloning policies.Source: §4.1.1 · p. 8

4.1.2 REAL-WORLD LANGUAGETABLE

  • Real-world LanguageTable success rose from ∼62–63% to ∼87–88% with ∼3% additional Block2Block episodes, exceeding behavioral cloning trained on 80% of the imitation dataset even with 20% data plus those episodes.Self-Improvement also requires only a fraction of the human effort needed for teleoperated imitation collection.Source: §4.1.2 · p. 9

4.1.3 SIMULATED ALOHA SINGLE INSERTION TASK

  • In simulated Aloha, 5K imitation plus 2.5K Self-Improvement episodes outperformed 10K imitation episodes and rivaled 15K imitation episodes.Self-Improvement significantly improved success rates at both tested dataset sizes.Source: §4.1.3 · p. 10

4.2 IMPORTANCE OF FOUNDATION MODEL PRETRAINING

  • Reward-model performance ordered PaLI above Uni-PaLI above Scratch; PaLI at 20% data produced better policies than Uni-PaLI at 80%, while Scratch struggled in low-data settings.Scratch reward models also showed high variance across random seeds.Source: §4.2 · pp. 10–11

4.3 GENERALIZATION

  • The proposed combination of online Self-Improvement and pretrained multimodal models targets novel tasks requiring behavioral generalization beyond the Stage 1 imitation data.The paper distinguishes learning new skills from semantic generalization, such as repeating familiar motions in new contexts.Source: §4.3 · p. 11, §4.3.2 · p. 12

4.3.1 DOMAIN TRANSFER BETWEEN SIMULATION AND REAL

  • With only 3% extra simulated LanguageTable episodes after training on 80% of the real dataset, Real2Sim Self-Improvement raised success from ∼22% to ∼59%; Uni-PaLI rewards led to slower improvement.The reported performance was equivalent to behavioral-cloning policies trained with 80% of the target-domain imitation dataset.Source: §4.3.1 · p. 11

4.3.2 STRONG GENERALIZATION TO LEARNING NOVEL SKILLS

  • On BananaTable, success rose from ∼63% to ∼85% after ∼8 hours of Self-Improvement using two robot stations, despite the task’s absence from the Stage 1 data.The policy learned to push from the banana’s middle or tips to prevent it from rotating.Source: §4.3.2 · p. 12

5 RELATED WORKS

Prior work largely trained embodied foundation-model policies through supervised learning; this work presents online Self-Improvement as a way to improve policies and acquire behaviors beyond training data. It also relates its approach to reward-learning and steps-to-go methods.

Embodied Foundation Models

  • The authors describe their work as the first to move beyond supervised learning for training robot foundation models, proposing reward-engineering-free online Self-Improvement for policy improvement and novel behaviors.They base their policy architecture on RT-2 and distinguish behavioral generalization beyond training data from semantic generalization to new contexts.Source: §5 · p. 12

Improving Robot Policies Without Ground-Truth Rewards

  • Prior reward approaches include LLM-written code rewards, learned latent rewards, relabeled-data RL, and timestep-distance methods, each with distinct settings or practical constraints.The authors identify steps-to-go methods as the closest related work and position their approach around foundation models and generalization.Source: §5 · p. 13
  • Steps-to-go thresholding is presented as a path to open-ended success detectors, alongside dense rewards for policy improvement.The cited related work typically trains success detectors through binary classification.Source: §5 · p. 13

6 FUTURE WORK AND LIMITATIONS

The paper identifies limits and future directions in skill-boundary annotation, reward inference for out-of-distribution failures, pretraining, and reinforcement-learning choices. It also points to extensions toward long-horizon tasks and more sample-efficient learning.

Episode Boundaries & Skill–Chaining

  • Manual episode and skill-boundary annotation is prohibitively expensive, constraining skill-chaining without scalable segmentation methods.The authors identify automated recovery of boundaries from raw interaction logs as future work.Source: §6 · pp. 13–14

Reward Models

  • Reward models face failure states outside imitation-data support, where recovery trajectories are absent; broader data or labeled policy rollouts are proposed directions.More expressive steps-to-go estimators could assign shaped rewards or trigger a recovery skill for such states.Source: §6 · p. 14

Embodied Foundation Models

  • The study fine-tunes vision-language backbones without robotics pretraining, motivating future curricula that add physical-reasoning priors while preserving visual-semantic knowledge.The authors also suggest general-purpose post-training as a possible route to zero-shot policies.Source: §6 · p. 14

RL Algorithms

  • On-policy REINFORCE avoids data reuse, while pushing Self-Improvement past its performance peak can degrade success rates.The authors suggest off-policy variants and better stopping criteria or adaptive regularizers as future directions.Source: §6 · p. 14

Summary

  • The authors describe substantial gains with minimal reward engineering and one human supervising multiple robots, while identifying long-horizon tasks and more sample-efficient RL as future needs.Richer reward inference and domain-aligned pretraining are also listed as directions for scaling autonomous skill acquisition.Source: §6 · p. 14

7 CONCLUSION

The paper proposes two-stage post-training for embodied foundation models: supervised fine-tuning followed by online Self-Improvement. Experiments report higher performance and sample efficiency, and the combination of web-scale pretraining and Self-Improvement supports acquisition of skills beyond imitation data.

7 CONCLUSION

  • The proposed two-stage approach uses behavioral cloning and steps-to-go prediction in SFT, followed by online Self-Improvement using steps-to-go predictions for RL post-training.The authors say this lets robots practice downstream tasks with minimal reward engineering and human supervision.Source: §7 · p. 14
  • Self-Improvement improves policy performance beyond supervised learning, and combining both stages is more sample-efficient than supervised learning alone.The authors report that web-scale pretraining combined with Self-Improvement yields better policies and is a key enabler of sample efficiency.Source: §7 · p. 15
  • Web-scale pretraining combined with Self-Improvement enables autonomous acquisition of skills that generalize beyond imitation-learning tasks.The authors describe this capability as unavailable to current methods.Source: §7 · p. 15

A IMPLEMENTATION DETAILS

The implementation uses PaLI as its pretrained vision-language model and an RT-2-equivalent policy representation. Steps-to-go predictions are mapped into the PaLI token space; environment, task, and tokenization details are pointed to appendices.

A.1 BACKGROUND

  • PaLI takes image and text inputs and produces text tokens, using a ViT and an encoder-decoder Transformer initialized from separately pretrained vision and language models.The model is then fine-tuned with vision-language objectives to obtain a multimodal foundation model.Source: §A.1 · p. 21
  • The robot policy follows RT-2: continuous actions are discretized into VLM tokens and predicted from image and text inputs through behavioral cloning.The paper uses PaLI as the underlying VLM for this policy representation.Source: §A.1 · p. 21

A.2 ENVIRONMENTS, TASKS, AND TOKENIZATION

  • Steps-to-go predictions are modeled by discretizing the possible step counts and mapping them to PaLI tokens; environment, task, and tokenization details are in Appendices D and E.Appendix D covers environments and tasks, while Appendix E covers tokenization.Source: §A.1 · p. 21, §A.2 · p. 21

A.3 TRAINING DETAILS

Training combines behavioral cloning and steps-to-go prediction in SFT with online Self-Improvement, using different checkpoint choices for policy initialization and reward or success prediction. The appendix also reports compute allocations and the rationale for a small REINFORCE multiplier.

A.3 TRAINING DETAILS

  • SFT uses behavioral cloning and steps-to-go prediction; LanguageTable also predicts episode instructions, while Stage 2 freezes the ViT and collects trajectories for 16 updates per RL loop.Stage 1 trains the entire PaLI model; the added instruction objective was not ablated. The frozen ViT choice was also not ablated.Source: §A.3 · p. 21

A.4 COMPUTE RESOURCES

  • Stage 2 allocates half the SFT resources to the learner and uses 4 TPUv4 each for the reward model and success detector.The learner allocation reflects the use of half the batch size.Source: §A.4 · p. 22

B CHECKPOINT SELECTION FOR STAGE 2 INITIALIZATION

  • For Aloha, policy initialization used a later, overfit checkpoint because the checkpoint with the best behavioral-cloning validation loss had poor success.For most other choices, the authors used checkpoints with the best validation loss for the corresponding objective.Source: B CHECKPOINT SELECTION FOR STAGE 2… · p. 22

C REINFORCE MULTIPLICATIVE CONSTANT

  • In simulation, the authors chose c = 5e-2 for stable REINFORCE training, using an approximate intuition rather than careful tuning.The scale was motivated by γ = 0.9 and an assumed range for N of approximately [−2, 2].Source: C REINFORCE MULTIPLICATIVE CONSTANT · p. 22

D ENVIRONMENTS AND TASKS

The experiments span LanguageTable, BananaTable, and bimanual Aloha tasks, with domain-specific action, observation, and steps-to-go representations. The appendix also describes the demonstrations and policy choices used for Aloha.

D.1 LANGUAGETABLE

  • LanguageTable uses 2D actions and a dataset of 181,020 human trajectories with 78,623 unique instructions; Stage 2 focuses on Block2Block tasks.The cited dataset is the original LanguageTable dataset.Source: §D.1 · pp. 22–23

D.2 BANANATABLE

  • BananaTable replaces the blocks with one banana and asks the robot to push it to a specified table location.Instructions use pushing synonyms and specify one of eight locations.Source: §D.2 · p. 23

D.3 ALOHA

  • Aloha is a 14-DoF bimanual task with 10Hz, five-action chunks forming a 70-dimensional action space; its insertion data comprise 800 demonstrations and generated successful-rollout datasets.Semi-global action representations and training beyond the best behavioral-cloning validation loss were critical to successful PaLI policies.Source: §D.3 · p. 23

E.1 REAL/SIM LANGUAGETABLE & BANANATABLE

  • LanguageTable and BananaTable actions are binned into four tokens, while steps-to-go is represented by 50 bins over 0–100 steps.The steps-to-go range reflects the upper percentile of imitation-dataset episode lengths.Source: §E.1 · pp. 23–24

E.2 ALOHA

  • Aloha actions use 70 tokens binned into 256 values, steps-to-go uses one token for each value from 0–300, and joint positions use 14 tokens.Action and joint values are discretized from the continuous range [−1, 1].Source: §E.2 · p. 24

F REAL-WORLD LANGUAGETABLE EXPERIMENTATION PROCEDURE

A single human monitored all real-world experiments and performed resets; operators supplied no labels or success indicators to the models.

F REAL-WORLD LANGUAGETABLE EXPERIMENTATION PROCEDURE

  • One human monitored all robots and reset them when a block fell or a station had gone 3–5 minutes without a shuffle and reset.The human provided no labels or success indicators to the models.Source: F REAL-WORLD LANGUAGETABLE… · p. 24

G ADDITIONAL PLOTS

Additional plots describe real-world LanguageTable and simulated LanguageTable and Aloha self-improvement experiments. The appendix also estimates the share of LanguageTable instructions that are Block2Block tasks.

G ADDITIONAL PLOTS

  • Stage 2 Self-Improvement is more sample-efficient than using the full robot time budget to collect imitation data for SFT in simulated LanguageTable and Aloha insertion.The real-world LanguageTable plots compare 80% Stage 1 data with three robots against 20% data with three or four robots; each experiment took about 20 hours.Source: Figure 11 · p. 25, Figure 10 · p. 25

H COMPUTING PERCENTAGE OF BLOCK2BLOCK INSTRUCTION IN LANGUAGETABLE

  • Block2Block instructions involve moving or pushing one block toward a specific other block, excluding separation, between-block placement, or placement near a group.The appendix used this definition to classify instructions as Block2Block or not.Source: H COMPUTING PERCENTAGE OF BLOCK2BLOCK… · p. 26
  • 47% of simulated and 49% of real LanguageTable instructions were labeled Block2Block, with estimates within 2.8% error at 99.9% confidence.The estimates used 5,000 randomly sampled instructions from each dataset.Source: H COMPUTING PERCENTAGE OF BLOCK2BLOCK… · p. 26

I INTERESTING OBSERVATIONS & INCOMPLETE EXPERIMENTS

Self-Improvement on LanguageTable Block2Block tasks improved performance across all LanguageTable tasks. The section also reports incomplete real-world experiments and describes two Self-Improvement infrastructure versions, with local policy inference introduced to reduce network latency.

I INTERESTING OBSERVATIONS & INCOMPLETE EXPERIMENTS

  • Self-Improvement on Block2Block tasks improved performance across all LanguageTable tasks in real-world rollouts.The all-instructions experiment also showed clear success-rate improvement, but its results were excluded after some stations were improperly set up; the experiment could not be retried.Source: I INTERESTING OBSERVATIONS &… · p. 27
  • Real-world Aloha Single-Insertion behavioral-cloning policies showed reasonable success, but a full Self-Improvement experiment was not completed.Version 1 could not reliably support Aloha's required 10 Hz control frequency; time constraints prevented completion after Version 2 was briefly tested.Source: I INTERESTING OBSERVATIONS &… · p. 27, J INFRASTRUCTURE OVERVIEW · pp. 27–28

J INFRASTRUCTURE OVERVIEW

  • Across the experiments, Version 1's distributed infrastructure supported simulated and real-world environments operating at control frequencies up to 5 Hz.The section introduces both infrastructure versions and notes Version 1 was used for all reported experiments.Source: J INFRASTRUCTURE OVERVIEW · p. 27

J.1 VERSION 1: NON-LOCAL POLICY

  • Version 1 separates actors from computation, using remote action inference and separate steps-to-go inference nodes for success detection and reward labeling.After episodes, steps-to-go predictions label observations; rewards are prediction differences, and a coordinator schedules learning from replay-buffer data.Source: §J.1 · p. 28

J.2 VERSION 2: LOCAL POLICY

  • Version 2 performs policy inference locally on actor machines, reducing network latency while requiring each machine to hold the full policy model.Learning nodes distribute updated weights to actors after each iteration; steps-to-go inference, reward labeling, and success detection remain separate.Source: §J.2 · p. 29

Related papers

Loading 2509.15155v1…