PaperlayerView exact revision on arXiv

Source-linked AI summary

Training in Deployment: Human-in-the-loop Online Post-Training for Robot Manipulation

Chun Gu, Xiaofei Wei, Yiming Chen, Bozhou Zhang, Juntong Zhang, Xiatian Zhu, Li Zhang

arXiv:2610.29264v1cs.RO

TL;DR

Offline teleoperation may miss contact-rich, recovery, and precision-critical states, while offline post-training adapts slowly and online RL often relies on difficult-to-estimate value functions. TIDE uses provenance-conditioned supervised learning to update a robot policy from human interventions and policy executions during deployment. Across four real-world tasks, it improves task progress and completion over warm-start and additional-offline-data baselines, with performance comparable to Fixed Policy.

  • Problem

    Offline demonstrations can miss deployment-critical manipulation states, while offline post-training cannot immediately incorporate deployment experience and online RL often relies on difficult-to-estimate value functions.

    Source: §I · p. 1

  • Method

    TIDE updates trainable policy modules during deployment using provenance-conditioned supervised learning on human-intervention and policy-executed data alongside offline task data.

    Source: §I · p. 2

  • Results

    Across four real-world tasks, TIDE achieves higher milestone-normalized task progress and binary completion than warm-start and additional-offline-data baselines, with final performance comparable to Fixed Policy.

    Source: B. Real-world experiments · p. 6, §V · p. 7

  • Takeaways & Limitations

    TIDE makes updated checkpoints available during deployment rather than only after a separate collection, training, and redeployment cycle.

    Source: B. Real-world experiments · p. 6, §V · p. 7

    The evaluation covers one bimanual platform, four tasks, and one world-action backbone, and does not test long-term retention, online-training-schedule sensitivity, or variation across repeated sessions.

    Source: §VI · p. 7

Abstract

from arXiv · show

Large pretrained robot foundation models provide strong priors for manipulation, yet adapting them to target tasks with offline teleoperated demonstrations often fails to cover the contact-rich, recovery, and precision-critical states encountered during deployment. Existing methods mitigate this issue by augmenting offline demonstrations with human-in-the-loop data or online reinforcement learning. Despite this, offline post-training adapts slowly because it cannot immediately absorb deployment experience, while online RL typically relies on value functions or auxiliary actor-critic models, increasing both training complexity and instability. We present TIDE (Training In DEployment), a human-in-the-loop online post-training system. TIDE introduces a provenance-conditioned supervised learning framework that updates the policy online from human interventions and policy-generated rollouts collected during deployment, without a learned critic, reward model, or action scorer. Across four real-world bimanual manipulation tasks, TIDE improves milestone-normalized task progress and binary full-task completion over the warm-start policy and matched additional offline demonstrations, and performs comparably to the fixed-policy collection baseline while making updated checkpoints available during deployment. Each task-specific online post-training session, from the start of deployment rollouts until the final checkpoint is available, completes in less than one hour while learning and checkpoint handling run asynchronously.

I. INTRODUCTION

Offline demonstrations can miss deployment-critical states, while existing online approaches face delays or difficult value estimation. TIDE uses provenance-conditioned supervised learning during deployment and improves task performance across four real-robot tasks.

I. INTRODUCTION

  • Offline teleoperation can miss contact-rich, recovery, and precision-critical states, while collecting more episodes to cover rare deployment states is data-inefficient.Offline-only post-training also cannot immediately absorb deployment experience; online RL often relies on critics that are difficult to train stably for high-dimensional action chunks.Source: §I · pp. 1–2
  • TIDE records whether executed actions came from a human or the policy, then uses provenance-conditioned supervised training to update trainable policy modules without a reward model, critic, or action scorer.The learner combines deployment data with offline task data and returns updated checkpoints at episode boundaries.Source: §I · p. 2
  • Across four real-robot tasks, TIDE improves no-takeover task performance over the warm-start policy and matched additional offline demonstrations, performing comparably to the Fixed Policy baseline.Updated checkpoints become available during deployment rather than after a separate collection, training, and redeployment cycle.Source: §I · p. 2
  • The observed intervention ratio decreases over deployment, indicating lower human-supervision burden, but it is not an autonomous-performance measure.Interventions and policy executions are treated as structured online experience rather than flattening deployment into additional expert demonstrations.Source: §I · p. 2

II. RELATED WORK

Robot foundation models combine broad pretrained priors with increasingly capable action and video-generation interfaces. Their deployment performance remains bounded by training-distribution coverage, motivating post-training through interaction with the deployment environment.

II. RELATED WORK

  • Large pretrained robot models map observations and language instructions to actions, supported by cross-embodiment datasets and increasingly capable action interfaces.Examples include flow-matching action experts over continuous chunks, efficient action tokenization, and models coupling action generation with future visual-state prediction.Source: A. Robot foundation models for… · p. 2
  • Policies can remain limited by training-distribution coverage as small errors compound into out-of-distribution states and precision-critical subtasks resist improvement after many additional teleoperation episodes.The passage motivates post-training procedures that refine pretrained policies through deployment-environment interaction.Source: A. Robot foundation models for… · p. 2
  • Robot foundation-model work spans instruction-conditioned systems such as RT-1, RT-2, OpenVLA, and Octo, with newer generalist and humanoid systems extending across embodiments and physical settings.The cited passage also describes video-generation priors as a route to robustness for unfamiliar skills and embodiments.Source: A. Robot foundation models for… · p. 2

B. Human-in-the-loop post-training

Human-in-the-loop methods learn from states visited by the current policy, while newer approaches move policy improvement closer to deployment in different ways. TIDE targets a concurrent real-robot loop with provenance-organized supervision and updates to all trainable policy modules.

B. Human-in-the-loop post-training

  • Human-in-the-loop imitation learning addresses covariate shift by letting operators intervene on states visited by the current policy, and later work varies intervention triggers, data weighting, and takeover methods.RLIF uses intervention signals as rewards, MILE models intervention and non-intervention feedback, and Sirius weights behavioral cloning by approximated human trust.Source: B. Human-in-the-loop post-training · p. 2
  • Interactive imitation-learning systems often update in batches; RoboCopilot remains round-based, and RECAP uses phase-based offline updates while leaving a fully concurrent online framework for future work.TIDE’s pipeline instead returns updated policies to the robot at episode boundaries while learning and data collection proceed asynchronously in one deployment session.Source: B. Human-in-the-loop post-training · p. 3, Figure 2 · p. 3
  • Several deployment-oriented systems retain constraints: RL Token trains an actor-critic over a frozen policy, Hi-ORS uses trajectory-level outcome supervision, and ConRFT retains a Q-function.RoboPocket collects corrections from visualized predictions without executing them on the physical robot.Source: B. Human-in-the-loop post-training · p. 3
  • TIDE targets real-robot deployment in the data loop, jointly updates the video backbone, cross-modal adapter, and action expert, and organizes supervision by action-source provenance.This contrasts with learned values or trajectory-level reward labels as supervision.Source: B. Human-in-the-loop post-training · p. 3

III. METHOD

TIDE starts from a pretrained world-action policy warm-started on offline task demonstrations. It then turns deployment execution into provenance-structured training data within the same session.

III. METHOD

  • The pretrained policy combines a video-generation backbone with a flow-matching action expert that uses a compact world prior and robot state to generate action chunks.The backbone also predicts future video frames over the action horizon.Source: §III · p. 3
  • Before deployment, TIDE warm-starts all trainable policy modules on offline human teleoperation data for the target task.The initialization is executable, but deployment can reveal states that offline demonstrations did not cover.Source: §III · p. 3
  • TIDE integrates robot execution, provenance recording, online learning, and checkpoint handoff in one deployment loop, converting deployment-time states into training signals during the same session.The method describes the runtime and provenance-labeled chunk training as its two components.Source: §III · p. 3

A. Training-in-deployment system

TIDE links robot execution, human intervention, recording, and online learning in one deployment session. Asynchronous control and episode-boundary checkpoint adoption keep actions flowing while preserving rollout consistency.

A. Training-in-deployment system

  • The system couples a robot client, inference-and-recording server, and online learner, with updated checkpoints returned at episode boundaries.The client streams executed-action records to the server, which makes them available to the learner.Source: A. Training-in-deployment system · p. 3
  • Asynchronous real-time chunking lets committed actions continue while future actions refresh in the background, reducing inference stalls and smoothing transitions.During takeover, VR delta control lets the operator command incremental motions and return control without resetting the episode.Source: A. Training-in-deployment system · pp. 3–4
  • Only manually confirmed successful episodes enter the online dataset, and the server adopts learner checkpoints at episode boundaries so each rollout uses one policy generation.Failed, timed-out, aborted, reset-before-completion, stale, and unrecovered episodes are discarded.Source: A. Training-in-deployment system · p. 4
  • The learner jointly updates the video diffusion backbone, cross-modal adapter/Q-former, action expert, and source embeddings while keeping the text encoder and video VAE frozen.It allocates 0.2 optimizer steps per newly committed control frame and publishes a checkpoint every five committed episodes.Source: A. Training-in-deployment system · p. 4

B. Provenance-conditioned learning

TIDE labels action chunks by whether they were executed by the human or policy, then conditions its online training objective on that provenance. Mixed-source horizons are excluded, while offline and online data and task-preserving augmentations are incorporated into training.

b) Provenance-labeled chunks

  • Human-executed chunks supervise difficult states and recovery, while policy-executed chunks add learning signal from behaviors the current policy can already perform.Chunk-level labels preserve the two sources within mixed deployment episodes rather than flattening each episode into an expert demonstration.Source: B. Provenance-conditioned learning · p. 4
  • Only horizons whose 50 actions all share one source receive a human or policy label; mixed-source horizons are excluded without an extra handoff window.Provenance identifies action source, not quality, and episode success only determines dataset inclusion.Source: B. Provenance-conditioned learning · pp. 4–5

b) Provenance-conditioned online training

  • The learner conditions both the video-generation backbone and action expert on a learnable embedding for each provenance label.Reported autonomous rollouts and final no-takeover evaluations fix the source condition to the human mode, ℓ=0.Source: B. Provenance-conditioned learning · p. 5
  • Online samples receive 5:1 weight over offline samples, while human- and policy-executed online chunks are balanced so sparse interventions are not overwhelmed.The fixed setting corresponds to buffer selection probabilities of 5/6 online and 1/6 offline.Source: B. Provenance-conditioned learning · p. 5
  • Space mirroring and time scaling augment samples only when task validity is preserved, and each augmented sample keeps its original provenance label.Mirroring expands left/right spatial configurations; time scaling exposes the model to moderate execution-rate variation.Source: B. Provenance-conditioned learning · p. 5

A. Experimental setup

Experiments use a bimanual UR5e platform across four manipulation tasks, with autonomous rollouts measuring milestone progress and full completion. TIDE is compared against matched offline demonstrations and a fixed-policy collection baseline, using the same warm-start checkpoint.

b) Tasks

  • Four bimanual tasks test precision contact, deformable manipulation, constrained placement, and recovery from local deformation or tangling.The tasks are Bit Insertion, Towel Folding, Towel Hanging, and Chain Packing.Source: A. Experimental setup · p. 5

c) Protocol and metrics

  • Each method–task pair receives 20 autonomous rollouts from the same predefined initial-state set, with no human takeover.Milestone progress is normalized by the task’s total milestones, and binary full-task completion is reported separately as k/20.Source: A. Experimental setup · p. 5
  • Timeouts, safety stops, and failures count only milestones already stably completed; intervention ratio measures the fraction of control steps under human takeover.Intervention ratio is used only as a within-task measure of supervision burden.Source: A. Experimental setup · p. 5

d) Baselines

  • All methods start from the same warm-start checkpoint; offline demonstrations are matched to online-data amount, while Fixed Policy delays training until collection ends.The Fixed Policy baseline matches TIDE’s collection duration, objective, buffer configuration, filtering rules, and optimizer-update budget.Source: A. Experimental setup · p. 6

e) Implementation details

  • Each task’s online post-training session lasts less than one hour from rollout start until the final checkpoint is deployable, including asynchronous training and checkpoint handling.Experiments use a 3B-parameter world-action policy on eight H100 GPUs; the interval excludes initial setup and final autonomous evaluation.Source: A. Experimental setup · p. 6

B. Real-world experiments

The experiments select joint-relative actions, compare autonomous no-takeover performance, track intervention burden, and test provenance-conditioned learning. TIDE performs comparably to Fixed Policy while outperforming Warm and Offline baselines on the reported task metrics.

Q1: Which action representation should be used

  • Joint-relative actions perform best among eight action spaces in a controlled offline-only comparison and are used in subsequent experiments.The comparison holds data, initialization, and training budget constant.Source: Figure 5 · p. 6, B. Real-world experiments · p. 6

Q2: Does TIDE improve autonomous task performance

  • Both Fixed Policy and TIDE outperform Warm and Offline on milestone-normalized progress across all four tasks; TIDE matches Fixed Policy on two tasks and exceeds it on the other two.Binary completion ties on Bit Insertion; TIDE completes one more rollout on each towel task and two more on Chain Packing.Source: Figure 6 · p. 6, B. Real-world experiments · p. 6
  • In illustrative no-takeover rollouts, the warm-start policy fails to complete the tasks, while final TIDE checkpoints complete the manipulation sequences.These examples are qualitative; the quantitative comparison uses 20-rollout evaluations.Source: Figure 7 · p. 7

Q3: How does observed supervision burden evolve during deployment

  • The observed windowed intervention ratio decreases after Bit Insertion’s early deployment phase, indicating lower supervision burden, but does not isolate learning from temporal factors.The ratio is the fraction of executed control steps under human takeover.Source: B. Real-world experiments · p. 7

Q4: How important is provenance-conditioned learning

  • On Bit Insertion, TIDE reaches 80.0% milestone-normalized task progress, versus 65.0% without provenance conditioning and 67.5% using only human-executed online chunks.This single-task ablation supports explicit source conditioning and learning from policy-executed chunks.Source: B. Real-world experiments · p. 7

V. CONCLUSION

TIDE makes real-robot deployment a learning process for pretrained manipulation policies, combining execution, takeover, online training, and checkpoint handoff. Across four tasks it improves over Warm and additional-offline-data baselines, performs comparably to Fixed Policy, and shows declining intervention ratio on Bit Insertion.

V. CONCLUSION

  • Across four real-world tasks, TIDE achieves higher milestone-normalized progress and binary completion than Warm and additional-offline-data baselines, with final performance comparable to Fixed Policy.Updated checkpoints become available during deployment; the observed intervention ratio also declines over the Bit Insertion session, indicating lower supervision burden rather than greater autonomy.Source: §V · p. 7
  • TIDE combines robot execution, human takeover, online data formation, joint optimization of trainable policy modules, and checkpoint handoff in one deployment session.Provenance-labeled chunks distinguish human-intervention from policy-executed behavior, supporting supervised online training without learned critics, trajectory-level reward labels, or action scorers.Source: §V · p. 7

VI. LIMITATIONS

The evaluation is limited in platform, task, and backbone coverage, and does not assess long-term retention or online-training schedule sensitivity.

VI. LIMITATIONS

  • Evaluation covers one bimanual platform, four tasks, and one world-action backbone; generalization across platforms and architectures remains unverified.The study also does not test long-term retention or sensitivity to the online-training schedule, or measure variation across repeated post-training sessions.Source: §VI · p. 7

Related papers

Loading 2610.29264v1…