PaperlayerView exact revision on arXiv

Source-linked AI summary

Steertuning Policies from a Few Human Videos

Billy Yan, Isaac Tu, Kelly Lee, Blake Chang, Kenny Ma, Kanad Patel, Dhawal Kabra, Neer Patel, Kevin Wu, Lerrel Pinto, Irmak Guzey

arXiv:2610.18711v1cs.ROcs.AI

TL;DR

Robot policies often require many demonstrations, while robust and rapid adaptation from human data alone to new environments remains unexplored. Steertuning pretrains on egocentric human data and combines few-shot fine-tuning with action steering; it reports 71% success across 12 tasks in 12 new environments.

  • Problem

    Robot behavior cloning can require hundreds or thousands of demonstrations per task, and robust, scalable adaptation from human data alone to new environments remains unexplored.

    Source: §I · p. 1

  • Method

    Steertuning pretrains a diffusion transformer on egocentric human data, then fine-tunes on five task demonstrations and steers sampled actions toward a selected human trajectory.

    Source: D. Pretraining and Steertuning… · p. 4, 1) Post-Training · p. 5

  • Results

    71% success across 12 tasks in 12 new environments is the reported Steertuning result.

    Source: §I · p. 2

  • Takeaways & Limitations

    Steertuning is an end-to-end mobile manipulation system that adapts to new tasks in new environments from human data in under five minutes.

    Source: §I · p. 1, A. Overview · p. 3

    Because human videos lack contact information, learned low-level actions remain relatively primitive, limiting dexterity.

    Source: §V · p. 8

Abstract

from arXiv · show

Current recipes for robot learning require massive amounts of robot demonstrations to obtain robust and adaptive policies. In this work, we introduce Steertuning, a system that enables robots to learn new tasks in new environments from 5 human demonstrations in under 5 Steertuning consists of two parts. (1) We pretrain a 400M-parameter diffusion policy on our egocentric human dataset, which consists of 100 hours of densely annotated random interactions. This pretraining makes Steertuning robust to background differences and enables spatial and object generalization when deployed on new tasks. (2) Given 5 demonstrations of a new task, we finetune our model for only 2 minutes (2000 training updates) and steer the policy's actions during reverse sampling using the demonstration whose observation is closest to that of the deployment environment. Together with a carefully designed data processing pipeline for embodiment transfer, this enables Steertuning to achieve a 71% success rate across 12 tasks in 12 new environments, ranging from lab settings to kitchens in new homes. The entire process, from data collection for a new task to robot policy deployment, takes less than 5 minutes. Our full dataset, model weights, codebase, and robot videos will be made available at our website: https://steertuning.github.io.

I. INTRODUCTION

Steertuning targets rapid adaptation from human data to new tasks and environments, where prior approaches either rely on many robot demonstrations or face trade-offs in human-video scale and annotation. It combines egocentric pretraining with brief fine-tuning and action steering, reporting success across 12 tasks and 12 new environments.

I. INTRODUCTION

  • The paper frames robust, scalable representation learning with rapid adaptation from human data alone as an unexplored problem.Steertuning is presented as an end-to-end system for adapting to new tasks in completely new environments.Source: §I · pp. 1–2
  • Existing robot behavior cloning can require hundreds or thousands of demonstrations per task, and large-model fine-tuning can still take days.The paper contrasts this burden with prior approaches that use pretrained policies and task-specific demonstrations.Source: §I · pp. 1–2
  • Prior human-data approaches trade calibrated environments and 3D annotations against the scalability of internet video, which often lacks deployment-relevant annotations.The introduction identifies smart glasses as a compromise that supports scalable data collection with annotations useful for robot deployment.Source: §I · p. 2
  • Steertuning pretrains on 100 hours of egocentric smart-glasses interactions, then adapts with five new-task demonstrations using two-minute fine-tuning and action steering.The full new-task adaptation process is described as taking under five minutes.Source: §I · p. 2
  • 71% success across 12 tasks in 12 new environments is the reported result for Steertuning.The system also generalizes when navigation and docking are handled by the mobile manipulator.Source: §I · p. 2
  • The system also deploys policies from a single language command alongside open-knowledge navigation and a VLM orchestrator.The reported generalization includes settings where the mobile manipulator handles navigation and docking.Source: §I · p. 2

A. Fine-Tuning and Steering Pretrained Policies

The paper situates Steertuning among methods that adapt pretrained policies by fine-tuning or steering, and among approaches that learn robot policies from human data. Its system combines these adaptation strategies with smart-glass data and mobile manipulation.

A. Fine-Tuning and Steering Pretrained Policies

  • Fine-tuning adapts pretrained policies using task-specific demonstrations, while inference-time steering modifies diffusion sampling without changing model weights.The paper notes that steering alone cannot teach behaviors the base policy does not know; Steertuning combines a two-minute LoRA fine-tune with demonstration-based steering.Source: A. Fine-Tuning and Steering Pretrained… · p. 2

B. Policy Learning from Human Data

  • Steertuning uses egocentric Aria Gen 2 data to adapt to new tasks in under five minutes and operate in new environments and homes.The paper contrasts this with human-video approaches that still require robot demonstrations for each new task.Source: B. Policy Learning from Human Data · pp. 2–3

A. Overview

  • Steertuning links egocentric human-data processing, policies that adapt to new tasks and environments, and open-knowledge navigation for language-commanded mobile manipulation.The overview presents these as three parts of an end-to-end system.Source: A. Overview · p. 3

B. Collecting and Processing Smart-Glass Data

The data pipeline collects egocentric smart-glass recordings for broad pretraining and task-specific adaptation, then segments long recordings into annotated interaction snippets. Language instructions and manipulated-object masks provide task-level context for learning.

1) Data Collection:

  • The pipeline uses 109.6 hours of everyday interactions for pretraining and five task-specific demonstrations per evaluation task for fine-tuning and steering.The recordings contain 3.69 million retained frames and 48.7 thousand auto-generated instructions; both data sources were collected with Aria Gen 2 glasses at 10 Hz.Source: 1) Data Collection · p. 3
  • Aria Gen 2 supplies synchronized RGB and stereo images, head pose, hand landmarks, and wrist pose, with metric depth recovered from stereo.The point cloud, hand trajectories, and head pose are expressed in the camera frame of the last observed frame for each snippet.Source: 1) Data Collection · p. 3

2) Task Segmentation:

  • Long recordings are split into short interaction snippets using hand–scene contact, then annotated with language and the manipulated object.Contact-based segmentation uses tracked hand landmarks near the scene; language annotations are verb-first instructions, and object masks are initialized from keypoints and propagated through each snippet.Source: 2) Task Segmentation · p. 3

3) Visual Domain Alignment:

The policy reduces the human-to-robot visual embodiment gap by removing embodiment geometry and representing observations and actions in a common camera frame. A diffusion transformer conditions action denoising on multimodal observations, while deployment steers predicted actions toward a human trajectory.

3) Visual Domain Alignment:

  • Removing human or robot arm geometry from the point cloud makes the policy observe the surrounding scene in both training and deployment.The pipeline also crops and resamples the point cloud to produce a consistent representation across embodiments.Source: 3) Visual Domain Alignment · p. 3

C. Observation and Action Representation

  • Training snippets use the anchor frame’s RGB camera coordinates, and each action represents 52 3D points over 15 future steps at stride three.The points encode two hands and the head; at the 10 Hz collection frequency, each action chunk covers 4.5 seconds.Source: C. Observation and Action Representation · p. 3

D. Pretraining and Steertuning Policies from Human Videos

  • At inference, deterministic DDIM sampling uses eight steps, and deployment steers predicted actions toward the nearest human trajectory.During fine-tuning, LoRA adapters are trained for 2,000 steps.Source: D. Pretraining and Steertuning… · p. 4, Figure 3 · p. 4
  • Observation tokens condition diffusion action prediction through both an AdaLN context token and cross-attention with the full observation sequence.The policy encodes point cloud, object mask, fingertip positions, and task text; Utonia and SigLIP are frozen while other components are trained end-to-end to predict noise.Source: D. Pretraining and Steertuning… · p. 4, Figure 3 · p. 4

1) Post-Training:

Steertuning adapts a pretrained policy by combining brief LoRA fine-tuning with demonstration-guided action steering. It selects a demonstration based on the current scene and reports a 71% overall success rate across 12 tasks.

1) Post-Training:

  • The best adaptation recipe combines LoRA fine-tuning with steering the action-noise distribution during reverse sampling using task demonstrations.The authors compare the effects of fine-tuning and steering on downstream robot tasks.Source: 1) Post-Training · p. 4
  • LoRA freezes pretrained weights, adds rank-8 adapters, and fine-tunes for 2,000 steps in two minutes; augmentations broaden the limited in-scene data.Augmentations include transformations to the anchor frame and proprioception, point-cloud RGB jitter, and text paraphrases.Source: 1) Post-Training · p. 4
  • During reverse sampling, the selected demonstration action is noised to the current diffusion level, shifted using a steering weight, and followed by standard DDIM sampling.Steering uses the human actions from the demonstration whose initial frame is closest to the current observation.Source: 1) Post-Training · pp. 4–5
  • 71% (7.1/10) overall success across 12 tasks, spanning nine new in-the-wild tasks and three tasks used for ablations.Per-task success rates are reported on the project website.Source: Figure 4 · p. 5
  • A single command such as putting a cup in the trash is decomposed into navigation, pickup, navigation, and drop subtasks.Pickup and drop demonstrations are collected and deployed as in Steertuning.Source: Figure 5 · p. 5
  • The policy selects the closest human trajectory by comparing object layouts in a gravity-aligned manipulated-surface frame.The layouts come from segmented table objects represented by their camera-relative 3D centroids; the selection has a significant impact on success rate.Source: 1) Post-Training · p. 5

1) Embodiment Transfer:

The deployment pipeline transfers human-trained policies to a YOR robot by removing embodiment geometry from observations and retargeting predicted motions to its hardware.

1) Embodiment Transfer:

  • The YOR robot uses Wuji hands; deployment removes the robot arm and hand from point-cloud observations, then maps predicted wrist poses and landmarks to the robot through inverse kinematics and retargeting.Real-time chunking temporally smooths predictions to minimize jitter.Source: 1) Embodiment Transfer · pp. 5–6
  • Predicted wrist poses are executed with Cartesian inverse kinematics, while kinematic retargeting makes the Wuji hands follow predicted landmarks relative to the wrists.Source: 1) Embodiment Transfer · p. 5
  • The scaling ablations vary pretraining and LoRA compute budgets, pretraining data hours, and the number of LoRA demonstrations; plotted values are mean success rates over three tasks with ±1 standard deviation bars.Steertuning settings are marked with an asterisk.Source: Figure 6 · p. 6

2) End-to-End Deployment:

Steertuning is integrated into a language-command deployment platform that separates navigation from manipulation. Experiments evaluate new-task deployments and report 71% overall success across 12 tasks in new environments.

2) End-to-End Deployment:

  • A language-command platform decomposes long-horizon tasks into navigation and manipulation; navigation uses a voxelized SLAM map linked to SAM3 object latents before the manipulation policy runs.The system uses Gemini ER 1.6 to produce the subtasks and follows the navigation approach in [22].Source: 2) End-to-End Deployment · p. 6

IV. EXPERIMENTS

  • The experiments ask how fine-tuning, action steering, training budgets, architectural components, trajectory selection, and object generalization affect policy performance.Source: §IV · p. 6

A. Task Descriptions

  • 71% (7.1/10) overall success across 12 tasks: nine in a rented Airbnb and three in the lab, each using five new human demonstrations.Ablations were run on the three lab tasks.Source: A. Task Descriptions · p. 6

B. What is the impact of fine-tuning and action steering on quick task adaptation?

Combining fine-tuning and action steering gives robust, fast few-shot adaptation: Steertuning outperforms the tested single-component baselines, while brief fine-tuning with steering approaches the error of a much longer training run.

B. What is the impact of fine-tuning and action steering on quick task adaptation?

  • 2k fine-tuning steps with steering weight w = 0.3 reach a minimum held-out landmark error close to 128k-step fine-tuning at a small fraction of the cost.At w = 1, DDIM sampling reduces to replaying the demonstration directly.Source: B. What is the impact of fine-tuning… · p. 6, Figure 7 · p. 6
  • Steering more strongly toward the selected demonstration brings mean action error toward the error of replaying that demonstration.Fig. 7 reports mean landmark error averaged over 12 tasks, with held-out prediction error measured against ground-truth action landmarks.Source: B. What is the impact of fine-tuning… · p. 6, Figure 7 · p. 6
  • Steertuning outperforms fine-tuning-only and steering-only baselines; the 150k-step fine-tuning baseline reaches comparable success but takes a day, while steering alone succeeds in only 23% of trials.The same 2k-step fine-tuning used by Steertuning fails completely on its own.Source: Table I · p. 7, B. What is the impact of fine-tuning… · p. 7

steps and data) on downstream robot policies?

Success rates increase with compute budget, but the selected recipe uses maximum-budget pretraining and the cheapest fine-tuning checkpoint that retains high success. The tested architecture and trajectory-selection ablations favor Utonia with DiT and object-based selection, respectively.

steps and data) on downstream robot policies?

  • Success rate increases with compute budget; the study uses the maximum pretraining budget and the cheapest fine-tuning checkpoint that still achieves high success.The fine-tuning checkpoint choice keeps task adaptation as fast as possible.Source: steps and data) on downstream robot… · p. 7

D. How well does Steertuning perform when ablated against different architectural components?

during action steering impact downstream task performance?

  • Object-based trajectory selection outperforms VINN and wrist-pose selection when all methods use identical steering after selecting a trajectory.Steertuning selects the demonstration by comparing deployment-time object layout with each demonstration.Source: Table III · p. 7, during action steering impact… · p. 7

F. How well does Steertuning generalize to new objects?

Steertuning remains robust to deployment-time object changes when the required action remains similar to the demonstrated one. Its dexterity is limited by visually annotated actions without contact information, and robot execution can also fail because of retargeting errors.

F. How well does Steertuning generalize to new objects?

  • Steertuning remains robust to visual object changes when the required action does not differ significantly from the demonstrated action.The authors attribute this robustness to large-scale pretraining and fine-tuning augmentations.Source: F. How well does Steertuning… · p. 7

V. DISCUSSION AND LIMITATIONS

  • Visual-only action and hand-pose annotations leave low-level actions relatively primitive, limiting the system's dexterity.The authors suggest contact information or RL-based adaptation could extend the range of learnable tasks.Source: §V · p. 8
  • Kinematic retargeting errors can make the robot fail even when human-policy predictions are accurate.Source: §V · p. 8

Related papers

Loading 2610.18711v1…