Source-linked AI summary
Learning to Act from Actionless Videos through Dense Correspondences
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, Joshua B. Tenenbaum
TL;DR
The paper addresses learning robot policies for diverse tasks and embodiments from videos without action annotations. AVDC synthesizes task-execution videos and uses dense correspondences and depth to infer actions, achieving diverse simulated and real-world execution, including a 90% zero-shot success rate in cross-embodiment transfer. Its scope is limited by failures under occlusion, challenging visual conditions, and the need for additional grasp or contact prediction.
Problem
Robot learning must accommodate substantially different state and action representations across diverse tasks, robots, environments, and goals, while existing video approaches rely on task-specific action labels.
Method
AVDC synthesizes a video from an RGBD observation and textual goal, estimates dense optical-flow correspondences, and combines them with initial depth to recover transformations and robot actions.
Results
AVDC performs simulated and real-world manipulation and navigation tasks, including 90% zero-shot success across 40 runs when transferring from human videos to robot execution.
Takeaways & Limitations
The approach supports action-free policy learning from RGB videos and provides an open-source framework enabling efficient video-policy training on 4 GPUs in a single day.
Takeaways & Limitations
Optical-flow tracking can fail under severe arm occlusion, rapidly changing lighting, or large object movements, while manipulation may require separate grasp or contact-surface prediction.
Abstract
from arXiv · showhide
In this work, we present an approach to construct a video-based robot policy capable of reliably executing diverse tasks across different robots and environments from few video demonstrations without using any action annotations. Our method leverages images as a task-agnostic representation, encoding both the state and action information, and text as a general representation for specifying robot goals. By synthesizing videos that ``hallucinate'' robot executing actions and in combination with dense correspondences between frames, our approach can infer the closed-formed action to execute to an environment without the need of any explicit action labels. This unique capability allows us to train the policy solely based on RGB videos and deploy learned policies to various robotic tasks. We demonstrate the efficacy of our approach in learning policies on table-top manipulation and navigation tasks. Additionally, we contribute an open-source framework for efficient video modeling, enabling the training of high-fidelity policy models with four GPUs within a single day.
1 INTRODUCTION
The paper targets robust policies across diverse robots and tasks without action labels by synthesizing task videos and recovering actions from dense correspondences. It also contributes an efficient open-source framework for video policy modeling.
- Diverse robotic tasks require policies to handle substantially different state and action representations across robots, environments, and goals.
- Existing video-based approaches either learn action-specific predictors or task-specific inverse dynamics, both relying on task-specific action labels.
- AVDC synthesizes desired task-execution videos, then infers actions from rigid 3D object or point transformations estimated using optical flow and segmentation.
- The approach is illustrated across table-top assembly, ego-centric object navigation, and real-world robot manipulation tasks.
- Over 256 TPU pods are required by a closely related video-policy approach, whereas the authors’ framework trains high-fidelity models on 4 GPUs in a single day.
- The contributions combine action inference without action labels, policies for manipulation and navigation, and an open-source efficient video-modeling framework.
2 RELATED WORK
Prior work studies robot learning from videos, learning from observation, and dense correspondences as alternatives to action-annotated imitation learning. These lines of work provide visual representations, expert state sequences, or implicit action and pose parameterizations.
- Video-based robot learning includes methods that construct visual representations from existing video datasets.
- Dense correspondences have emerged as implicit parameterizations of actions and poses in both 2D and 3D.
- Learning from observation differs from imitation learning by using expert state sequences, such as video frames, instead of expert actions.
- AVDC is presented as a framework combining video-based learning with dense correspondences for action inference.
3 ACTIONS FROM VIDEO DENSE CORRESPONDENCES
AVDC synthesizes task-conditioned videos from an RGBD observation and text, extracts dense correspondences through optical flow, and converts them into 3D object or camera transformations. These transformations are then used with robotics primitives to infer manipulation and navigation actions, while architectural optimizations make video generation practical on four GPUs within one day.
- Framework: AVDC takes an initial RGBD scene observation and textual task description, then synthesizes a video that implicitly captures required actions.
- Video Synthesis: The conditional diffusion model predicts a fixed sequence of future frames, using the initial frame and text description as conditions.In the experiments, the model predicts T = 8 future frames.
- Video Synthesis: The video model uses a U-Net with factorized spatial-temporal convolutions to improve training and inference efficiency without sacrificing generation quality.
- Dense Correspondences: GMFlow estimates optical flow between consecutive synthesized frames, producing pixel-level dense correspondences used to track scene and object movements.
- 3D Motion Reconstruction: Given initial-frame depth and tracked pixels, AVDC reconstructs sequences of 3D rigid transformations for objects or camera motion.Object-centric motion assumes a fixed camera, while navigation uses the duality between object and camera motion.
- Action Inference: The inferred transformations are converted into robot actions using grasp sampling, subgoal computation, and position control, without directly learning an inverse-dynamics action regressor.
4 EXPERIMENTS
Experiments evaluate AVDC against behavioral-cloning and video-based baselines across simulated manipulation and navigation, plus cross-embodiment and real-scene transfer. Results support dense-correspondence action inference, closed-loop replanning, and actionless video learning, while optical-flow errors remain a primary failure mode.
- 4.2 Meta-World: AVDC consistently outperforms BC-Scratch, BC-R3M, and UniPi across Meta-World manipulation tasks, despite the BC baselines receiving expert actions.Meta-World uses 11 tasks, three camera poses, and randomized episode initializations.
- 4.2 Meta-World: AVDC’s closed-loop variant outperforms AVDC (No Replan), and success rates increase consistently as the maximum number of replanning trials grows.These comparisons support replanning when execution fails.
- 4.2 Meta-World: AVDC synthesizes task videos, predicts optical flow, and infers actions that guide the robot arm through the Meta-World assembly task.Qualitative visualizations show the synthesized execution, predicted arm motion, and resulting action guidance.
- 4.6 Failure Modes: Optical-flow tracking errors are the primary observed failure mode, with low resolution making small-object pixel errors produce large 3D reconstruction errors.The authors suggest higher-resolution synthesis or an in-domain flow model as potential improvements.
- 4.3 iTHOR: In iTHOR navigation, AVDC achieves a 31.3% average success rate, while both behavioral-cloning baselines fail to achieve meaningful performance.Policies navigate to randomized target objects across four room types and 12 tasks.
- 4.4 Cross-Embodiment Learning: AVDC achieves a 90% zero-shot success rate over 40 simulated pushing runs, transferring from out-of-domain human videos to robot execution without fine-tuning.The result is reported as cross-embodiment learning from human videos.
- 4.5 Bridge: The Bridge-trained video diffusion model generalizes to real scenes without fine-tuning, while the full pipeline synthesizes videos, predicts flow, identifies objects, and infers actions.Successful robot trajectories are shown qualitatively on Bridge scenes.
5 DISCUSSION
AVDC learns actions from RGB video by using dense correspondences between synthesized frames, and demonstrates applicability across simulated and real manipulation and navigation tasks. Its limitations include occlusion, challenging visual conditions, unavailable force information, and robot-specific grasp or contact requirements.
- Object tracking may fail when the robot arm occludes most of an object, while rapidly changing lighting or large pose movements can impair optical-flow prediction.
- Real-world manipulation still requires grasp or contact-surface prediction, and force information cannot be obtained from RGB videos.
- AVDC infers transforms on objects or surroundings from dense correspondences between synthesized frames to regress actions directly in environments.The method learns from RGB video demonstrations without action annotations.
- The approach is demonstrated on both simulated and real-world manipulation and navigation tasks.
- The open-source implementation supports efficient video-policy training in academic and industry settings.
B EXTENDED QUALITATIVE RESULTS
The supplementary results cover synthesized videos and task-execution videos across Meta-World, iTHOR, Visual Pusher, Bridge, and real-world Franka Panda tasks. Bridge-trained video diffusion also produces reasonable videos for complex real-world kitchens despite toy-kitchen training data.
- The supplementary website includes synthesized-video results for Meta-World, iTHOR, Visual Pusher, and Bridge.
- Task-execution videos span Meta-World, iTHOR, Visual Pusher, and real-world Franka Emika Panda tasks.
- A Bridge-trained video diffusion model produces reasonable videos for complex real-world kitchens from RGB images and textual task descriptions.Most Bridge tasks were recorded in toy kitchens.
- Figure 11 presents successful and failed object masks extracted by Language Segment Anything.
D OBJECT MASK WITH SEGMENTATION MODELS
The segmentation study replaces Meta-World ground-truth object masks with masks predicted by Language Segment-Anything and measures the resulting task success rate.
- 34.5% average success across 11 Meta-World tasks was achieved using predicted object masks.This experiment used Language Segment-Anything instead of ground-truth object masks.
- The predicted-mask setup scored 8.6% lower than the comparison condition reported in the experiment.
E ADDITIONAL ABLATION STUDIES
The ablations evaluate first-frame conditioning and text-encoder choices using pixel-level video-generation error. RGB-channel-wise concatenation outperforms frame-wise concatenation early in Bridge training, while the two text encoders perform similarly.
- First-frame conditioning: The RGB-channel-wise conditioning strategy, cat c, consistently outperforms frame-wise concatenation, cat t, during early Bridge training.The comparison uses MSE between ground-truth and synthesized-video last frames.
- First-frame conditioning: Each conditioning-strategy data point averages MSE over 4000 video-generation samples, with error bars showing the standard error.
- Text encoders: The 63M-parameter CLIP-text encoder and 110M-parameter T5-base encoder show no significant difference in video-generation quality.Quality is evaluated using pixel-level MSE.
F MODEL ARCHITECTURE AND TRAINING DETAIL
The model uses fixed CLIP-Text embeddings pooled by a Perceiver and added to diffusion time embeddings, with specified training hyperparameters and Perceiver configurations.
- Model conditioning: Text descriptions are encoded with a fixed CLIP-Text encoder, pooled into one vector by a Perceiver, and added to the diffusion model’s time embedding.The model does not use cross-attention for text inputs.
- Training configuration: Training uses 100 timesteps, v-prediction, a cosine beta schedule, L2 loss, learning rate 1e-4, and EMA decay 0.999.Other listed settings include dropout=0, min SNR gamma=5, and EMA update steps=10.
G.1 TRAINING COST
The framework trains environment-specific video policy models on four V100 GPUs, completing smaller datasets in about a day while requiring longer training for Bridge. Inference includes video synthesis, flow prediction, action regression, and execution, with planning costs that rise with replanning.
- Training cost: All models are trained on 4 V100 GPUs with 32GB memory each.The one-day training claim applies to diffusion models trained from scratch on small, environment-specific datasets.
- Training cost: Bridge training requires about two days for consistent, high-quality results, followed by 4 hours of fine-tuning on 20 human videos for the real-world experiment.The real-world setup uses 40,000 Bridge videos and 20 human videos.
- Training cost: Meta-World and iTHOR each require about 24 hours of training, using 165 and 240 videos respectively.
- Inference cost: Inference experiments were conducted on an RTX 3080Ti GPU, with a detailed Meta-World runtime breakdown provided for each method step.
- Inference cost: Text-conditioned video generation is the most time-consuming inference step at roughly 10.57 seconds, compared with 0.28 seconds for flow prediction and 1.31 seconds for action regression.
- Inference cost: 18 seconds per action-planning round yields a total Meta-World planning cost of about 18–108 seconds across 1–6 rounds.The range reflects a maximum of five replans.
G.4 IMPROVING INFERENCE EFFICIENCY WITH DENOISING DIFFUSION IMPLICIT MODELS
The method can accelerate video-policy inference by reducing diffusion sampling steps with DDIM, while its broader pipeline infers actions from generated videos through correspondences, transformations, and replanning.
- Inference acceleration: 10-step DDIM provides a 10x video-generation speedup while retaining satisfactory generated-video quality.
- Inference acceleration: 37.5% overall Meta-World success with 10-step DDIM is competitive with the original 100-step method’s 43.1% success rate.
- Inference acceleration: The resulting trade-off is a 10x faster video-generation step with a 5.6% drop in task performance for time-critical tasks.
- Action inference: The policy synthesizes videos from an initial frame and natural-language task description, using dense correspondences to infer subgoals and robot actions.
- Action inference: In Meta-World, object masks and depth support rigid-transformation estimation, while a 10cm vertical-displacement heuristic selects grasp versus push actions.
- Replanning: Closed-loop replanning regenerates videos from the current state and recomputes subgoals; iTHOR replans when retained correspondences fall below 10% of the original sample.
- Navigation: For iTHOR navigation, scene correspondences support scene transformations that are mapped to MoveForward, RotateLeft, RotateRight, or Done actions.
H.3 REAL-WORLD FRANKA EMIKA PANDA ARM EXPERIMENTAL SETUP
The real-world evaluation uses a Franka Panda arm with a fixed RGBD camera and fine-tuning on human demonstrations. It assumes top-graspable objects without reorientation, and most tested trials failed because of planning or video-continuity errors.
- Hardware: The setup uses a Franka Emika Panda arm, parallel two-jaw gripper, joint-position control, and a calibrated Intel Realsense D435 RGBD camera fixed relative to the table.
- Dataset and task: The model is fine-tuned with 20 human demonstrations in a tabletop setting containing fruits, dishes, and utensils as distractors.The task is to pick up fruits and place them at target locations.
- Assumptions: Real-world action inference assumes the target object supports a top-grasp and requires no reorientation.Object segmentation is manually specified for computing poses in this evaluation.
- Failure analysis: 8 of 10 real-world trials failed; 75% of failures came from incorrect video-diffusion plans and 25% from discontinuous generations in which objects disappeared.