PaperlayerView exact revision on arXiv

Source-linked AI summary

Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation

Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, Shubham Tulsiani

arXiv:2405.01527v2cs.ROcs.CV

TL;DR

Generalizable robot manipulation needs direct execution in unseen scenarios without test-time training, while existing approaches rely heavily on diverse robot demonstrations. Track2Act learns goal-conditioned point tracks from web videos, converts them into robot plans, and refines execution with a residual policy trained on limited embodiment-specific data. Real-world results show broad generalization across unseen tasks, objects, and scenes, while the demonstrated setting remains short-horizon and single-object.

  • Problem

    Directly deployable manipulation in new scenarios requires generalization without test-time demonstrations, but collecting diverse real-world robot interaction data is challenging and existing scaled datasets remain task-limited.

  • Method

    Track2Act predicts goal-conditioned point tracks from diverse web videos, converts them into 3D object transforms and robot poses, and learns residual corrections from limited embodiment-specific data.

  • Results

    Real-world Spot experiments show broad generalization across diverse tasks involving unseen objects and unseen scenes with a common goal-conditioned policy.

  • Takeaways & Limitations

    Passive web videos can provide embodiment-agnostic interaction plans that support diverse real-world manipulation without large-scale in-domain datasets.

  • Takeaways & Limitations

    The demonstrated tasks are short-horizon and involve manipulating a single object, while higher-generalization settings can still fail through grasping, infeasible motion, or goal mismatch.

Abstract

from arXiv · show

We seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation: interacting with unseen objects in novel scenes without test-time adaptation. While typical approaches rely on a large amount of demonstration data for such generalization, we propose an approach that leverages web videos to predict plausible interaction plans and learns a task-agnostic transformation to obtain robot actions in the real world. Our framework,Track2Act predicts tracks of how points in an image should move in future time-steps based on a goal, and can be trained with diverse videos on the web including those of humans and robots manipulating everyday objects. We use these 2D track predictions to infer a sequence of rigid transforms of the object to be manipulated, and obtain robot end-effector poses that can be executed in an open-loop manner. We then refine this open-loop plan by predicting residual actions through a closed loop policy trained with a few embodiment-specific demonstrations. We show that this approach of combining scalably learned track prediction with a residual policy requiring minimal in-domain robot-specific data enables diverse generalizable robot manipulation, and present a wide array of real-world robot manipulation results across unseen tasks, objects, and scenes. https://homangab.github.io/track2act/

1 Introduction

Track2Act targets directly deployable, generalizable manipulation by learning embodiment-agnostic interaction plans from web videos and converting them into robot actions. A residual policy trained with limited robot data refines these plans for closed-loop execution across unseen tasks, objects, and scenes.

  • 1 Introduction: Directly executable manipulation is motivated by repeated, safe deployment without test-time demonstrations, self-practice, or exploratory actions.The target application includes everyday manipulation across offices and kitchens.
  • 1 Introduction: Track2Act factorizes manipulation into web-video-learned interaction plans and a residual policy using limited embodiment-specific robot data.The plan predicts how image points move toward a goal, while residual actions correct execution errors.
  • 1 Introduction: Point tracks encode future 2D locations from an initial image, goal image, and sampled points, then support 3D rigid-transform and end-effector-pose inference.Depth from the initial scene is used to recover object transforms for robot execution.
  • 1 Introduction: The framework uses diverse human and robot web videos to learn embodiment-agnostic plans without deployment-robot data or online exploration.The contributions describe obtaining 3D rigid transforms directly and adding approximately 400 embodiment-specific demonstrations for residual correction.
  • 1 Introduction: Real-world Spot experiments show broad generalization across diverse tasks with unseen objects and scenes using a common goal-conditioned policy.The stated results emphasize the potential of passive web videos for learning interaction plans without large-scale in-domain manipulation datasets.

2 Related Works

Related work spans video-based interaction understanding, visual correspondence and tracking, visual imitation, and attempts to leverage non-robot datasets for manipulation. Track2Act builds on these directions by predicting point tracks from web videos for robot interaction planning.

  • 2 Related Works: Prior work studies hand-object interactions through large video datasets, hand and object pose estimation, and interaction hotspot or grasp prediction.These approaches address visual understanding of everyday activities and manipulation-relevant interactions.
  • 2 Related Works: Track2Act builds on video tracking methods that generate point correspondences across frames and adapts them to predict future tracks conditioned on an image goal.Off-the-shelf tracking approaches provide ground-truth tracks from web videos for training.
  • 2 Related Works: Visual imitation and large-scale robot datasets support manipulation learning but require many expert trajectories and remain limited in generalization to unseen objects and scenarios.Related approaches often demand years of robot-data collection.
  • 2 Related Works: Other work generates videos from task descriptions and scenes, while Track2Act instead uses predicted point motion to infer robot actions.The supplied related-work passages position track prediction as the basis for the paper's interaction-plan representation.

3 Approach

Track2Act factorizes manipulation into web-trained, embodiment-agnostic point-track prediction and a residual policy that converts and corrects the resulting robot plan. It predicts object motion from 2D tracks, infers rigid transforms and open-loop end-effector poses, then uses embodiment-specific corrections for closed-loop deployment.

  • Overview and Setup: Track2Act factorizes manipulation into an embodiment-agnostic interaction plan from point tracks and a residual policy for robot action correction.The factorized design combines web-video learning with limited embodiment-specific robot data.
  • Point Track Prediction from Web Videos: The diffusion track predictor generates future locations for randomized initial image points conditioned on the initial image and goal.Training uses variable numbers and locations of points, without assumptions about tracked objects or camera motion.
  • Inferring Coarse Manipulator Trajectory from Interaction Plan: Predicted moving-point tracks and initial depth are used to estimate per-time object rigid transforms whose projections match the 2D trajectories.The transforms are embodiment-agnostic and are estimated from multiple 3D points using projection constraints.
  • Inferring Coarse Manipulator Trajectory from Interaction Plan: The estimated transforms produce an open-loop end-effector trajectory after an initial approach and grasp of the manipulated object.The first pose moves the end-effector to the center of the initial 3D points, while later poses apply the inferred transforms.
  • Closed-loop Manipulation with Residual Policy Correction: A closed-loop residual policy predicts per-timestep corrections that are added to the open-loop end-effector poses to address prediction and contact errors.The correction policy is trained with embodiment-specific data, while the prediction model is intended to generalize to unseen scenarios.

4 Experiment Setup

The experiments evaluate Track2Act on web and robot-video data, using point-track accuracy and robot success across defined generalization levels. Comparisons include flow, video, behavior-cloning, affordance, hand-object-mask, and open-loop action baselines.

  • Robot Setup: Robot experiments use a Spot mobile manipulator with a front-facing RealSense camera and end-effector control.
  • Evaluation Metrics: Track prediction is evaluated against Co-Tracker tracks using the fraction of points within a pixel threshold, averaged over time and thresholds from 1 to 10 pixels.
  • Evaluation Metrics: Robot success requires the manipulated object's final pose to match the pose in the goal image.
  • Generalization Protocol: Generalization spans mild, standard, combinatorial, and type settings, from unseen configurations of seen objects to unseen object types or activities in unseen scenes.
  • Baselines: Baselines include goal-conditioned, affordance-conditioned, video-conditioned, and hand-object-mask-conditioned behavior cloning, while the open-loop Track2Act variant uses no embodiment-specific training data.
  • Training Data: Training uses passive human and robot videos, with Co-Tracker applied to 400,000 clips to obtain ground-truth point tracks.

5 Results

Track2Act predicts plausible point tracks across unseen videos and converts them into successful robot manipulation through residual closed-loop correction. Across generalization levels, the residual policy improves over open-loop execution and outperforms competing approaches, although the hardest settings remain challenging.

  • 5.1 Point Track Prediction Results: Qualitative predictions correspond to object manipulation goals, while stationary background points support estimating rigid object transforms when the camera is fixed.
  • 5.1 Point Track Prediction Results: Track prediction achieves higher accuracy than flow-based and video-based baselines because direct point tracks capture nonlinear motion without requiring implausible RGB-video generation.
  • 5.2 Robot Manipulation Results: Track2Act's residual policy outperforms open-loop rigid-transform execution by correcting inaccurate grasps and recovering from trajectory failures.
  • 5.2 Robot Manipulation Results: Goal-conditioned behavior cloning is slightly worse than the residual policy under mild generalization and substantially worse or unsuccessful under standard, compositional, and type generalization.
  • 5.2 Robot Manipulation Results: Web-video baselines using affordances, generated videos, or hand-object masks provide insufficient cues for unseen manipulation compared with track-based inference of 3D rigid transforms.
  • 5.3 Analysis of Failures: Higher-generalization compositional and type settings remain difficult because the residual policy can grasp incorrectly, attempt infeasible motions, or violate the goal.

6 Discussion and Conclusion

Track2Act combines web-video-learned, embodiment-agnostic manipulation plans with a small embodiment-specific residual policy to generalize across diverse real-world tasks. Its current scope is short-horizon manipulation of a single object, leaving multi-object long-horizon tasks for future work.

  • Discussion and Conclusion: The framework combines large-scale web-video plans with a small amount of embodiment-specific data for closed-loop residual correction.
  • Discussion and Conclusion: The demonstrated tasks are short-horizon and manipulate a single object, motivating future extension to successive manipulations of multiple objects.

Appendix

The appendix documents the track-prediction setup, including DiT-based diffusion, point-track denoising, video assumptions, and Spot robot evaluation details.

  • Appendix: Experiments use a Spot robot with end-effector control, a base-mounted front-facing RealSense camera, a two-fingered gripper, and 50-step rollouts at 5 Hz.
  • 6.3 Track Prediction Model details: Track prediction is implemented as a DiT-based denoising diffusion process conditioned on initial and goal frames.The model predicts future locations for points selected in the initial frame.
  • 6.3 Track Prediction Model details: The training setup uses 4–5-second clips, with the first frame as input and the last frame as the goal for longer videos.
  • 6.3 Track Prediction Model details: The model makes no assumptions about tracked objects or camera motion and uses minimally curated videos beyond clip duration.
  • 6.3 Track Prediction Model details: The track-prediction model uses 24 DiT blocks, hidden size 1024, 16 heads, and 512-dimensional image embeddings.

6.4 Residual Policy Model details

The residual policy predicts future corrections to the open-loop end-effector plan and executes the first predicted correction using a small robot-demonstration dataset.

  • 6.4 Residual Policy Model details: The residual policy corrects predicted end-effector poses by adding a learned residual action to each open-loop pose.The corrected pose is defined as ˆa_t = ¯a_t + ∆a_t.
  • 6.4 Residual Policy Model details: Instead of predicting one correction, the policy predicts residuals several steps ahead and executes only the first action to mitigate compounding errors.
  • 6.4 Residual Policy Model details: The residual policy is trained from approximately 400 robot trajectories containing camera observations and end-effector-pose actions.
  • 6.4 Residual Policy Model details: The residual model is a DiT-based Transformer with 12 blocks, hidden size 512, 8 heads, and image embeddings of dimension 512.

6.5 Training Data for Track Prediction

Track prediction is trained from approximately 400,000 clips drawn from four web-video sources, with Co-Tracker tracks providing supervision for randomly selected initial-frame points.

  • 6.5 Training Data for Track Prediction: The training set combines Something-Something-v2, Epic-Kitchens, RT1, and BridgeData videos.
  • 6.5 Training Data for Track Prediction: Approximately 400,000 clips are processed with 400-point grids, and Co-Tracker supplies ground-truth intermediate tracks for supervision.

6.6 Training Data for Residual Policy

The residual policy uses 400 teleoperated Spot trajectories from 10 tasks across three locations, while evaluation includes broader task, object, and scene generalization.

  • 6.6 Training Data for Residual Policy: Residual-policy demonstrations were collected with a joystick-controlled Spot robot across 10 tasks in three physical locations.
  • 6.6 Training Data for Residual Policy: Evaluation scenarios matching the demonstration tasks define mild generalization, while other axes involve unseen instances and categories.
  • 6.6 Training Data for Residual Policy: The embodiment-specific dataset is 3–4 orders of magnitude smaller than datasets required by related policy-learning approaches.

6.7 Details on baselines

The study compares Track2Act against goal-conditioned and web-data-based baselines using the same embodiment-specific demonstrations. These comparisons isolate the potential benefits of web data and point-track prediction for generalizable manipulation.

  • Goal-Conditioned BC uses the same residual-policy training data and conditions on the goal image, testing the contribution of web data to generalization.
  • Affordance-Conditioned BC predicts manipulable-scene affordances, whereas Track2Act predicts time-series point tracks for manipulation.
  • Video-Conditioned BC predicts RGB video before tracking, while Hand-Object Mask Conditioned BC predicts hand-object plans; both use web-trained predictors and embodiment-specific data.The hand-object baseline conditions on predicted plans rather than paired ground-truth human-robot plans.
  • The experiments compare Track2Act with goal-conditioned behavior cloning and three web-data-based prediction approaches for robot manipulation.The web-data baselines use affordances, predicted RGB video, or hand-object interaction plans.

6.8 Qualitative Results for baselines

The paper presents qualitative rollout comparisons across standard, mild, type, and compositional generalization, alongside visualizations of alternative prediction baselines.

  • Detailed qualitative video results for Track2Act are provided on the project website.
  • Qualitative comparisons show baseline and Track2Act rollouts for the same goals under type, compositional, standard, and mild generalization settings.The rollout views use a third-person camera.
  • The paper also visualizes Hand-Object Mask Prediction and Affordance Prediction outputs across different initial and goal images in the robot environment.
Loading 2405.01527v2…