PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation

Mel Vecerik, Carl Doersch, Yi Yang, Todor Davchev, Yusuf Aytar, Guangyao Zhou, Raia Hadsell, Lourdes Agapito, Jon Scholz

arXiv:2308.15975v2cs.ROcs.AIcs.CV

TL;DR

Robotics needs ways to teach new behaviors quickly without task-specific engineering or large data requirements. RoboTAP uses dense point tracking to extract demonstrated motions and guide a low-level controller; the authors report solving novel visuomotor tasks from a few minutes of robot interaction, with limitations in motion planning and replanning.

  • Problem

    Dense tracking has not yet been explored as a manipulation primitive, while existing approaches may lack either task generality or data efficiency.

    Source: §I · p. 1

  • Method

    RoboTAP uses dense tracking to identify relevant motion and points in demonstrations, then builds a motion plan for a low-level visual-servoing controller.

    Source: §I · p. 1

  • Results

    The authors report that RoboTAP solves novel visuomotor tasks from a few minutes of robot interaction without task-specific training or neural-network fine-tuning.

    Source: §V · p. 7

  • Takeaways & Limitations

    The authors see RoboTAP as most useful when visuomotor skills must be taught quickly and the desired behavior can be demonstrated a few times.

    Source: §V · p. 7

    The purely visual controller precludes complex motion planning or force control, and the motion plan executes without replanning if behavior fails or the environment changes unexpectedly.

    Source: §V · p. 7

Abstract

from arXiv · show

For robots to be useful outside labs and specialized factories we need a way to teach them new useful behaviors quickly. Current approaches lack either the generality to onboard new tasks without task-specific engineering, or else lack the data-efficiency to do so in an amount of time that enables practical use. In this work we explore dense tracking as a representational vehicle to allow faster and more general learning from demonstration. Our approach utilizes Track-Any-Point (TAP) models to isolate the relevant motion in a demonstration, and parameterize a low-level controller to reproduce this motion across changes in the scene configuration. We show this results in robust robot policies that can solve complex object-arrangement tasks such as shape-matching, stacking, and even full path-following tasks such as applying glue and sticking objects together, all from demonstrations that can be collected in minutes.

I. INTRODUCTION

RoboTAP investigates dense tracking as a perception primitive for manipulation without task-specific engineering. It frames low-level manipulation as identifying relevant points, locating them, and moving them, then applies this approach to few-shot object-arrangement tasks.

I. INTRODUCTION

  • Dense tracking is explored as a way to solve a wide range of manipulation problems without task-specific engineering.The paper positions this as a representation for manipulation, related to—but distinct from—large-scale sequence-modeling approaches.Source: §I · p. 1
  • The proposed factorization uses dense tracking to identify what is relevant, where it is, and how to move it, with point tracks interfacing low-dimensional functions.Together, these functions are intended to express a large space of behaviors.Source: §I · p. 1
  • RoboTAP is demonstrated on multi-object rearrangement and a glue-application task requiring over 1,000 precise real-valued actions.The authors describe the glue task as messy and irreversible, making it infeasible for approaches that require large-scale data gathering.Source: §I · p. 1

1) End-to-End Approaches:

  • End-to-end image-to-action policies can represent complex behaviors but often require large amounts of data, including target objects or classes.This is the data-efficiency challenge motivating the paper’s alternative representation.Source: 1) End-to-End Approaches · p. 2

2) Pose-based Approaches:

The paper contrasts pose- and keypoint-based policies with dense point tracking, then presents RoboTAP as a demonstration-conditioned system that extracts stage-wise goals and executes them with a low-level controller. Its point-tracking setup is intended to work across novel tasks without depth information or a calibrated camera.

2) Pose-based Approaches:

  • Pose-based policies use object poses regressed from images, but pose is ill-defined for deformable or symmetric objects and difficult and time-consuming to supervise accurately and generally.Source: 2) Pose-based Approaches · p. 2

3) Keypoint-based Approaches:

  • Learned keypoint systems can require retraining for new point sets or object classes, while RoboTAP aims to use point tracking for motion tracking, cross-scene correspondence, and segmentation.The described system uses no depth information and a single non-calibrated gripper-mounted camera.Source: 3) Keypoint-based Approaches · p. 2

B. Point Tracking in Computer Vision

  • High-quality point tracks with few outliers make Euclidean distance between point configurations a usable similarity measure for scenes or motions.The paper notes that TAPIR can be modified to run online frame by frame for robotics applications.Source: B. Point Tracking in Computer Vision · pp. 2–3

III. APPROACH

  • RoboTAP conditions a policy on demonstration data, extracts point queries and target locations, packs their trajectory slices into a motion plan, and executes stages sequentially.The stages advance using a simple final-error criterion.Source: §III · p. 3

A. Temporal and Spatial Decomposition

RoboTAP builds a motion plan by segmenting demonstrations and selecting active points that characterize each segment. Dense tracks support point selection and motion-based object clustering, while the method assumes tasks can be segmented using grasp, release, or contact events.

A. Temporal and Spatial Decomposition

  • The method assumes demonstrations contain shared motion phases and explicitly identifies segments and consistently moving active points rather than learning the point set end to end.Temporal segments are obtained by thresholding gripper actions and forces, under the assumption that tasks involve grasping, releasing, or making contact.Source: A. Temporal and Spatial Decomposition · p. 3
  • Active points tend to end at the same camera-relative location across demonstrations; points that do not move during a segment are removed.The paper supplements these criteria with object-part voting to handle occlusion, detector failures, and imprecise demonstrations.Source: A. Temporal and Spatial Decomposition · p. 3
  • The method clusters tracked points by fitting 3D point locations and per-frame rigid transforms to their reprojections, assuming approximately rigid objects.It uses TAPIR motion estimates for clustering without semantic labels, and models camera motion as motion of all objects.Source: A. Temporal and Spatial Decomposition · pp. 3–4
  • The fitting objective is squared reprojection error, and the authors report that robust losses and RANSAC-style proposals did not perform as well as simple L2.They do not model outliers, relying instead on TAPIR’s occlusion estimation so predicted points can be modeled.Source: A. Temporal and Spatial Decomposition · p. 4
  • Neural networks parameterize point locations and frame-wise rigid transforms to encode the assumption that nearby image points and nearby frames have similar 3D configurations.During optimization, clusters are split to avoid local minima; selection then uses votes, movement thresholds, visibility, and proximity checks to filter points.Source: A. Temporal and Spatial Decomposition · p. 4

B. Robot controller

The controller uses tracked image points and their goal locations to compute gripper motion through a visual-servoing approximation. Its design accounts for noise and occlusions through confidence filtering, orthogonalization, and a variance-matching update.

B. Robot controller

  • The controller moves the gripper in four degrees of freedom to align current image points with goal locations while remaining robust to noise and occlusions.It estimates point motion from gripper movement using an image Jacobian for the wrist camera.Source: B. Robot controller · p. 4
  • Assuming a 90° vertical field of view and unit depth, the controller uses an approximate image Jacobian and tunes rotation and translation gains for stability.Source: B. Robot controller · p. 4
  • The controller minimizes point-to-goal L2 error under a linear approximation, orthogonalizing average translation before estimating rotation and scaling.It filters points not confidently detected in both the current frame and demonstration, uses variance matching for the z-axis update, and averages demo targets near the end of a servoing stage.Source: B. Robot controller · p. 4

C. Online TAPIR

RoboTAP is evaluated on complex, multi-stage manipulation, with reported robustness to pose changes, distractors, precise placement and trajectories, and non-rigid objects. The section also describes an online TAPIR reformulation and a robotics-focused tracking dataset.

C. Online TAPIR

  • TAPIR’s query features and per-frame initialization can be computed separately, while causal refinement uses stored activation history across timesteps.The method replaces temporal convolutions with causal ones so each time-t activation depends only on times ≤t.Source: C. Online TAPIR · p. 5
  • Replacing temporal convolutions with causal convolutions enables TAPIR refinement to run online, with minimal performance impact from the causal model.The online reformulation is evaluated on TAP-Vid benchmarks; Tables III and IV report comparisons.Source: C. Online TAPIR · p. 5, §IV · p. 5, Table III · p. 6

IV. EVALUATION

  • RoboTAP tackles complex multi-stage tasks and handles changed start and end poses, precise placement and trajectories, workspace distractors, and non-rigid objects.The evaluated tasks include placement, insertion, and gluing.Source: §IV · p. 5

A. Robotics TAP-Vid Dataset addition

  • The new robotics TAP-Vid addition contains 265 real-world manipulation videos, human-labeled point tracks, and static-or-moving labels for cameras and points.Videos include static basket and moving wrist camera views; the dataset provides five points per object and five background points per video.Source: A. Robotics TAP-Vid Dataset addition · p. 5

B. Online TAPIR

Online TAPIR retains accuracy close to offline TAPIR while significantly outperforming TAP-Net and PIPs. The robot experiments assess demonstrations, task performance, and recurring failure conditions.

B. Online TAPIR

  • On RoboTAP, online TAPIR achieves average Jaccard (AJ) 59.1, nearly matching offline TAPIR’s 59.6, and significantly outperforming TAP-Net and PIPs.The authors also report accuracy similar to state-of-the-art offline TAPIR on the TAP-Vid benchmark.Source: B. Online TAPIR · p. 6

C. Robot Setup

  • Demonstrations use kinesthetic teaching and require no action supervision beyond gripper opening and closing; collection takes about 30 seconds for pick-and-place and up to 2 minutes for four-object insertion.The gluing task takes about 1 minute to demonstrate.Source: C. Robot Setup · p. 6
  • Across nine tasks, performance was generally robust when the algorithm’s basic assumptions were met, but a simple success metric would not capture its behavior.The evaluation includes clean scenes and examines degradation with increased clutter and occlusions.Source: C. Robot Setup · p. 6
  • Fig. 4 illustrates successful real-robot tasks in scenes with clutter, distracting objects, and partial occlusions.The robot uses a Franka Panda and wrist-mounted camera without camera calibration.Source: Figure 4 · p. 6, C. Robot Setup · p. 6
  • Failures arose when active points were occluded, tracked features became too small after scale changes, distractors resembled essential objects, or gripper/object collisions occurred without clutter perception.Reliable performance was not observed for the LEGO stack or four-object stack.Source: C. Robot Setup · pp. 6–7

D. Real world precision

RoboTAP’s precision was evaluated across demonstrated and novel target locations, with placement error increasing for targets outside the demonstrated configurations. The conclusion frames the system as useful for quick teaching while noting limits in force-based control and replanning.

D. Real world precision

  • Placement error was comparable to demonstration error, but reached several millimeters along the plate and was not suitable for sub-mm insertion tasks.The authors note that they had not extensively tuned the controller for that purpose.Source: D. Real world precision · p. 7
  • <4 mm placement error was observed for a target between demonstrated positions, while moving or rotating the target produced error below 1 cm.The evaluation used six demonstrations and 30 trials for each of three target locations.Source: Figure 6 · p. 7, D. Real world precision · p. 7

V. CONCLUSION

  • RoboTAP solves novel visuomotor tasks from a few minutes of robot interaction without task-specific training or neural-network fine-tuning.Adding tasks, including hyperparameter tuning, took minutes according to the authors.Source: §V · p. 7
  • The system is most useful when quick teaching is needed and the desired behavior can be demonstrated a few times.Source: §V · p. 7
  • A purely visual low-level controller precludes complex motion planning or force control, and executing a one-time plan without replanning can fail after behavior failures or unexpected environment changes.Source: §V · p. 7

APPENDIX I SIMULATED ABLATIONS

The appendix evaluates visual servoing choices in simulation and describes clustering implementation for representing multiple objects. The controller ablations identify instability and performance losses, while recursive cluster splitting addresses optimization collapse into a single cluster.

APPENDIX I SIMULATED ABLATIONS

  • The simulation contains 480 trajectories spanning 20 objects and 24 motions per object, with scale changes up to 5x and backgrounds sampled from real demonstration data.The controller is evaluated on whether it converges to each trajectory’s final state.Source: Appendix I · pp. 8–9
  • Using six degrees of freedom for visual servoing can work, but is significantly less stable; omitting Jacobian evaluation at both locations causes a significant drop, and omitting orthogonalization causes a smaller drop.Both Jacobian-related drops are especially notable with large scale changes.Source: Appendix I · p. 9

APPENDIX II CLUSTERING IMPLEMENTATION DETAILS

  • The clustering objective minimizes reprojection error between predicted image points and projections of canonical 3D point locations transformed by per-frame object poses.The model parameterizes point locations and transforms with neural networks and adds priors because the problem is underconstrained.Source: APPENDIX II CLUSTERING IMPLEMENTATION… · p. 9
  • Object transforms are generated from a temporally smoothed latent representation and constrained to in-plane rigid transforms.Source: APPENDIX II CLUSTERING IMPLEMENTATION… · p. 9
  • Point-location hypotheses use visible-trajectory distances to 384 sampled centroids, followed by a Gaussian encoding and neural-network projection.Shared parameters across clusters can yield similar point locations when doing so does not increase loss.Source: APPENDIX II CLUSTERING IMPLEMENTATION… · p. 9
  • Recursive splitting addresses local minima where camera motion dominates and one cluster absorbs the points; the algorithm tests possible splits and can delete clusters afterward to mitigate over-segmentation.Source: APPENDIX II CLUSTERING IMPLEMENTATION… · pp. 9–10

APPENDIX III IMPLEMENTATION DETAILS OF ONLINE TAPIR

Online TAPIR replaces TAPIR’s temporal depthwise layers with causal convolutions while retaining its parameter count and training procedure. At test time, it maintains convolution context across frames.

APPENDIX III IMPLEMENTATION DETAILS OF ONLINE TAPIR

APPENDIX IV PRECISION EXPERIMENTS - FURTHER INFORMATION

The appendix provides additional detail on the precision experiments. Across 30 evaluations, the target was held fixed and the gear was manually returned to a similar position; Table VII presents the Fig. 6 results as point-coordinate means and spreads.

APPENDIX IV PRECISION EXPERIMENTS - FURTHER INFORMATION

APPENDIX V EXTRACTING MOTION FROM DEMONSTRATIONS

Motion extraction splits demonstrations into aligned temporal segments and selects active points for each segment. The appendix details event-based segmentation, motion resampling, point filtering and clustering, while noting that some processing parameters were selected manually.

Temporal segmentation

  • Motion segmentation uses gripper-open events and force events to define motion-plan boundaries for the controllers.Gripper events occur at selected openness thresholds; force events are identified after smoothing the vertical force signal and thresholding its normalized value at 0.5.Source: Appendix V · p. 11
  • Demonstration segments are resampled along distance traveled, using end-effector and finger velocities to align motion rather than oversampling static phases.The motion metric combines end-effector linear velocity and finger opening or closing.Source: Appendix V · p. 11
  • A motion is treated as a reliable shared primitive when its minimum vertical velocity exceeds both half its standard deviation and 3% of the maximum demonstrated velocity.Source: Appendix V · p. 11

Active point extraction

  • Initial active-point filtering rejects points that are not visible, do not move enough, or end at inconsistent image locations across demonstrations.The filters use visibility, movement relative to the 90th-quantile point motion, and variance in final point locations.Source: Appendix V · p. 11
  • Clustering expands the selected points across scales because points salient in close-up final frames may not be detected when the gripper is farther away or views the object differently.Clusters with sufficient votes are merged to form a bank of relevant points across the motion.Source: Appendix V · p. 11

Parameter selection

  • Processing parameters could be automated, but the authors manually selected and tuned them using intermediate visualizations and variable distributions.The tuned-parameter summary is identified as Table VIII.Source: Appendix V · p. 12, Table VIII · p. 12

APPENDIX VI CONTROLLER IMPLEMENTATION DETAILS

The controller uses point-based visual servoing to follow demonstration trajectories and reach a consistent final state. Its implementation selects targets and detections, orthogonalizes Jacobian components, and modifies the update to address z-translation bias, while relying on heuristics for practical approximation issues.

APPENDIX VI CONTROLLER IMPLEMENTATION DETAILS

  • The Jacobian update chooses an action to reduce the error between target and detected points using a linear approximation, then repeats the update.Source: APPENDIX VI CONTROLLER IMPLEMENTATION… · p. 12
  • The approximation has two practical issues: the best servoing points may change with state, and the Jacobian may poorly approximate desired motion.The paper uses analytical heuristics that work for approximately rigid objects rather than learning to resolve these issues.Source: APPENDIX VI CONTROLLER IMPLEMENTATION… · p. 12
  • Only the 30% most visible or confidently detected points are used for servoing, based on TAPIR visibility scores across the followed demonstration frame and current frame.Source: APPENDIX VI CONTROLLER IMPLEMENTATION… · p. 12
  • The controller follows a nearest demonstration during each segment, then uses visibility-weighted averages across demonstrations to reach a consistent final state.It alternates servoing toward a frame and advancing through the demonstration according to per-point error thresholds.Source: APPENDIX VI CONTROLLER IMPLEMENTATION… · p. 12
  • The Jacobian's last four columns are orthogonalized against its first two because translation dominates the loss and other motion components correlate with translation.The first two columns correspond to x and y translation; tilt columns are used only in simulated ablations.Source: APPENDIX VI CONTROLLER IMPLEMENTATION… · p. 13
  • The controller averages forward and inverse Jacobian updates to counter the z-translation bias caused by detection noise.The forward update alone can produce a positive z action even at the goal; the adjusted expression is zero when the target and detected points have matching squared norms.Source: APPENDIX VI CONTROLLER IMPLEMENTATION… · p. 13

APPENDIX VII ROBOTAP DATASET

The RoboTAP dataset draws videos from three real-world robotics collections and includes annotated point tracks across rigid and deformable objects. Its tracks cover a range of durations, visibility conditions and motion extents, with optical-flow assistance used during annotation.

APPENDIX VII ROBOTAP DATASET

  • Videos come from the Manipulation Task Suite and sampled Sketchy, S3K, and NIST datasets, spanning varied camera setups and rigid and deformable objects.Examples include colored variable-shape rigid objects, a ball, rope and cloth, as well as shoes, toys and gears.Source: APPENDIX VII ROBOTAP DATASET · p. 13
  • Annotators mark each point track with ENTER, MOVE, and EXIT points and may restart tracks after occlusion.Optical-flow track assistance interpolates intermediate frames to improve annotation stability, accuracy and productivity.Source: APPENDIX VII ROBOTAP DATASET · pp. 13–14, Figure 8 · p. 14
  • Among 11,592 point tracks, around 5,000 remain visible across an episode, while the rest are occluded in a relatively uniform distribution.Most contiguous tracks span 20–150 frames, with some exceeding 1,000 frames.Source: APPENDIX VII ROBOTAP DATASET · p. 14
  • Track motion ranges from a quarter of the image for most points to distances spanning the whole video for the most extreme points.The dataset includes three main original video resolutions: 222 x 292, 480 x 640, and 768 x 1024.Source: APPENDIX VII ROBOTAP DATASET · p. 14, Figure 10 · p. 16

Related papers

Loading 2308.15975v2…