PaperlayerView exact revision on arXiv

Source-linked AI summary

BIND: Binding 3D Robot Actions to 2D Image Features

Cameron Smith, Arsh Tangri, Vitor Guizilini, Yue Wang, Zubair Irshad, Sergey Zakharov

arXiv:2609.38443v1cs.ROcs.CV

TL;DR

Visuomotor policies commonly regress actions from globally pooled image features, leaving them to learn the link between robot actions and the image features they project onto. BIND supplies that link by pairing candidate end-effector positions with projected image features and scoring the pairs. The paper reports strong data efficiency and graceful degradation on held-out object positions and camera viewpoints, where coordinate-regression baselines fail.

  • Problem

    Global-regression policies must learn from demonstrations how robot actions correspond to their projecting image features, even though camera geometry determines that relationship.

    Source: §I · p. 1

  • Method

    BIND discretizes candidate end-effector positions, binds each to image features at its projection in every camera view, and scores the resulting action-feature pairs.

    Source: §I · p. 1, §III · p. 3

  • Results

    BIND reaches 100 (10/10 successes) with 5 demonstrations and degrades more gracefully than ACT and Motion Tracks on held-out object positions and camera viewpoints.

    Source: A. Data Efficiency · p. 6, B. Out-of-Distribution Object Position · p. 6, C. Out-of-Distribution Camera Viewpoint · p. 6

  • Takeaways & Limitations

    BIND’s projection-anchored action selection performs strongly across data-efficiency, object-position, and viewpoint evaluations, as well as long-horizon dexterous tasks.

    Source: §V · p. 7

Abstract

from arXiv · show

We introduce BIND, a new action representation for visuomotor robot policies that binds 3D robot actions to their corresponding 2D image features, yielding strong data efficiency gains and robustness to out-of-distribution object positions and camera viewpoints. The action heads of current robot policies are typically formulated as an MLP regression from a single global feature vector produced by a pre-trained vision encoder. This global formulation requires the policy network to discover, from demonstrations alone, the relationship between target robot actions and the image features they project onto. The consequence is that although modern image features are semantically descriptive, spatially robust, and even multiview-consistent, the policies built on them are brittle to subtle changes in camera viewpoint and object placement--and surprisingly data-inefficient. BIND closes this gap by supplying the action-feature relationship through camera geometry rather than learning: it discretizes a volume of candidate end effector positions, attaches each candidate to the pre-trained features at its projection in each camera view, and selects actions by scoring each candidate's position and image-bound feature combination. On a real robot, we study data efficiency and out-of-distribution robustness to unseen object positions and camera viewpoints, as well as general long-horizon task execution and dexterity. We find BIND to be highly data-efficient and robust: it achieves near-perfect success on tasks with as few as 5 demonstrations, and degrades gracefully under steep camera-viewpoint shifts and held-out object positions where coordinate-regression baselines completely fail.

I. Introduction

Global-regression policies must learn the relationship between robot actions and their projecting image features from demonstrations, despite that relationship being determined by camera geometry. BIND supplies the relationship directly by binding candidate 3D positions to projected image features; related approaches differ in how they represent and connect actions to visual features.

I. Introduction

  • BIND discretizes candidate end-effector positions and binds each to image features at its projection in every camera view, using camera geometry to supply the action-feature relationship.The policy scores the resulting action-feature pairs.Source: §I · p. 1, Figure 1 · p. 2

II. Related Work

  • Global-regression policies pool image features into one vector and decode actions from it, leaving the action-to-image-feature association to be learned from demonstrations.The paper notes that strong pretrained image features do not ensure policy robustness to changes in camera pose, lighting, or object placement.Source: §II · pp. 1–2, §I · p. 1
  • Voxel-categorical methods predict end-effector translations over 3D grids, but their voxelized perception stacks provide features mainly at observed surfaces; BIND also binds free-space candidates and predicts dense action chunks.BIND operates on posed RGB images without depth sensors and uses pretrained image backbones.Source: §II · p. 2
  • Rendered-view heatmap methods [18], [19] predict 2D heatmaps that require triangulation to recover 3D actions, whereas BIND projects enumerated 3D candidates into image features.The paper says the rendered-view pipelines also typically predict sparse waypoints and require depth sensors and point-cloud rendering.Source: §II · p. 2
  • Transporter Networks [20] and CLIPort [21] bind actions to image features for top-down planar tasks, while BIND predicts full multistep 6-DoF trajectories.Affordance models [22]–[24] instead predict a static interaction heatmap per image.Source: §II · p. 3

III. Method

BIND turns the action head into a classifier over candidate robot actions, with each candidate paired to image features sampled at its projection in every camera view.

III. Method

  • BIND discretizes candidate end-effector positions, projects each candidate into every camera view to sample corresponding image features, then scores the concatenated position-feature pairs.This replaces global action regression with an image-feature-aware action head.Source: §III · p. 3

A. Problem Setup

The setup describes behavioral cloning from RGB observations and end-effector actions, with the policy predicting a sequence of future actions from the current observation.

A. Problem Setup

  • Each action includes an end-effector position, rotation, and gripper value, while each camera view has intrinsic and robot-to-camera extrinsic matrices.The policy predicts the next T dense-in-time end-effector actions from the current observation.Source: A. Problem Setup · p. 3

B. Step 1: Discretizing the Action Space

Rather than discretizing the full robot action space, BIND discretizes candidate end-effector positions and predicts rotation and gripper separately.

B. Step 1: Discretizing the Action Space

  • Discretizing the full robot action space, such as joint positions, is computationally intractable, so BIND discretizes candidate end-effector positions instead.Rotation and gripper are predicted separately.Source: B. Step 1: Discretizing the Action Space · pp. 3–4
  • Candidate positions are constructed by unprojecting lower-resolution image-grid pixels at multiple discrete heights along each pixel’s ray.The volume may be constructed in other ways, but this is the paper’s practical choice.Source: B. Step 1: Discretizing the Action Space · p. 4
  • For each future timestep, the network classifies candidate positions, training against the candidate nearest the ground-truth position and selecting the highest-logit candidate at inference.Training uses cross-entropy supervision.Source: B. Step 1: Discretizing the Action Space · p. 4

C. Step 2: Binding Actions to Image Features by Projection

Calibrated camera pose and intrinsics determine where each candidate 3D action projects in each view, allowing BIND to sample the corresponding image features.

C. Step 2: Binding Actions to Image Features by Projection

  • For candidate position x and each view, calibrated robot-camera pose and intrinsics give the corresponding pixel location.The view’s spatial feature map is bilinearly sampled at that location.Source: C. Step 2: Binding Actions to Image… · p. 4
  • Table I reports RoboTwin success rates across five tasks, with ManiFlow results quoted from a viewpoint-mismatched evaluation.The caption notes that ManiFlow-3D uses point clouds, while ManiFlow-2D, ACT, and Diffusion Policy use RGB and proprioception.Source: Table I · p. 4

D. Scoring Action–Feature Pairs

BIND scores candidate action positions together with their projected image features, using a height encoding and an MLP to produce per-timestep action probabilities.

D. Scoring Action–Feature Pairs

  • Each candidate’s sampled cross-view features are combined with a positional encoding of height and passed through an MLP for scoring.The height embedding is a 16-dimensional sin/cos encoding of world height, and the MLP is a small few-layer network.Source: D. Scoring Action–Feature Pairs · pp. 4–5
  • The forward pass encodes scene and wrist views into pixel-aligned feature maps, scores candidate position-feature pairs, then selects a 3D trajectory by per-timestep argmax.The MLP produces per-timestep probability volumes for the candidate actions.Source: Figure 4 · p. 5
  • The accompanying tables cover real-robot cup pick-and-place, LIBERO Spatial and Object success, controlled MuJoCo pick-and-place, dexterous tasks, and a backbone ablation.The supplied captions specify these scopes but provide no result values.Source: Table II · p. 5, Table III · p. 5, Table IV · p. 5
  • The image encoder is trained end-to-end with the action head; the authors report that fine-tuning is critical for features to encode robot trajectories implicitly..Source: D. Scoring Action–Feature Pairs · pp. 5–6

E. Rotation and Gripper Prediction

Rotation and gripper state are predicted as categorical distributions for each extracted 3D position keypoint.

E. Rotation and Gripper Prediction

  • Rotation and gripper state are predicted in discretized bins for each keypoint, using fused features indexed at ground-truth candidates during training or argmax candidates at inference.The features are flattened across the chunk’s timesteps and an MLP outputs the rotation and gripper trajectories for the whole chunk.Source: E. Rotation and Gripper Prediction · p. 6

IV. Experiments

The experiments assess data efficiency, robustness to out-of-distribution object positions and camera viewpoints, and long-horizon dexterous tasks. They use real-world and simulation settings, shared backbones, and several policy baselines.

IV. Experiments

  • BIND is evaluated for data efficiency, OOD object-position and camera-viewpoint robustness, and long-horizon dexterous tasks in real-world and simulation settings.The simulation studies provide reproducibility and precise definitions of camera and object distributions; the action head is also tested with pure-vision and vision-language backbones.Source: §IV · p. 6
  • Baselines include ACT, Motion Tracks, and MolmoAct2; controlled simulation adds Diffusion Policy and DP3, while RoboTwin includes reported ManiFlow results.Methods share a DINOv3 ViT-S/16+ backbone, train from a single viewpoint without data augmentation, and are scored by weighted mean task progress from 0 to 100.Source: §IV · p. 6
  • Core data-efficiency and robustness experiments use cup pick-and-place, while dexterous tasks include teapot handling, cup stacking, cup-spill cleanup, and bimanual towel folding.The towel task requires folding a flat towel cleanly in half.Source: §IV · p. 6

A. Data Efficiency

BIND achieves high pick-and-place progress with few demonstrations and degrades more gracefully than ACT and Motion Tracks under held-out object positions and camera viewpoints. It also succeeds on four multi-stage tasks using 30–40 demonstrations.

A. Data Efficiency

  • BIND scores 86–100 with every training set of at least 5 episodes, including 100 (10/10 successes) from 5 demonstrations; ACT never exceeds 36% and Motion Tracks 31%.At 3 episodes, BIND scores 44; cup positions are randomized during training and evaluation.Source: A. Data Efficiency · p. 6

B. Out-of-Distribution Object Position

  • On held-out right-half cup placements, BIND scores 63 versus 0 for both ACT and Motion Tracks, falling from 98 in-distribution without becoming position-invariant.BIND’s failures tend to reach near the object, while the baselines do not move toward the correct side of the table.Source: B. Out-of-Distribution Object Position · p. 6

C. Out-of-Distribution Camera Viewpoint

  • Across three test camera positions after single-position training, BIND scores 79 versus 1 for ACT and 4 for Motion Tracks, including reasonable behavior at near-birds-eye views.The authors describe small camera shifts as having almost no consequence, while extreme shifts can still succeed; BIND is not viewpoint-invariant.Source: C. Out-of-Distribution Camera Viewpoint · p. 6

D. Long-Horizon Dexterous Tasks

  • Across four multi-stage tasks, BIND scores 93 versus 2 for ACT on teapot, 81 versus 8 on cup stacking, 100 versus 0 on spill cleanup, and 97 versus 63 on fold towel.The tasks use 30–40 demonstrations; fully fine-tuned MolmoAct2 scores 0 on the three cup-and-teapot tasks and 47 on fold towel.Source: D. Long-Horizon Dexterous Tasks · pp. 6–7

E. Backbone Ablation

BIND works with a range of pretrained encoders, while controlled simulation tests show stronger generalization to held-out object positions and camera viewpoints than the five compared baselines.

E. Backbone Ablation

  • On in-distribution pick-and-place, BIND scores 82% with DINOv3, 88% with DepthAnything v3, 80% with DynaFlip, 78% with PaliGemma, and 60% with π0.The authors report that general-vision encoders, especially DINO, have qualitatively smoother feature maps than VLM backbones and yield the best results.Source: E. Backbone Ablation · p. 7

F. Controlled Simulation Analysis

  • On mirrored held-out workspace positions, BIND scores 96 while every baseline scores 0; in-distribution, BIND scores 94 versus 82 for Diffusion Policy and 81 for MolmoAct2.Training uses only the left half of the workspace and testing uses the unseen right half.Source: F. Controlled Simulation Analysis · p. 7
  • Across 196 zero-shot held-out viewpoints, BIND scores 49, while all baselines score 12 or lower; it degrades gracefully rather than remaining viewpoint-invariant.Training uses a single camera, and the experiment is conducted in a controlled MuJoCo cube pick-and-place setup against five baselines.Source: F. Controlled Simulation Analysis · p. 7

G. LIBERO Benchmark

BIND performs strongly on LIBERO Spatial and Object and leads the strongest baseline on five RoboTwin bimanual tasks while using RGB alone. The conclusion also reports data-efficient pick-and-place, robustness to distribution shifts, and long-horizon task success.

G. LIBERO Benchmark

  • Trained from scratch on each suite, BIND reaches 92.0% on LIBERO Spatial and 94.0% on Object, exceeding every sub-95 7B baseline on at least one suite.The results are within a few points of the large-scale pretrained frontier.Source: G. LIBERO Benchmark · p. 7

H. RoboTwin Benchmark

  • On five RoboTwin bimanual tasks, BIND reaches a mean success rate of 77.8%, ahead of point-cloud-based ManiFlow-3D (61.9), while operating from RGB alone.Source: H. RoboTwin Benchmark · p. 7

V. Conclusion

  • The conclusion reports strong pick-and-place success from as few as 5 demonstrations, graceful OOD robustness, and long-horizon dexterous task completion with 30–40 demonstrations.BIND binds each candidate end-effector position to the 2D image features at its projection.Source: §V · p. 7

Related papers

Loading 2609.38443v1…