PaperlayerView exact revision on arXiv

Source-linked AI summary

EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras

Kush Hari, Justin Kerr, Nidhya Shivakumar, Samarth Mahapatra, Carmelo Sferrazza, Jiahui Lei, Jitendra Malik, C. Karen Liu, Ken Goldberg, Angjoo Kanazawa

arXiv:2610.03710v1cs.ROcs.AI

TL;DR

Fine-grained bimanual manipulation commonly relies on wrist cameras, but their views can be occluded and their placement constrains gripper design. EyeRobot 2.0 uses active gaze and fixation-relative actions with a single stereo camera, outperforming passive stereo and maintaining strong performance when wrist views are occluded.

  • Problem

    Wrist-mounted cameras can be occluded by grasped objects and constrain manipulator design, while stereo-only behavior cloning may lack acuity for precise tasks.

    Source: §1 · p. 2, §5 · p. 7

  • Method

    EyeRobot 2.0 learns hierarchical active fixation and expresses gripper proprioception and action chunks in a fixation-relative SE(3) frame using stereo observations.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    EyeRobot 2.0 outperforms static stereo policies by 40% in real tasks and 20% in simulation, and delivers 2× improvement over ego+wrist policies on tasks with wrist-camera occlusion.

    Source: §5 · pp. 7–8

  • Takeaways & Limitations

    Active fixation with stereo can outperform passive stereo across precision tasks and match or exceed wrist-camera performance within the evaluated tasks.

    Source: §7 · p. 9

    EyeRobot 2.0 trains task-specific gaze and gripper policies and does not control head or neck movement, assuming a suitable head position is available.

    Source: §6 · p. 9

Abstract

from arXiv · show

Inspired by human vision, we introduce a framework using active gaze to enable fine-grained bimanual manipulation with only a single stereo camera. EyeRobot 2.0 physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze on it. The resulting images are processed foveally by allocating more visual tokens to the image centers, focusing computation on task-relevant features. Such Active Visual Fixation (AVF) requires carefully coordinated gaze during task execution, which we accomplish hierarchically by first training a low-level gaze servoing policy conditioned on a goal object, then training a target selector which emits fixation goals based on task progress. Both modules are trained with RL on real-world data: the first is trained with a dense geometric reward and the second co-trains with the BC gripper policy which allows it to discover fixation sequences that can resemble a human's fixation sequence while performing the task. EyeRobot 2.0 further takes advantage of fixation by canonicalizing gripper information into a fixation-relative SE(3) frame, which compacts the size of the action distribution to learn. We collect teleoperation data for 7 real-world and 6 simulated tasks, and conduct over 1000 physical and 1800 simulated robot trials comparing EyeRobot 2.0 against passive stereo and ego + wrist camera policies trained on the same data. Removing wrist cameras is costly for standard policies: with only passive stereo, real-world success drops from 52% to 27%. EyeRobot 2.0 closes this gap with only stereo, outperforming passive stereo by 40% in real and 20% in sim. It matches ego + wrist policies when their wrist views are clear (69% vs. 64%), and more than doubles their success when grasped objects occlude the wrist cameras (48% vs. 22%)

1 Introduction

EyeRobot 2.0 uses active visual fixation to support precise, multi-stage bimanual manipulation with a single fixed stereo camera. It combines hierarchical gaze control with fixation-centered gripper actions and outperforms static stereo baselines across real and simulated tasks.

1 Introduction

  • Active visual fixation directs foveal vision to task-relevant regions and supplies 3D anchor points that can simplify spatial reasoning and divide tasks into gaze-linked sub-goals.Wrist-mounted cameras can provide close views, but may be occluded by grasped objects and constrain gripper design.Source: §1 · p. 2
  • EyeRobot 2.0 uses only one fixed stereo camera for precise tasks such as capping a marker and for multi-stage tasks such as opening an oven and extracting a pan.Both the gaze policy and target selector are trained with reinforcement learning.Source: §1 · p. 2
  • The framework swivels two eye viewpoints toward a 3D fixation point, processes fixation-centered foveated stereo inputs, and canonicalizes gripper proprioception and action chunks into the rotated fixation frame.A low-level gaze policy tracks objects, while a high-level selector chooses what to look at and when.Source: §1 · p. 2
  • EyeRobot 2.0 outperforms static stereo policies by 40% in real tasks and 20% in simulation, while matching ego+wrist policies with clear wrist views and maintaining stronger performance under wrist occlusion.The target selector directs gaze to stage-relevant objects, such as the marker during capping or the bag during repositioning.Source: §1 · p. 2, Figure 2 · p. 3

2 Related Work

Prior work studies learning manipulation from demonstrations and using active vision for perception and manipulation. EyeRobot 2.0 builds on these lines with gaze learned through a BC-accuracy reward rather than human gaze supervision.

2.1 Learning manipulation from human demonstrations

  • Behavior cloning learns complex behaviors from teleoperated demonstrations, while recent work also explores scaling datasets and learning from ego-centric human data.A domain gap remains between first-person human video and robot policies that rely on wrist cameras.Source: §2.1 · p. 3

2.2 Active vision for manipulation

  • Prior active-vision systems address search, detection, tracking, reconstruction, and manipulation, including camera movement driven by teleoperation.EyeRobot 2.0 focuses instead on coordinated fine-grained eye movements with a well-positioned viewpoint.Source: §2.2 · p. 3
  • Some prior methods use human eye tracking to condition gripper policies or crop fingertip images, while simulation-trained gaze methods can struggle with sim-to-real transfer.EyeRobot [7] avoids gaze demonstrations by co-training eye and arm policies with action feedback.Source: §2.2 · p. 3
  • EyeRobot 2.0 extends the co-training approach to finer manipulation with active stereo fixation, fixation-centric actions, and a hierarchical target selector.The target selector is trained with reinforcement learning using a behavior-cloning accuracy reward rather than human gaze supervision.Source: §2.2 · p. 3, §2.1 · p. 3

3 Manipulation with Active Visual Fixation

Active Visual Fixation combines stereo-based gaze control, learned task-dependent target sequencing, and gripper actions expressed relative to the current fixation. The method uses fixation-rendered views for training and couples gaze learning to behavior-cloning prediction accuracy.

3 Manipulation with Active Visual Fixation

  • AVF separates task-specific gripper and fixation policies: two viewpoints converge on a shared 3D fixation point, and fixation-centered observations condition SE(3) action chunks in a dynamically shifting fixation frame.The policies operate from approximately head-mounted stereo observations and reproduce teleoperated behavior with bounded pose variations.Source: §3 · p. 4

3.1 Learning Gaze Patterns with RL

  • Stereo demonstrations are rendered into fixation views by warping calibrated camera images as if the eyes converged on a common 3D point, without requiring 3D reconstruction.This lets the same stereo camera support data collection, training, and deployment.Source: §3.1 · p. 4
  • The goal-conditioned gaze policy is trained with PPO using a geometric reward based on the distance between its fixation point and an object's stereo-depth centroid.It uses prompted objects detected from task language and can fixate on task-relevant objects at 30 Hz.Source: §3.1 · pp. 4–5
  • A lightweight target selector chooses per-task prompts, and its RL reward comes from the gripper policy's action-prediction accuracy on fixation-generated views.The selector is co-trained with the gripper BC policy and emits a new target every 15 frames to let gaze settle.Source: §3.1 · p. 5, Figure 4 · p. 5
  • The target selector's reward uses negative arclength Fréchet distance for the active arm, avoiding sensitivity to spurious bimanual correlations and providing a velocity-invariant measure.The active arm is selected by comparing end-effector motion in the ground-truth action chunk.Source: §3.1 · p. 5

3.2 Fixation-Centric Action Chunking

  • Representing SE(3) proprioception and action chunks in the cyclopean fixation frame standardizes fixation direction and compacts the training distribution.The BC policy is jointly trained on fixation-induced rollouts, with batches sampled from a buffer of the past 100 rollouts.Source: §3.2 · pp. 5–6

4 Experiments

The experiments use teleoperation demonstrations from seven physical and six simulated tasks. The physical tasks emphasize bimanual, precise, and long-horizon manipulation, while simulation evaluates basic pick-and-place behavior.

Simulation Environment

  • The simulation environment is intended as a digital twin for controlled method comparisons using the real-world training pipeline, not for sim-to-real transfer or learning eye gaze.It reproduces the manipulators and camera characteristics, with non-pinhole images rendered by warping pinhole images.Source: §4 · p. 6

Tasks and Data

  • The study collected demonstrations for seven physical and six simulated tasks, focusing on five multi-stage real tasks involving bimanual coordination, precision tolerances, or long-horizon behavior.Examples include marker capping, zipping, straw insertion, object handover, and opening an oven to remove a pan.Source: §4 · p. 6
  • Simulated tasks evaluate basic pick-and-place behavior because contact-rich physical interactions are challenging to simulate in real time.The dataset contains between 10 and 53 minutes of data per task.Source: §4 · p. 6

5 Results

The policies were trained from the same demonstrations, and EyeRobot 2.0 outperformed static stereo across real and simulated tasks. Results also compare it with ego+wrist policies and examine distractors, gaze behavior, and component ablations.

5 Results

  • EyeRobot 2.0 outperforms static stereo across all 13 tasks, averaging +20% in sim and +40% in real.Figure 6 reports end-to-end success; each real task uses 25 trials per policy and each simulated task uses 100.Source: Figure 6 · p. 7, §5 · p. 7
  • When grasped objects occlude wrist cameras, ego+wrist performance falls near the static stereo baseline while EyeRobot 2.0 maintains performance, achieving an overall 2× improvement.Where wrist views remain clear on precise physical tasks, the two approaches perform similarly.Source: Figure 7 · p. 8, §5 · p. 8
  • With distractors, active gaze yields significantly higher first-grasp success; wrist policies often reach for the most distinct object and then lose sight of the target.Table 1 reports end-to-end and first-stage success for the same object positions.Source: §5 · p. 8, Table 1 · p. 8
  • During the marker task, gaze shifts from the cap to the marker before pickup and remains on the marker through capping, sometimes using a static object as an anchor.The target selector learns intuitive gaze switching, including looking at an object until just after it has been grasped.Source: §5 · p. 8

5.1 Ablations

  • Removing the multi-scale foveated image pyramid harms performance by 20%, especially on precise physical tasks, but has no effect in simulation.The authors suggest simulation’s low execution variance may reduce the need for closed-loop visual servoing compared with real-world trials.Source: §5.1 · pp. 8–9, Table 2 · p. 8
  • Using world-frame rather than fixation-relative actions performs on par with the gaze-free ego baseline, a −21% result in simulation.The authors note that a global output space with gaze may still permit overfitting to trajectory positions in the low-data regime.Source: §5.1 · p. 9
  • A single-image ablation without fixation depth performs on average 18% worse than EyeRobot 2.0 across real-world tasks.The authors interpret this as evidence that the model benefits from binocular information.Source: Table 2 · p. 8, §5.1 · p. 9

6 Limitations and Future Work

The framework currently trains task-specific gaze and gripper policies and leaves head or neck movement uncontrolled. The authors also identify limits in multi-task adaptation and sub-object-level gaze.

6 Limitations and Future Work

  • EyeRobot 2.0 trains task-specific gaze and gripper policies; extending it to multi-task performance remains future work.Generic target-selector prompts may require gaze training on a sufficiently large corpus and potentially different prompt-generating architectures.Source: §6 · p. 9
  • The framework does not control head or neck movement and assumes a good head position is already available.Source: §6 · p. 9
  • Object-level pretraining is sufficient for many manipulation tasks but cannot yet train sub-object-level gaze.The authors identify sub-object-level gaze as a possible source of further manipulation benefits.Source: §6 · p. 9

7 Conclusion

EyeRobot 2.0 uses active visual fixation for fine-grained bimanual manipulation with a fixed stereo camera. The experiments suggest it outperforms non-active stereo and matches or exceeds wrist-camera performance, while component ablations reduce performance.

7 Conclusion

  • The experiments suggest active fixation leverages stereo data more effectively than non-active stereo and can match or exceed wrist-camera performance in fine-grained real-world manipulation.Ablations show performance drops when multi-resolution inputs, fixation-centric actions, or stereo information are removed.Source: §7 · p. 9

A Additional Results

The appendix examines stage-level failures, sim-to-real policy rankings, learned gaze patterns, and 3D fixation accuracy.

Failure modes analysis

  • AVF always grasped the small teabag, while the ego-only baseline did so in 32% of tea-task trials.Ego-only also struggled at precision stages; ego+wrist recovered grasping precision but failed when grasped objects occluded its cameras.Source: A Additional Results · p. 10

Sim-real correlation

  • Policy rankings matched between simulation and real-world trials on pot and tape, although absolute simulated success did not predict real-world success.Simulation may favor wrist cameras; pot lid closure was easier under simulated physics.Source: A Additional Results · p. 10

Fixation patterns

  • In simulation, all policies looked at the first grasped object in the first stage; gaze switched to the second object in tape handover, dish rack, and tray, but stayed on the first in pot and spoon tasks.The marker-in-bag gaze probabilities were more ambiguous because the objects were colocated.Source: A Additional Results · p. 10

3D fixation accuracy

  • 3.5 cm was the average fixation error after gaze pre-training, computed over 800 random trajectory frames.Though trained only on still frames, the policy could reliably track moving objects during execution.Source: A Additional Results · p. 10

B Model Architecture and Training Details

The architecture combines foveated stereo inputs with transformer-based gaze and gripper policies, while additional results characterize gaze selection, fixation error, and stage-wise success.

Visual representation

  • Foveated stereo multicrops are centered at the fixation point and encoded separately, providing tokens at multiple resolutions for transformer attention.The frozen DINOv3 ViT-S/16 backbone extracts the image tokens.Source: B Model Architecture and Training… · p. 10

Fixation evaluation

  • Fixation error is reported as 3D Euclidean error averaged over the final 100 training iterations of the best seed per task.The supplied table caption identifies the measure and aggregation, but gives no error values.Source: Table 3 · p. 10

Token positions

  • Image tokens receive global-image x,y positions, while other tokens use the image center; tokens also receive timestep positions, with action chunks assigned future timesteps.Source: B Model Architecture and Training… · p. 10

Gaze selection results

  • In simulation, EyeRobot 2.0 looked at the first object in every first task stage, but sometimes did not look at the second-stage place action in pot and spoon hanging tasks.The tiger task was omitted because it has only one stage.Source: Figure 8 · p. 11

Physical evaluation figures

Policy inputs

Policy architecture

  • The fixation policy uses a gaze actor-critic, while a target-selector MLP chooses target prompts; both policy types decode actions from image, proprioception, and goal-prompt observations.The target selector also estimates value, and its hidden layers use 10% dropout alongside proprioceptive-token dropout during BC sampling.Source: B Model Architecture and Training… · pp. 10–11

B.1 Hyperparameters

The appendix details training choices, baseline calibration, task and data setups, and robot inference procedures.

B.1 Hyperparameters

  • Gaze pre-training uses PPO-style RL with γ = 0.95 and λ = 0.95, and runs for 15M rollout steps across 64 parallel environments.The policy early-terminates updates when discrete-policy KL divergence exceeds 0.02.Source: §B.1 · pp. 12–13
  • The gaze policy uses 119 discretized action bins, formed from polar azimuth/elevation directions and magnitudes plus depth deltas; discretization helps break ties in ambiguous actions.Source: §B.1 · pp. 12–13
  • EE-relative actions performed best across the simulated baseline action-space grid search and were used for real-world baselines; baselines showed train-position overfitting on some tasks, particularly tape and plate.The faded bars evaluate the training positional distributions, while the reported main-paper results use a different-seed test distribution.Source: Figure 12 · p. 13, §B.1 · p. 13
  • During BC-RL post-training, the BC policy is isolated from RL gradients, while the target selector and BC policy are optimized separately and trained using a 100-episode experience buffer.Source: §B.1 · p. 13
  • Baseline comparisons use end-effector SE(3) actions relative to the first frame in each chunk, with world-relative actions producing no performance difference on wrench and toaster tasks.Baseline models were trained at 256 image resolution.Source: §B.1 · pp. 13–14
  • The real task suite includes bimanual, fine-tolerance, visually challenging, and multi-stage behaviors; the simulated suite covers pick-place tasks with single-arm, handover, and bimanual variants.The tape training positions form a grid, whereas other tasks use approximately uniform random object positions.Source: Figure 13 · p. 14, Figure 14 · p. 15, Figure 15 · p. 16

C Robot inference

  • Robot action chunks are predicted as 30 Hz open-loop trajectories and interpolated to 200 Hz, with temporal ensembling and an ensembled-velocity feedforward term for the follower arms.End-to-end throughput is around 40 Hz for baselines and 20 Hz for AVF policies.Source: C Robot inference · pp. 14–15

D Data Details

The appendix describes task-position distributions, simulation configuration, standardized test resets, and matched distractor evaluations.

D Data Details

  • Tape demonstrations use a grid of object positions, while other tasks use approximately uniformly random training positions; baseline policies are sensitive to these positional distributions.The performance gap between AVF and wrist policies on the grid-collected tape task lessens with randomly distributed initial states.Source: D Data Details · p. 16

E Simulation Details

  • Simulation uses MuJoCo’s Newton solver, implicit-fast integration at a 0.5 ms timestep, and shared environment parameters and initialization seeds across models.Collision meshes use convex decomposition, and the environments have hand-coded initial object distributions and physical success measures.Source: E Simulation Details · p. 16

F.1 Test Distribution

  • All policies are evaluated on the same test positions, aligned to the target positions within human placement error.The test-distribution figure composites 25 test overlay images, and those reset positions are standardized across policies.Source: §F.1 · p. 16, Figure 16 · p. 17

F.2 Distractor objects

  • Distractor evaluations repeat the same 25 test-object and distractor-object positions across all models.The wrench-task figure shows a random subsample of distractor positions.Source: §F.2 · p. 16, Figure 17 · p. 17

F.3 Per-stage success criteria for real tasks

Real-task stage success is judged against task-specific completion criteria, with up to three retries before a stage is counted as failed. The criteria distinguish complete, stable manipulation from partial or unstable outcomes.

Scoring protocol

  • Each stage allows 3 retries, and a failure is also recorded if the robot stops moving for more than 10 seconds.Source: §F.3 · p. 17

Tape

  • Tape success requires picking and lifting the yellow tape, handing it to the right gripper, then placing it stably on the grey roll.Source: §F.3 · p. 17

Pot

  • Pot-lid placement must seal the rim with no visible gaps; merely resting the lid on the pot does not count.Source: §F.3 · p. 17

Toaster

  • Toaster success requires opening the door, fully extending the rack, placing the tray without dropping or twisting it, pushing the rack back, and closing the door.Source: §F.3 · p. 17

Marker and bag

  • Marker and bag stages require friction-fit capping, fully inserting the marker into the pouch, and zipping it nearly to the edge before releasing both grippers.Source: §F.3 · pp. 17–18

Boba and teabag

  • Boba and teabag stages require fully seating the straw, stably placing the cup on its coaster, and leaving the teabag fully inside the cup after release.Source: §F.3 · p. 18

Wrench

  • Wrench success requires securely holding the toolbox tab, opening the lid beyond 90°, and lifting the wrench to a steady state without dropping it.Source: §F.3 · p. 18
Loading 2610.03710v1…