Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulation

• Needle threading requires a precise policy because clearance is small, the thread is deformable, and needle posture varies across trials.Source: §I · p. 1
• Human manipulation separates fast peripheral-vision reaching from slow foveated-vision homing, whereas most robot visuomotor methods process the entire visual input.Source: §I · p. 1
• The method uses a 10° gaze-centered foveated image at 142 × 120 resolution and a 128 × 72 resized full image for peripheral vision.Source: §I · p. 1
• Peripheral vision controls fast actions near the target, while foveated vision controls slow precise thread grasping and needle insertion.Source: §I · pp. 1–2
• The explicit gaze mechanism focuses on important scene features, uses fewer neural-network parameters, and improves computational efficiency by retaining high resolution only around the gaze.Source: §I · p. 2

• Needle threading lacks force feedback and requires visual adaptation to thread deformation, making high-precision model-based control difficult.Source: §II · p. 2
• Prior needle-threading systems used imaginary guides, laser depth sensing, rigid-thread assumptions, or specialized mechanisms, while general-purpose learning of pick-and-threading remained unaddressed.Source: §II · p. 2
• Active-vision studies have controlled camera attention or implemented hardware foveation, but this work focuses on visual attention acquired from human gaze for real-world robot manipulation.Source: §II · p. 2
• The proposed method adjusts its policy to different grasped-thread postures, addressing variability in thread configuration during manipulation.Source: Figure 2 · p. 2
• Action-speed data are modeled with two Gaussian distributions, whose intersection defines the threshold separating slow and fast actions.Source: Figure 3 · p. 2

A. High-resolution image processing with human gaze

• A teleoperated UR5 uses ZED Mini stereo images and an HMD-mounted eye tracker to record gaze for constructing visual inputs.Source: A. High-resolution image processing… · p. 2

B. Action separation by speed of action

• Human-demonstration actions are separated by Euclidean positional speed because rotation is not significant in the needle-threading and bolt-picking tasks.Source: B. Action separation by speed of action · p. 3
• A two-component Gaussian mixture model fitted to action speeds sets the fast/slow threshold at the distributions’ intersection.Source: B. Action separation by speed of action · p. 3
• An action-speed classifier selects fast or slow control, while a recovery classifier detects failed threading and switches to annotated retreating recovery actions.Source: B. Action separation by speed of action · p. 3, Figure 5 · p. 3

C. Recovery action

• The approach assumes robots should recover from failures because human demonstrators also fail frequently and retry until threading succeeds.Human demonstrations had a 36.4% failure rate per trial.Source: C. Recovery action · p. 3
• Recovery states are annotated separately from standard threading actions, and failure actions are removed from the training data.Source: C. Recovery action · p. 3
• The pipeline resizes stereo images into peripheral views, predicts gaze, crops foveated views, and selects fast, slow, or recovery actions through classifiers.Algorithm 1 uses peripheral images for fast action and foveated images for slow or recovery action.Source: C. Recovery action · p. 4
• Failure-state recognition triggers a recovery action, which is executed for a predicted number of steps before standard threading resumes.The recovery classifier identifies failure from the current foveated image, while the recovery step predictor estimates the required duration.Source: C. Recovery action · pp. 3–4

D. Model architecture

• Stereo RGB images are resized into 128 × 72 peripheral images for fast-action and gaze prediction, while gaze-centered crops support detailed control.Source: D. Model architecture · p. 4
• The foveated representation predicts gripper commands, standard and recovery actions, failure state, recovery duration, and action speed.Source: D. Model architecture · p. 4
• A mixture density network predicts the two-dimensional gaze-position distribution from concatenated left/right peripheral images.Its outputs are the mean, standard deviation, and correlation of the gaze-position probability distribution.Source: D. Model architecture · p. 4
• Spatial softmax extracts coordinate features for fast-action and gaze prediction, whereas convolutional and max-pooling layers preserve three-dimensional thread-and-needle information.Source: D. Model architecture · pp. 4–5
• The networks use fully connected layers with batch normalization and ReLU activations, with three layers for action and gripper models and two for other models.Each fully connected layer has 200 nodes.Source: D. Model architecture · p. 5

A. Task setup

• Needle threading required picking up a thread and inserting it into a needle eye randomly placed over a 9cm×9cm table area.Trials varied both needle location and thread grasping position.Source: A. Task setup · p. 5
• Bolt picking tested generality on precise manipulation by requiring an M6×14 bolt to be grasped accurately without colliding with the table.Source: A. Task setup · p. 5

B. Assessment of the proposed method

• 81.25% needle-threading and 68.75% bolt-picking success rates were achieved using 203.8 and 39.56 minutes of training data, respectively.Source: B. Assessment of the proposed method · p. 5

C. Assessment of foveated vision

• Replacing foveated images with peripheral images reduced needle-threading success to 12.5%, while raw-image architectures failed at thread and bolt picking.Source: C. Assessment of foveated vision · p. 5
• The proposed representation required 8.10 times fewer MACs per step and approximately 27.7 times less training time than processing raw 1280 × 720 images.The difference between MAC and time savings was attributed to overheads such as memory copy.Source: C. Assessment of foveated vision · p. 5

D. Assessment of the action separation

• Training both fast and slow actions with peripheral images reduced needle-threading success to 0%, and the action-separation threshold was selected at the intersection of two Gaussian distributions.The reported intersection is 0.005685.Source: D. Assessment of the action separation · p. 5

E. Assessment of visuomotor control separation

• Assigning peripheral vision to fast action and foveated vision to slow action was the only tested separation that enabled thread pickup; alternative allocations failed or reduced success.Source: E. Assessment of visuomotor control… · p. 5

F. Assessment of the recovery action

• Recovering from 12 of 13 failures outperformed a single network’s recovery from one of eight failures.Without the recovery step predictor, success dropped to 31.25%.Source: F. Assessment of the recovery action · p. 5

G. Assessment of the resolution

• A half-resolution image reduced success to 50%, demonstrating that high-resolution images were required for needle threading.Source: G. Assessment of the resolution · p. 5
• The robot recognized failure at 16.0 s, recovered at 18.0 s, and succeeded in threading at 22.0 s.Source: Figure 8 · p. 6
• The method also performed manipulation when the thread deformed after colliding with the needle.Source: Figure 9 · p. 6

H. Assessment of stereo vision

• Removing stereo vision caused the success rate to decrease dramatically.Source: H. Assessment of stereo vision · pp. 6–7

V. DISCUSSION

• The method enabled a general-purpose robot manipulator to perform high-precision needle threading and partially generalized to bolt picking.Source: §V · p. 7
• The method achieved appropriate control even when the thread was deformed.Source: §V · p. 7
• Separated visuomotor control improved task performance and computational efficiency compared with processing high-resolution visual input.Source: §V · p. 7

A. Calculation of foveated vision from the camera image

• The 10° central retina corresponds to a 142 × 120 image region within the camera’s 1280 × 720 image.The ZED Mini camera field of view is 90° horizontally and 60° vertically.Source: A. Calculation of foveated vision from… · p. 7

B. Training details

• Training used 1,713 needle-threading episodes and 775 bolt-picking episodes, with separate validation sets for both tasks.The training durations were 203.8 minutes for needle threading and 39.56 minutes for bolt picking.Source: B. Training details · p. 7
• Classification accuracy was reported in Table V, while fast- and slow-action sequences were visualized across thresholds in Fig. 10.Source: Table V · p. 7, Figure 10 · p. 7
• Each network component was selected by lowest validation loss over 30 epochs using a 1e−4 learning rate and RAdam optimization.Source: B. Training details · p. 7
• Training used a Tesla V100 GPU, while robot control used an Intel i7-8700K CPU and a GeForce GTX 1080 Ti.Source: B. Training details · p. 7

C. Evaluation details

• A trial was classified as a failure when it exceeded the maximum of 500 steps.The needle used a 5 cm × 5 cm sponge block, and approximately 1–2 mm of thread end was hardened with glue.Source: C. Evaluation details · p. 7
• The additional experiment with a moistened, non-hardened thread recorded 68.75% successful threading.Source: C. Evaluation details · p. 7

D. Classifier accuracy

• Table V reported the classification accuracy of the recovery and action-speed classifiers.Source: D. Classifier accuracy · p. 7, Figure 11 · p. 8

E. Assessment of the action separation threshold

• Threshold=0.00569 produced the fewest action shifts, indicating the least interleaved segmentation of slow and fast actions.Fig. 10 visualized the threshold-separated actions, and Fig. 11 counted fast-to-slow and slow-to-fast shifts.Source: E. Assessment of the action separation… · p. 8, Figure 11 · p. 8