PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoThumb: A Vision-Driven Supernumerary Robotic Thumb for Autonomous Grasp and Holding Assistance

Jianyu Hou, Jiawei Fu, Weikang Wan, Minghui Jiang, Jiajun Li, Hao Su

arXiv:2610.37582v1cs.RO

TL;DR

Existing supernumerary fingers are bulky and often depend on explicit hand-motion triggers, limiting everyday-object interaction and context-aware grasping. EgoThumb combines a compact forearm-actuated thumb with palm-facing visual imitation learning. It reports high success across users and tasks, with performance limits under large spatial variation and long horizons.

  • Problem

    Existing supernumerary fingers are bulky for small everyday objects and often rely on explicit hand-motion triggers that lack environmental context.

    Source: §I · p. 2

  • Method

    EgoThumb uses forearm-mounted dual-tendon actuation and a palm-facing camera, with a visual imitation-learning policy mapping egocentric images to joint positions.

    Source: §I · p. 2

  • Results

    Across four tasks, success averaged 0.9 on the demonstrator and at least 0.875 for User 1 and User 2; under expanded object areas, rates exceeded 0.73 for Pick Up Cube and Unscrew Cap.

    Source: B. Results and Analysis · pp. 5–6

  • Takeaways & Limitations

    EgoThumb supports hold-and-manipulate assistance and shows generalization across users and environments, while long-horizon tasks with large spatial variation remain challenging.

    Source: §V · p. 7

    Performance degrades on long-horizon Stir Tea and Transfer Ball tasks with large spatial variation in initial and target areas.

    Source: §V · p. 7

Abstract

from arXiv · show

Supernumerary robotic fingers augment human dexterity by enabling a single hand to perform complex "hold-and-manipulate" tasks that typically require bimanual coordination. However, existing devices are often hindered by bulky hardware that limits interaction to large objects and rely on explicit hand motion triggers that lack environmental context. To address these challenges, we present EgoThumb, a fully open-source, vision-driven supernumerary robotic thumb. EgoThumb features a compact, dual tendon-driven finger whose actuation is relocated to the forearm, reducing distal weight while enabling precise active joint control. The system utilizes a visual imitation learning policy that maps egocentric images from a palm-facing camera directly to continuous joint positions. By implicitly decoding human intent from visual cues capturing both the hand and the object of interest, EgoThumb achieves robust autonomous manipulation and strong generalization across diverse users and environments. All hardware designs and software implementations are fully open-sourced at https://ego-thumb.github.io.

I. INTRODUCTION

Supernumerary robotic fingers can let one hand perform hold-and-manipulate tasks, but bulky designs and explicit-motion control limit everyday-object interaction and contextual autonomy. EgoThumb pairs forearm-actuated hardware with palm-facing vision and imitation learning for autonomous grasp assistance.

I. INTRODUCTION

  • Bulky hardware limits existing supernumerary fingers to larger objects, while teleoperation and low-dimensional hand-motion triggers lack contextual autonomy.The motivating example is grasping a bottle as the hand approaches without an explicit trigger motion.Source: §I · p. 2
  • EgoThumb uses dual-tendon joints with forearm-mounted actuators for active flexion and extension control in a lightweight wearable finger.Moving the actuators to the forearm reduces distal weight.Source: §I · p. 2
  • A palm-facing camera captures hand motion and surrounding context, while visual imitation learning maps egocentric images directly to joint positions.The reported system performs across four real-world tasks and generalizes across users and environmental backgrounds without explicitly measuring hand motion.Source: §I · p. 2

II. RELATED WORK

Prior supernumerary robotic fingers trade compactness, control authority, or autonomy. EgoThumb combines forearm-mounted dual-tendon actuation with visual-context-based grasp timing.

A. Design of Supernumerary Robotic Fingers

  • Direct-drive fingers can be bulky and tiring, while pneumatic designs add volume and compact tendon-driven alternatives often use passive extension that reduces positioning accuracy.These limitations constrain interaction with smaller everyday objects.Source: A. Design of Supernumerary Robotic… · p. 2
  • EgoThumb’s forearm-mounted dual-tendon drive actively controls flexion and extension, minimizing distal weight and enabling a compact design for everyday objects.Source: A. Design of Supernumerary Robotic… · p. 2

B. Control of Supernumerary Robotic Fingers

C. Imitation Learning for Robot Hands

Prior vision-motor imitation-learning frameworks target standalone robots, whereas EgoThumb frames wearable-finger control in a shared human-robot workspace. Its system combines a camera-equipped wearable thumb with forearm actuation and synchronized communications.

C. Imitation Learning for Robot Hands

  • EgoThumb adapts visual imitation learning to a shared workspace, mapping egocentric images directly to wearable-finger joint positions rather than targeting a standalone robot.Source: C. Imitation Learning for Robot Hands · p. 2

III. SYSTEM DESIGN

  • The system uses Zenoh and Protocol Buffers for low-latency cross-platform communication and NTP to synchronize its Raspberry Pis and PCs.Source: §III · p. 2

A. EgoThumb Design

  • The system pairs a compact tendon-driven wearable thumb and wide-field palm-facing camera with a forearm-mounted actuation module.Source: A. EgoThumb Design · p. 2

1) Wearable Thumb and Palm-Facing Camera:

The wearable thumb simplifies its joints for grasping and holding, uses dual tendons for active bidirectional positioning, and places a wide-view camera on the palm side to capture the interaction zone.

1) Wearable Thumb and Palm-Facing Camera:

  • The thumb has CMC, MCP, and IP joints, with the CMC simplified to a single degree of freedom because high-DoF motion is unnecessary for grasping and holding.CMC abduction and adduction are termed flexion and extension in the paper.Source: 1) Wearable Thumb and Palm-Facing Camera · pp. 2–3
  • PLA phalanges have a silicone-gel palmar coating for stable grasping, and adjustable straps secure the device across hand sizes.Source: Figure 2 · p. 3, 1) Wearable Thumb and Palm-Facing Camera · p. 3
  • Three forearm-mounted Dynamixel-driven ratchet spools convert motor rotation into tendon movements and can rapidly retension tendons to mitigate slack.Source: Figure 2 · p. 3
  • Dual tendons actively control flexion and extension, with mechanical ranges of 180° at CMC, 84.14° at MCP, and 141.34° at IP.The ranges are stated to cover the grasping and holding postures required by the tasks.Source: 1) Wearable Thumb and Palm-Facing Camera · p. 3
  • A palm-facing 220° fisheye camera captures the hand, robotic thumb, and target object; cropped raw frames are used without undistortion or intermediate hand-pose estimation.The radial-side placement maintains an unobstructed interaction-zone view, and the crop prioritizes that workspace.Source: 1) Wearable Thumb and Palm-Facing Camera · p. 3

2) Actuation Module:

The actuation module converts motor rotation into tendon-driven flexion and extension, while forearm-mounted actuation and supporting hardware enable controlled, wearable operation.

2) Actuation Module:

  • Ratchet spools anchor two tendons, converting motor rotation into flexion and extension; rapid retensioning mitigates tendon slack.The Dynamixel motors provide 1.4 N·m stall torque and 12-bit encoders with 0.088° resolution for proprioceptive feedback.Source: 2) Actuation Module · pp. 3–4
  • PTFE tubes maintain invariant tendon path lengths during wrist articulation, decoupling finger motion from wrist movement.Source: 2) Actuation Module · p. 4
  • The embedded Raspberry Pi runs motor control at 60 Hz and acquires images at 20 Hz, while policy inference runs on an external PC.The Raspberry Pi performs no vision processing.Source: 2) Actuation Module · p. 4

B. Teleoperation System

The paper uses a foot-operated interface to collect demonstrations, then trains a vision-based imitation policy to control EgoThumb from camera and joint-state history.

B. Teleoperation System

  • A foot-operated interface uses four force-sensitive resistors, sampled at 50 Hz by a 16-bit ADC, to collect teleoperation demonstrations.A secondary Raspberry Pi 3B+ handles signal processing.Source: B. Teleoperation System · p. 4
  • The operator uses left-toe FSRs to control MCP and IP flexion or extension velocity and right-toe FSRs to control the CMC joint.A linear synergy sets θIP = 2/3θMCP; camera images, joint positions, and teleoperation target actions are logged for training, while teleoperation hardware is disconnected during autonomous deployment.Source: B. Teleoperation System · p. 4

C. Imitation Policy Learning

  • The policy maps recent palm-facing images and proprioceptive joint positions to a horizon of future target actions.It uses an observation history of T = 2 and prediction horizon H = 8.Source: C. Imitation Policy Learning · p. 4
  • A CLIP-pretrained Vision Transformer encodes images, whose latent features are concatenated with proprioceptive states before iterative denoising generates future actions.Continuous latency matching synchronizes the asynchronous visual stream and hardware states; deployment inference runs on an external PC.Source: C. Imitation Policy Learning · p. 4

IV. EXPERIMENTS

The experiments ask whether EgoThumb assists grasping while freeing the human hand, and whether it generalizes across users and uncontrolled environments.

  • The first research question tests whether EgoThumb can assist grasping while freeing the human hand for other motions.Source: §IV · p. 4
  • The second research question tests generalization to different hands.Source: §IV · p. 4
  • The third research question tests generalization to “in-the-wild” environments.Source: §IV · p. 4

A. Evaluation Setup

The evaluation covers four collaborative manipulation tasks, with training demonstrations and testing across task episodes and users.

A. Evaluation Setup

  • A demonstrator collected 50 training episodes per task, with objects placed randomly within initial and target areas.The imitation policy was trained for 120 epochs.Source: A. Evaluation Setup · pp. 4–5
  • The four evaluation tasks are Pick Up Cube, Unscrew Cap, Stir Tea, and Transfer Ball.The rollouts show the user’s first-person view, the policy’s palm-facing camera view, and timelines of coordinated human and EgoThumb actions.Source: A. Evaluation Setup · p. 4, Figure 3 · p. 5
  • Evaluation used 10 episodes per task with the same initial and target areas as training.Source: A. Evaluation Setup · p. 5
  • Two operators whose hands were absent from training evaluated generalization, using consistent initial object configurations across users.The table caption reports strong performance across all four tasks and effective generalization to both novel participants.Source: A. Evaluation Setup · p. 5, Table I · p. 5

B. Results and Analysis

EgoThumb provides stable grasp assistance across four tasks and generalizes to new users. Its performance varies with spatial and background changes, with the strongest results in seen environments and simpler spatial-variation tasks.

B. Results and Analysis

  • The demonstrator achieved an average success rate of 0.9 across all four tasks, with stable grasping during dynamic hand interactions.The authors attribute this to occlusion-free camera placement and a dual-tendon, silicone-skinned finger.Source: B. Results and Analysis · p. 5
  • User 1 and User 2 each maintained average success rates of at least 0.875 across all four tasks despite novel hand appearances and poses.Adjustable fasteners accommodate different hands, and the rigid camera mount keeps the thumb’s pose constant in the visual frame.Source: B. Results and Analysis · pp. 5–6
  • With expanded object areas, Pick Up Cube and Unscrew Cap exceeded 0.73 average success, while Stir Tea averaged 0.53 and Transfer Ball 0.17.The authors associate Stir Tea’s drop with foreground visual distractors and Transfer Ball’s with its long-horizon sequential planning demands.Source: Table II · p. 5, B. Results and Analysis · p. 6, Figure 4 · p. 6
  • On Stir Tea, success rates remained above 0.8 in seen environments and above 0.67 in unseen environments across three users.Increasing training demonstrations from 25 to 105 generally improved performance in both settings for all users.Source: B. Results and Analysis · p. 6, Table III · p. 6

C. Ablation Study

Ablations show that latency matching and image cropping are both necessary for the tested Unscrew Cap policy. Separately, increasing demonstration count generally improves performance in seen and unseen environments.

C. Ablation Study

  • Scaling demonstrations from 25 to 105 generally improved policy performance in both seen and unseen environments across all users.Figure 5 encodes seen environments with solid lines and unseen environments with dashed lines.Source: Figure 5 · p. 7
  • Disabling either latency matching or image cropping yielded 0 success rate versus 0.7 for the full system.Both ablations used Unscrew Cap with the in-distribution demonstrator and otherwise matched the main experiment setup.Source: C. Ablation Study · pp. 6–7, Table IV · p. 7
  • Latency matching aligns asynchronous observations and compensates for pipeline delay; disabling temporal compensation caused complete policy failure.The ablation replaced interpolation with nearest-neighbor state matching and removed predictive action delay.Source: C. Ablation Study · p. 7
  • Without image cropping, the policy failed completely, consistent with cropping’s role in isolating the hand and target object from distractors.Uncropped frames were processed with a ResNet34 encoder during training and inference.Source: C. Ablation Study · p. 7

V. CONCLUSION AND LIMITATION

EgoThumb pairs a compact tendon-driven thumb with egocentric visual imitation learning for hold-and-manipulate tasks. The conclusion reports generalization across users and environments, while noting limits in task scope, sensing, data, and evaluation.

V. CONCLUSION AND LIMITATION

  • EgoThumb combines a lightweight tendon-driven joint design with egocentric visual imitation learning to augment dexterity in hold-and-manipulate tasks.The system autonomously decodes human intent and is reported to generalize across users and environments.Source: §V · p. 7
  • Performance degrades on long-horizon tasks with large spatial variation, and varied hand approach angles remain untested.The palm-facing camera may reduce wearing comfort; rapid hand motions and high-speed manipulation are also unevaluated.Source: §V · p. 7
  • Evaluation uses binary task success over 10 trials per condition; finer coordination measures and larger-scale statistical analysis remain needed.Source: §V · p. 7

Related papers

Loading 2610.37582v1…