Source-linked AI summary
Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation
Kai Stewart, Yasunori Toshimitsu, Robert K. Katzschmann
TL;DR
Dexterous in-hand manipulation is difficult because contact-rich dynamics challenge simulation-based RL and demonstration-based IL. The paper uses real-time task-Jacobian estimation for model-free, demonstration-free pen writing, achieving rapid, adaptive, sub-millimeter performance across physical and simulated anthropomorphic hands. Its scope remains limited by planar control and continuous-contact assumptions.
Problem
Contact-rich in-hand manipulation remains challenging because RL faces contact-rich sim-to-real gaps, while IL requires demonstrations that are difficult to collect and retarget.
Method
The method estimates and updates a command-space task Jacobian online to control in-hand pen writing without an analytic hand–object model, simulation training, or precollected demonstrations.
Results
The physical hand begins writing after approximately 18 s of initialization, achieves sub-millimeter tracking with 0.64 ± 0.10 mm error, and the formulation also operates on two simulated hands.
Takeaways & Limitations
Online embodied learning can support contact-rich, continuous-trajectory in-hand writing with computationally lightweight control across different hand embodiments.
Takeaways & Limitations
The physical system controls only the planar action space, so uncontrolled 2–3 mm z drift prevents writing on rigid surfaces and multi-stroke writing without arm assistance.
Abstract
from arXiv · showhide
Dexterous in-hand manipulation of a grasped object with an anthropomorphic hand is an unsolved frontier for robot dexterity. The contact-richness and highly dynamic nature of object-hand interactions tend to require extensive modeling or data-collection efforts for learning-based approaches. Modern simulators used for reinforcement learning (RL) cannot fully replicate the required contact complexity, while collecting dexterous demonstrations for imitation learning (IL) remains an open problem. In this research, we present an embodied control approach based on real-time task Jacobian estimation of the combined hand and object system on the physical robot. Using only the CPU on a laptop, the proposed controller begins in-hand pen writing after approximately 18 s of initialization and continues to adapt online, without an analytic hand--object kinematic/contact model, simulation training, or precollected task demonstrations. We demonstrate that the same estimator/controller formulation works on three anthropomorphic robotic hand systems (one physical, two simulated) to show human-like, in-hand articulation of a grasped pen by an embodiment-independent formulation. Sub-millimeter in-plane precision (mean 0.6 mm across runs) is achieved across letters and shapes written in the air and on paper on a physical robot. To our knowledge, this is the first demonstration of an anthropomorphic hand writing arbitrary single-stroke trajectories with a grasped pen through purely in-hand motion, and it showcases an alternative to compute- and data-heavy approaches such as RL and IL for achieving dexterous manipulation through computationally simple and data-efficient algorithms.
I. INTRODUCTION
The paper targets contact-rich in-hand manipulation, where RL and IL face simulation, modeling, and demonstration-data challenges. It introduces lightweight online task-Jacobian control for pen writing without an analytic hand–object model, simulation training, or precollected demonstrations.
- RL depends on domain randomization for sim-to-real contact gaps, while IL is bottlenecked by demonstrations that cannot fully capture occluded fingers and contacts.
- Contact-rich in-hand manipulation remains difficult because changing hand–object contacts make actuation-to-object motion nonlinear, configuration-dependent, and time-dependent.
- Online task-Jacobian estimation provides closed-loop in-hand control without an analytic hand–object model, simulation training, or precollected task demonstrations.
- The 17-DoF ORCA hand writes closed shapes and letter glyphs through finger motion alone, using a laptop CPU and webcam while achieving sub-millimeter in-plane error.
- The estimator/controller is demonstrated across three robot platforms, and ablations characterize tracking metrics, robustness, and essential system elements.
II. RELATED WORK
Prior Jacobian-based and pen-writing methods established online visuomotor estimation but often required substantial data, limited tool applications, or narrower writing evaluations. This work extends the formulation to broad, finger-driven in-hand writing across physical and simulated hands.
- Prior visuomotor Jacobian-field methods required extensive data, including about 2–3 hours of robot motion from a multi-view RGBD setup.
- Earlier Jacobian approaches demonstrated broom or tool manipulation, but tool applications remained limited compared with the present writing task.
- Inverse-Jacobian writing methods avoided matrix inversion but did not clearly support Jacobian-nullspace computation, limiting extension to grip stabilization.
- Related pen-writing studies used finger-driven manipulation, but evaluations included open-loop transitions or restricted trajectories and, in one case, only the letter S.
- The present method writes the whole alphabet after approximately 18 s of initialization and transfers the formulation to two additional simulated hands.
III. SYSTEM ARCHITECTURE
The system closes a perception–estimation–action loop around an anthropomorphic hand, using vision to track the pen and an online command-space Jacobian to generate finger commands. A dedicated excitation phase and nullspace stabilization support learning and grasp maintenance.
- The ORCA hand has 17 actuated degrees of freedom, but the controller holds the wrist, ring, and pinky fixed while articulating the pen with thumb, index, and middle fingers.
- A compliant TPU sleeve enlarges the pen’s effective diameter and adds mechanical compliance at the pen–grip interface.
- A webcam tracks pen and board markers to recover the pen tip in the paper frame, while Kalman filtering rejects outliers and provides a common timing axis.
- The controller transfers forward-Jacobian estimation from serial-arm visual servoing to redundant in-hand manipulation through deliberate excitation and nullspace grip stabilization.
- The event-driven feedback loop combines the estimated Jacobian, reference trajectory, and current state at an approximately 15 Hz perception-limited control rate.
A. Problem formulation
The formulation estimates an unknown local mapping from commanded finger motion to planar pen-tip motion, then inverts it for damped trajectory tracking. Recursive updates, excitation, gating, forgetting, and covariance clipping keep the estimate adaptive and stable.
- A. Problem formulation: The controller commands 10 actuated finger joints to control the pen tip in a 2-D paper plane, while wrist, ring, and pinky joints remain fixed.
- A. Problem formulation: The perceive–estimate–act loop tracks the pen tip, updates and inverts J, computes its nullspace, and sends the resulting motor commands to the hand.
- A. Problem formulation: On-paper writing relies on a raised or folded sheet to absorb approximately 2–3 mm of uncontrolled z drift and maintain pen contact.
- A. Problem formulation: Locally, the unknown task Jacobian relates joint velocities to planar task velocities, and the method estimates it from commanded increments and observed pen motion before damped inversion.
- B. Recursive online task Jacobian estimation: The estimator uses a recursive least-squares or Kalman-filter-style formulation with diagonal covariance storage, initialized from p_init.
- B. Recursive online task Jacobian estimation: Updates occur only when measured joint motion exceeds a threshold, preventing drift from observation noise while the hand is nearly still.
- B. Recursive online task Jacobian estimation: The regressor uses the previous commanded joint increment rather than noisier measured velocities because the controller operationally maps commands to task motion.
- B. Recursive online task Jacobian estimation: A forgetting factor preserves adaptation to changing dynamics, while covariance clipping stabilizes updates without removing adaptivity.
C. Excitation phase
Each run begins with a short, predefined grip-pose excitation that bootstraps the task Jacobian before writing while preserving a secure grasp.
- C. Excitation phase: Approximately 18 s of predefined joint excitation bootstraps the Jacobian estimate before tracking begins.The excitation sweeps the hand through six grip poses along a Catmull–Rom spline.
- C. Excitation phase: Table I lists the default parameters used for the physical ORCA experiments.
- C. Excitation phase: Manual waypoint setup took only minutes, and the same waypoints were reused unchanged across physical ORCA experiments.The setup was not systematically optimized.
D. Writing-path generation
After excitation, a path generator sends planar reference trajectories at constant tangential speed, supporting both cubic-Bézier letters and arbitrary single-stroke shapes.
- D. Writing-path generation: Reference trajectories are reduced to planar curves and traversed at constant tangential speed, making pen-tip speed independent of curvature.
- D. Writing-path generation: Letters A–Z are represented as cubic-Bézier glyphs, alongside parametric shapes and free-form SVG drawings within the writable range.
- D. Writing-path generation: The task command uses a PID-plus-feedforward law based on tracking error, with anti-windup on the integral term and path-tangent feedforward velocity.
E. Control law
The controller updates and damped-inverts the estimated Jacobian, adds nullspace posture stabilization, and filters joint commands before execution.
- E. Control law: Each task command updates the Jacobian, applies a damped right pseudoinverse, and adds posture stabilization through an approximate nullspace.
- E. Control law: Figure 3 shows in-air circle and free-form closed-shape tracing with sub-millimeter in-plane tracking throughout.
- E. Control law: The posture term pulls the hand toward its initial grip while minimizing interference with the task, and damping bounds commands when conditioning is poor.
- E. Control law: Joint increments are clipped to per-joint velocity limits, low-pass filtered, and integrated into absolute joint-position commands.
V. RESULTS
The fully configured system achieves sub-millimeter in-plane tracking for diverse in-air and on-paper writing tasks, including closed shapes, SVG paths, letters, and “hello.”
- V. RESULTS: The hand traces closed shapes and SVG paths, writes on paper, and completes a continuous pass through all 26 Latin letters by finger motion alone.
- V. RESULTS: ∼0.62±0.12 mm average in-plane error was achieved across twenty-two post-excitation in-air runs, with a ∼1.4 mm 95th percentile.The range was ∼0.39–0.86 mm, and the best run reached ∼0.39 mm.
- V. RESULTS: ∼0.67±0.08 mm average in-plane error was achieved across sixteen on-paper runs, with a ∼1.4 mm 95th percentile.The runs covered shapes, letters, and “hello” written letter by letter while the fingers produced every stroke.
- V. RESULTS: ∼0.64 ± 0.10 mm is the pooled in-plane error across n=38 fully configured in-air and on-paper runs.This is the aggregate result across the reported physical experiments.
- V. RESULTS: Table II reports in-plane x,y pen-tip tracking error for the in-air and on-paper conditions.The out-of-plane z axis is excluded from the tracking metric.
B. Ablations
The ablations show that continuous Jacobian adaptation and nullspace grip regularization are central to reliable in-hand writing, while higher speed trades accuracy for throughput.
- Online vs. frozen Jacobian: 0.62 ± 0.06 mm with 4/4 runs completing after a 30 s learning period, whereas freezing immediately or after ∼12 s caused divergence or ∼1.0–1.1 mm error.Continuous updating remained best suited to disturbances unseen by a frozen map.
- Grip regularization: Four of ten runs failed outright without nullspace grip regularization, and only four completed near baseline at 0.55–0.87 mm.Disabling the posture term destabilized the grasp and caused fingers to lose the pen.
- Writing speed: At increased speeds, tracking degraded nonlinearly: reported means ranged from ∼0.8 mm to 2.3 mm at 2× and ∼2.1 mm at 3×, while one 4× run remained at ∼1.1 mm.Progressively ramping speed during a run produced large transients or divergence.
C. Sustained operation without recalibration
Continuous Jacobian re-estimation sustains sub-millimeter tracking during long writing runs without recalibration, while revealing limits from uncontrolled motion, contact continuity, speed, and visual measurement.
- C. Sustained operation without recalibration: 0.59–0.69 mm per-window mean error across six windows remained stationary during a 31-minute alphabet-writing run, with no upward trend.The run covered 26,600 post-excitation steps and completed the full alphabet without losing the grip.
- C. Sustained operation without recalibration: Continuous learning keeps the estimate accurate over long runs and adapts to slow grip changes, consistent with the reported slight downward drift in the rolling error envelope.The ablations identify online updating as improving long-session accuracy and robustness.
- C. Sustained operation without recalibration: 0.64 ± 0.10 mm in-plane error demonstrates sub-millimeter tracking for contact-rich continuous writing on a high-DoF anthropomorphic hand.The controller used an online forward task Jacobian estimated live without a hand–object model, simulation training, or demonstrations.
- C. Sustained operation without recalibration: The method controls only the in-plane action space physically; uncontrolled 2–3 mm z drift requires elevated paper and prevents rigid-surface or fully hand-driven multi-stroke writing.Arm assistance was hardcoded to reposition between characters, while fingers generated each stroke.
- C. Sustained operation without recalibration: The formulation applies only to continuous tasks with unchanged contact state and cannot handle regrasping or finger-gaiting motions.Extending rapid online learning to contact-discontinuous manipulation remains open.
- C. Sustained operation without recalibration: Accuracy is bounded by the vision pipeline because written output was not independently checked against physical ground truth such as deposited ink.Absolute values therefore inherit camera-calibration and marker-tracking accuracy.
VII. CONCLUSION
The paper demonstrates rapid, adaptive in-hand writing on ORCA and portability across two simulated hand embodiments. It presents lightweight embodied learning as an alternative to modeling- and data-intensive approaches.
- VII. CONCLUSION: After approximately 18 s of initialization, ORCA writes contact-rich trajectories by finger motion alone while continuously adapting its Jacobian estimate.The approach operates without extensive modeling or data-collection efforts.
- VII. CONCLUSION: The same formulation operates across substantially different hand kinematics, demonstrated on two additional simulated hand embodiments.The conclusion frames this as evidence of broad adaptability.