PaperlayerView exact revision on arXiv

Source-linked AI summary

No Eyes, No Problem: What Does Touch Contribute to Blind Baoding Ball Manipulation?

Ayush Deshmukh, Nalin Agarwal, Elle Miller, Sethu Vijayakumar

arXiv:2610.23438v2cs.RO

TL;DR

Sustained blind manipulation without vision remains a challenge for robots, and transfer from simulation to a physical dexterous hand is hindered by mismatches in contact, tactile observations, and dynamics. This work trains a Baoding ball rotation policy with proprioception and binary tactile feedback and transfers it zero-shot to a Shadow Dexterous Hand Lite. The physical hand completes 225 consecutive 180° rotations in a continuous trial without vision or object-state information.

  • Problem

    Sustained real-world blind manipulation remains challenging, with sim-to-real transfer hindered by changing contacts and discrepancies in tactile observations and physical dynamics.

    Source: §I · p. 2, A. Problem Setting · p. 3

  • Method

    The work trains a blind Baoding rotation policy and transfers it zero-shot using a hardware-aligned simulation, tactile calibration, and episode-level randomisations.

    Source: §I · p. 2, §VII · p. 8

  • Results

    225 consecutive 180° rotations were completed in a single continuous physical-hand trial using proprioception and binary tactile feedback without vision or object-state estimation.

    Source: §VII · p. 8

  • Takeaways & Limitations

    The reported run demonstrates sustained blind Baoding rotation on the Shadow Dexterous Hand Lite using proprioception and binary tactile sensing.

    Source: §VII · p. 8

Abstract

from arXiv · show

Humans manipulate objects within their hands through touch alone with remarkable ease, but for robots, achieving sustained real-world manipulation without vision remains an open challenge. Although reinforcement learning (RL) can generate sophisticated blind manipulation policies in simulation, transferring them to a real dexterous hand remains hindered by continuously changing contacts, motion-induced tactile information drift, and the gap between simulated and physical dynamics. We study this problem through Baoding ball rotation on the Shadow Dexterous Hand Lite, using only proprioception and binary tactile feedback as observations. We achieve successful zero-shot transfer through a combination of a sensor-aligned, coupling-aware embodiment in simulation, tactile calibration, and two novel episodic domain randomisations: slew-rate action limits and STuck-AT Taxels. On the physical hand, the policy achieves sustained blind Baoding rotation, completing $\textbf{225 consecutive $\mathbf{180^\circ}$ rotations (112.5 full rotations) at $\sim0.36 \;\text{rot/s}$}$ in a single continuous trial without any vision or object-state information. Furthermore, we evaluate the contribution of tactile sensing relative to proprioception alone and identify the hand regions where tactile information is most important. This is the strongest reported hardware result for Baoding ball manipulation in both rotation count and speed, irrespective of the sensing modality used. By effectively bridging the sim-to-real gap, our framework sets a new benchmark for sustained tactile-driven dexterous manipulation on physical hardware.

I. INTRODUCTION

The paper asks whether touch can substitute for explicit object-state information in blind in-hand manipulation, then tests zero-shot transfer for the more demanding task of coordinating two Baoding balls. It reports a sustained hardware run using proprioception and binary contact, alongside methods for aligning simulation and improving robustness.

I. INTRODUCTION

  • The policy receives only joint encoders and 16 binary contact signals, unlike prior approaches using cameras, motion-capture markers, or privileged simulator state.Baoding manipulation coordinates two simultaneously circulating objects and must maintain their phase relationship without directly observing it.Source: §I · p. 2
  • The paper targets the open challenge of transferring sustained blind Baoding rotation from simulation to physical hardware.Prior hardware approaches typically relied on external object-state information or tactile sensing, while recent proprioception-and-binary-contact results were limited to simulation.Source: §I · p. 2
  • 225 consecutive 180° rotations (112.5 full rotations) were completed at ∼0.36 rot/s in one continuous physical-hand trial using touch and proprioception alone.The policy was trained entirely in simulation and used no vision or object-state information.Source: §I · p. 2
  • The simulation matches the physical hand’s tendon coupling and tactile sensor sites, while per-channel hysteresis-based calibration mitigates tactile sensor noise.The contribution list also introduces SlewDR and STAT as episode-level randomisations to encourage smoother policy outputs and reduce reliance on individual tactile channels.Source: §I · p. 2
  • The study benchmarks rotation across increasing feedback levels and cumulative randomisation ablations.It also evaluates tactile feedback’s contribution to performance and long-horizon recovery.Source: §I · p. 2

II. RELATED WORKS

Related work spans manipulation that uses external object-state sensing and blind manipulation that relies on the hand’s own sensing. Baoding rotation extends the blind setting to coordinating two free objects, while sustained zero-shot hardware transfer remains an open challenge.

II. RELATED WORKS

  • Prior manipulation approaches obtain object state through pose, point clouds, depth cameras, or vision with dense touch, despite occlusion when fingers enclose the object.Proprioception and joint position and torque sensing have also supported blind single-object rotation and cube reorientation.Source: §II · p. 2
  • Existing blind hardware results with sustained manipulation involve a single object, while prior blind Baoding results using binary touch remain in simulation.The paper identifies sustained, rapid hardware rotation as an unshown result in this prior work.Source: §II · pp. 2–3
  • Baoding manipulation requires coordinating two balls through ball–hand and ball–ball contacts, making it more challenging than single-object manipulation.Prior hardware studies largely used external object information; one tactile-array study reported only a qualitative half-rotation.Source: §II · p. 2

A. Problem Setting

  • The task is formulated as a POMDP, with simulated dynamics and observations idealized relative to physical tendon, actuator, contact, calibration, latency, and tactile-activation effects.Because the same command or physical state may yield different hardware states or observations, simulation-optimized behavior is not guaranteed to reproduce on hardware.Source: A. Problem Setting · p. 3

B. Agent and Learning Setup

The policy uses the same blind observation and action layout in simulation and on hardware. The setup describes the Shadow Hand Lite’s sensing and mechanical coupling, then modifies the simulated embodiment to match the physical hand.

B. Agent and Learning Setup

  • The agent receives a four-frame stack of 52 proprioceptive values and 16 binary tactile values, for a 272-dimensional observation, and issues 13 joint-position commands at 60 Hz.No camera, depth, motion capture, or privileged state is available at inference.Source: B. Agent and Learning Setup · p. 3

C. Hardware Platform and Sensing

  • The Shadow Dexterous Hand Lite has 16 kinematic joints but 13 independently commandable actuators because the PIP and DIP joints of three fingers are mechanically coupled.Its tactile system uses 12 FSRs and four BioTac fingertip sensors; the 16 binary channels are thresholded contact signals.Source: C. Hardware Platform and Sensing · p. 3

D. Simulation Embodiment

  • The physical hand’s coupled joints move sequentially, whereas the default Isaac Lab model moves the DIP as a fixed multiple of PIP angle, so the default model is not hardware-aligned.The simulation applies commands sequentially around the 100° coupling threshold and adds collision bodies at physical sensor locations.Source: D. Simulation Embodiment · pp. 3–4

E. Bridging the Sim-to-Real Gap

The sim-to-real strategy combines physical parameter variation with action-rate randomisation to address discrepancies between simulated and hardware behavior. In particular, the slew-rate limit constrains changes in commanded joint position and is randomized by episode during training.

1) Physical Parameter Randomisation:

  • Ball mass is randomized from 45–100 g, and static and dynamic friction are independently varied across balls and hand regions.Friction ranges reflect regional surface differences, with higher values on distal segments for BioTac silicone skin and lower values on the smoother palm.Source: 1) Physical Parameter Randomisation · p. 4

2) Slew-rate Domain Randomisation (SlewDR):

  • The slew-rate limit constrains changes in commanded joint position because the policy commanded motion faster than hardware could execute, potentially due to tendon friction and reflected motor inertia.The same limit is randomized in simulation with a scale factor resampled from U(0.3, 1.0) at each episode reset.Source: 2) Slew-rate Domain Randomisation… · p. 4
  • The deployment rate limit uses the rated joint velocity and 1/60 s control interval, and its scale factor remains fixed across deployment episodes rather than being resampled.The published target is rate-limited using the desired joint position and the command actually sent to the robot.Source: 2) Slew-rate Domain Randomisation… · p. 4

3) Stuck-AT Taxels (STAT):

The section introduces episode-level tactile corruption to discourage dependence on unreliable channels, then calibrates binary FSR contacts with hysteresis to address threshold drift during deployment.

3) Stuck-AT Taxels (STAT):

  • STAT holds a randomly selected subset of k of the twelve FSR channels at fixed values for each training episode, while other channels report simulated contact.The approach is intended to prevent reliance on any single tactile channel.Source: 3) Stuck-AT Taxels (STAT) · p. 4

4) Tactile Calibration:

  • Static FSR thresholds were insufficient because readings shifted with hand pose, wire movement, and motion, so each channel uses separate high and low thresholds.The two thresholds provide hysteresis for binary contact conversion.Source: 4) Tactile Calibration · p. 4
  • The high threshold uses the 99.5th percentile of an empty-hand baseline plus 5, while the low threshold uses its median with empirical offsets.A Schmitt rule switches contact on above the high threshold, off below the low threshold, and otherwise retains the previous state.Source: 4) Tactile Calibration · p. 4

IV. EXPERIMENTAL SETUP

The evaluation transfers policies to a physical hand without hardware training or fine-tuning and compares them under a fixed protocol. Hardware metrics capture rotation count, time to drop, rotational speed, and exchange efficiency.

A. Evaluation Protocol

  • Hardware policies were evaluated zero-shot in ten independent trials per configuration, using the same pair of 1.58-in, 55g balls without a time limit.The hand was mounted approximately 15° downward from horizontal and controlled at 60 Hz; PD gains were fixed across configurations.Source: A. Evaluation Protocol · pp. 4–5
  • Simulation training applies corrected coupling before physics, while deployment transfers the policy unchanged at 60 Hz with tactile calibration and rate-limited actions.Episode resets randomize STAT and SlewDR alongside ball mass and friction.Source: Figure 4 · p. 5

1) Real-world Metrics:

  • Full rotations count complete 360° rotations before a ball leaves the hand, with each full rotation comprising two successful ball exchanges.Time-to-drop measures elapsed time from both balls being seated until either leaves the hand.Source: 1) Real-world Metrics · p. 5
  • Rotational speed is R10 divided by min(10, tdrop), while exchange efficiency is the fraction of gait cycles yielding a completed exchange.Efficiency uses gait frequency and T=min(10, tdrop), and is bounded by 1 because at most one exchange occurs per cycle.Source: 1) Real-world Metrics · p. 5

2) Simulation-to-real Metrics:

Simulation comparisons use a common early evaluation window, report episode drop rate, and aggregate results across repeated episodes and training seeds.

2) Simulation-to-real Metrics:

  • Simulation rotational speed is measured as R10 divided by min(10, tsurv), matching the early window used for hardware comparisons.tsurv is episode survival time in seconds.Source: 2) Simulation-to-real Metrics · p. 5
  • Drop Rate is the fraction of simulation episodes in which either ball leaves the hand within 10 seconds.Source: 2) Simulation-to-real Metrics · p. 5
  • Simulation results aggregate 768 episodes across three training seeds; Figure 5 plots two joint trajectories over four rotation cycles for the stated policy conditions.The top row compares proprioception-only with and without balls in hand, and the bottom row compares C3-TacOff against C3.Source: 2) Simulation-to-real Metrics · p. 5, Figure 5 · p. 6

V. RESULTS

C3 sustains the strongest reported hardware performance among the described configurations, while removing tactile observations or using intermediate randomization configurations substantially reduces sustained rotation. C3 and C3-TacOff also show the closest simulation-to-hardware agreement in rotation speed.

A. Real-world Performance

  • 34.75 ± 26.06 full rotations, 94.8 ± 68.0 s time-to-drop, and 0.360 ± 0.061 rotation speed were achieved by C3 across 10 hardware trials.All trials reached at least two full rotations, and 9/10 continued beyond 10 seconds.Source: A. Real-world Performance · p. 6
  • Removing tactile observations from C3 reduced full rotations from 34.75 to 5.00 and time-to-drop from 94.8 s to 19.0 s.Rotation speed decreased from 0.36 to 0.256, while the separately trained proprioception-only policy achieved 4.40 rotations versus 1.70 for open-loop replay.Source: A. Real-world Performance · p. 6
  • C3 increased from 0.50 to 34.75 full rotations and from 4.1 to 94.8 s time-to-drop relative to C2.C1 and C2 had median rotation counts of 0.00 and 0.50, respectively, and median time-to-drop below 6 s; C1-to-C2 showed no measurable gain.Source: A. Real-world Performance · p. 6

B. Simulation-to-real Comparison

C. Long-Horizon Demonstration

The C3 policy sustained 112.5 full rotations in a separate 310-second hardware deployment. Comparisons with tactile input removed indicate that touch supports both successful exchanges and recovery from unfavorable ball configurations.

C. Long-Horizon Demonstration

  • 112.5 full 360° rotations were completed in 310 seconds in a separate C3 deployment outside the 10-trial evaluation.It used the same checkpoint and deployment conditions as that evaluation.Source: C. Long-Horizon Demonstration · p. 6

A. What is tactile providing?

  • C3-TacOff performed similarly to the separately trained proprioception-only policy, but had substantially lower exchange efficiency than C3 despite a similar gait frequency.C3-TacOff completed approximately 8.1 cycles versus 7.9 for C3.Source: A. What is tactile providing? · p. 6
  • C3 achieved approximately 7× as many rotations and remained active nearly 5× as long as C3-TacOff before dropping.In simulation, removing tactile input raised C3's drop rate from approximately 2% to 32%; tactile feedback also supported recovery from unfavorable ball configurations.Source: A. What is tactile providing? · p. 7
  • Tactile feedback supports exchange completion and recovery, while the learned gait can still execute without touch.Without tactile feedback, many gait cycles do not produce successful exchanges; when balls leave the nominal configuration, touch helps restore it.Source: A. What is tactile providing? · p. 7

B. How do these randomization enable transfer?

The randomisations address a mismatch between rapid simulated commands and slower hardware execution. Slew-rate randomisation improved transfer over C1, while adding STAT produced the most robust policy and reduced reliance on individual tactile channels.

B. How do these randomization enable transfer?

  • C1's simulated gait approached one rotation per second, but transferred poorly to hardware, where execution was substantially slower and difficult to sustain.The simulated actions were larger and more rapid than those observed on hardware.Source: B. How do these randomization enable… · p. 7
  • C1 crossed the approximate rotation-onset return mark at around 80 M training steps, versus approximately 160–170 M for C2 and C3.C1 also reached roughly twice their final return.Source: Figure 6 · p. 7
  • C2's slew-rate limit made motion smoother and easier for hardware to track, increasing mean hardware rotations over C1, but all C2 hardware trials still ended in a drop.The randomised rates made training harder and reduced simulated gait speed, bringing simulation and hardware speeds closer.Source: B. How do these randomization enable… · pp. 7–8
  • STAT combined with slew-rate randomisation outperformed previous policies across every measured hardware metric.Removing tactile input from the STAT-trained policy raised the simulation drop rate from 2.2% to 32.2%, less than the degradation observed for C2.Source: B. How do these randomization enable… · p. 8
  • STAT shifts the learned gait toward primary responsibility for motion, with tactile feedback supporting local adjustment and correction.This reduces the policy's dependence on continuous tactile input to sustain motion.Source: B. How do these randomization enable… · p. 8

C. Which Taxels are important?

Tactile activation is concentrated mainly around the middle and ring fingers, though its exact distribution shifts between simulation and hardware. Simulated contact frequency can guide sensor placement, but robust transfer depends less on exact active taxels than on avoiding reliance on any one channel.

C. Which Taxels are important?

  • Tactile activations concentrated around the middle and ring fingers, with hardware activations shifting toward proximal FSRs compared with simulation.The index finger and palm showed comparatively limited activation.Source: C. Which Taxels are important? · p. 8
  • Similar activation regions across simulation and hardware make simulated contact frequency a useful sensor-placement guide, but not a predictor of exact channelwise activation.STAT addresses the sensitivity to sensor faults or contact-density changes that can arise when a policy depends on individual channels.Source: C. Which Taxels are important? · p. 8

VII. CONCLUSIONS

  • The policy completed 225 consecutive 180° exchanges in one continuous run using proprioception and binary tactile contact without vision or object-state estimation.The conclusions identify FSR contact latency and missed registration of a smooth, fast-sliding ball as limitations that force a slower gait.Source: §VII · p. 8
Loading 2610.23438v2…