Source-linked AI summary
TacBPM: A Tactile-conditioned Behavior Prior Model for Dexterous Reorientation
Jie Yin, Wanli Xing, Zeyuan Zhao, Xuezhou Zhu, Zhijie Deng, Kaifeng Zhang
TL;DR
Dexterous reorientation requires contact-rich policies that adapt finger gaits to geometry, pose, and tactile feedback. TacBPM distills multi-scale specialists into a tactile-conditioned prior reused through residual latent actions. Across the reported tasks, it improves exploration over raw joint-action learning, with raw-action PPO rarely succeeding on complex objects within the matched budget.
Problem
Dexterous reorientation requires closed-loop contact strategies, while existing policies often remain strongest in narrower regimes and downstream learning must re-explore low-level joint commands.
Source: §I · p. 1Method
TacBPM distills multi-scale sphere specialists into a tactile-proprioceptive latent controller and reuses its fixed prior through residual latent corrections, with decoder adaptation for new in-hand geometries.
Source: §I · pp. 1–2Results
TacBPM improves exploration over raw joint-action learning across in-hand transfer, commanded-axis rotation, and an arm-hand extension, while raw-action PPO rarely succeeds on complex-object reorientation within the matched budget.
Source: 2) Transfer results · p. 5, §V · p. 8Takeaways & Limitations
The results indicate that a contact-conditioned latent prior can be reused across geometry and task changes, with decoder adaptation supporting in-hand transfer to new geometries.
Source: §V · p. 8
Abstract
from arXiv · showhide
Dexterous in-hand manipulation requires policies that coordinate high-DoF hand joints through intermittent, contact-rich interaction. Beyond target-orientation tracking, such policies must discover finger gaits that preserve object stability while adapting to geometry, anisotropy, pose, contact, and sensing changes. We propose \method, a tactile-conditioned behavior prior model for dexterous reorientation. \method distills multi-scale sphere specialists into a latent controller and lets downstream policies reuse the fixed tactile prior through residual latent actions, reducing renewed exploration from raw joint commands. The prior conditions on tactile-proprioceptive history so latent behavior reflects the current hand-object interaction. We evaluate arbitrary-pose transfer across anisotropic objects, commanded-axis rotation, and an arm-hand Grasp-to-AnyPose task in which the robot must grasp, lift, transport, and reach goal poses for novel tool geometries and generalized placements. Extensive experiments demonstrate that the proposed method accelerates training and enables stable policies where matched raw-action PPO remains near failure, with successful sim-to-real transfer in in-hand and arm-hand tasks.
I. INTRODUCTION
Dexterous reorientation requires closed-loop contact strategies that adapt to object geometry, pose, and fingertip feedback. TacBPM distills multi-scale specialists into a tactile-conditioned prior reused through residual latent control for broader reorientation tasks.
I. INTRODUCTION
- In-hand reorientation must adapt contact strategies to changing object motion, hand configuration, and fingertip feedback across shape, scale, pose, friction, and sensing variation.The task settings include arbitrary target orientations, six signed-axis rotation modes, and arm-hand grasp acquisition, transport, and goal-pose reaching.Source: §I · p. 1
- Existing specialist policies often remain tied to fixed shapes, known grasps, particular feedback, or complete-actor transfer, leaving downstream learning to re-explore fragile low-level joint commands.Random actions may break contact or fail to discover stabilizing finger gaits.Source: §I · p. 1
- TacBPM distills eight sphere specialists spanning scale factors 0.3 to 1.0 and conditions latent behavior on recent hand state, previous targets, and tactile contact magnitudes and positions.For arm-hand validation, the method paradigm also uses scale-randomized rubber-hammer teachers and diverse tool geometries.Source: §I · p. 1
- TacBPM improves exploration over raw joint-action learning across in-hand transfer, commanded-axis rotation, and an arm-hand extension.The introduction frames the tasks as tests of arbitrary-pose intent, transfer across scale and anisotropic shape, and validation across embodiments.Source: §I · pp. 1–2
- The downstream policy keeps the tactile prior frozen and learns residual latent corrections, while in-hand decoder finetuning adapts to new geometries without changing the contact-conditioned prior.The prior anchors exploration near contact-preserving interaction patterns, while the policy selects low-dimensional corrections.Source: §I · pp. 1–2
- The contribution is a variational prior-decoder controller that adapts latent behavior to contact magnitude, contact position, and scale-induced contact changes.The evaluation spans unseen sphere-scale extrapolation, anisotropic-object transfer, commanded-axis rotation, and arm-hand goal-pose reaching.Source: §I · p. 2
II. RELATED WORK
Prior work offers reusable latent skills, tactile observations, and dexterous reorientation policies, but TacBPM combines tactile-conditioned behavior priors with residual-latent reuse across tasks.
A. Reusable Skill Representations
- Reusable skill representations compress high-dimensional motion into latent actions, but prior humanoid and character-control work primarily models proprioceptive whole-body locomotion rather than fingertip-conditioned hand-object interaction.TacBPM instead distills sphere specialists and exposes residual latent corrections rather than transferring a complete executable actor.Source: A. Reusable Skill Representations · p. 2
B. Tactile-Proprioceptive Control
- Tactile systems use contact signals for manipulation and state inference, while TacBPM uses tactile-proprioceptive history to parameterize latent control itself.Unlike PTLD's recovery of privileged policy state for deployment, TacBPM studies a reusable tactile behavior prior and decoder finetuning for transfer.Source: B. Tactile-Proprioceptive Control · p. 2
C. Dexterous In-Hand Rotation and Reorientation
- Many dexterous reorientation methods optimize task-specific policies in raw-action spaces, so changing object shapes, sensing assumptions, or task definitions often requires retraining and reward retuning.TacBPM is complementary: it compresses contact-rich motion patterns into a tactile-conditioned prior for residual-latent reuse.Source: C. Dexterous In-Hand Rotation and… · p. 2
III. METHOD
TacBPM compresses contact-rich finger gaits into structured latent control, distills behavior from multi-scale specialists, and represents tactile-proprioceptive history for contact-conditioned behavior.
A. Overview
- TacBPM learns a tactile-conditioned Gaussian prior and deterministic decoder, then lets downstream policies use residual latent codes instead of direct 22-D actions.The in-hand setup reuses a fixed prior while adapting the decoder; the arm-hand study applies the same teacher-distillation and residual-latent paradigm.Source: A. Overview · p. 2
B. Specialist Policy Training
- Eight sphere specialists use scale factors from 0.3 to 1.0, varying hand aperture, contact distribution, and rolling and regrasping regimes while keeping object geometry simple.Online multi-teacher distillation assigns environments to teachers and interleaves their target actions in a DAgger-style batch aligned with student rollouts.Source: B. Specialist Policy Training · p. 2
C. Tactile-Conditioned State Representation
- The student state stacks three recent frames of joint positions, previous targets, tactile contact magnitudes, and 3D contact positions into a 192-dimensional history.The normalized tactile-proprioceptive feature is used by the deployed prior and decoder; during distillation, the encoder also receives an object longest-side cue.Source: C. Tactile-Conditioned State… · pp. 2–3
D. Specialist-to-Generalist Variational Distillation
TacBPM distills sphere specialists into an encoder-prior-decoder latent controller conditioned on tactile-proprioceptive features. Online distillation matches specialist actions while regularizing latent predictions and temporal smoothness.
D. Specialist-to-Generalist Variational Distillation
- TacBPM uses an encoder, prior, and decoder: the encoder and prior predict task-conditioned and task-agnostic latent distributions, and the decoder outputs deterministic hand position targets.The latent controller receives tactile-proprioceptive features extracted from the current state.Source: D. Specialist-to-Generalist… · p. 3
- The posterior and prior are diagonal Gaussians; distillation samples posterior latents with reparameterization, while deterministic evaluation uses the corresponding mean.Source: D. Specialist-to-Generalist… · p. 3
- Online distillation rolls out the student in the environment and matches its actions to those of the assigned sphere-scale specialist at the same state.The objective combines action matching, KL regularization to the learned prior, and temporal smoothness over consecutive posterior means.Source: D. Specialist-to-Generalist… · p. 3
- The KL term trains the prior to predict task-relevant latent codes without the task goal, while temporal regularization discourages discontinuous latent jumps.Source: D. Specialist-to-Generalist… · p. 3
E. Residual Latent Downstream Control
Downstream control keeps the tactile prior fixed and uses residual latent actions for task-driven adaptation. Grasp-to-AnyPose applies the prior-centered paradigm to arm-hand grasping, transport, and goal-pose reorientation.
E. Residual Latent Downstream Control
- Downstream PPO samples residual latent actions around the fixed tactile prior, constraining exploration while allowing task-driven deviations.The prior and state normalizers stay frozen; the downstream policy and value heads and VAE decoder are trainable for in-hand adaptation.Source: E. Residual Latent Downstream Control · p. 3
- Decoder finetuning adapts latent-command realization to new geometry without changing the contact-conditioned prior.A frozen-decoder ablation isolates the effect of decoder adaptation.Source: E. Residual Latent Downstream Control · p. 3
F. Arm-Hand Embodiment Validation
- Grasp-to-AnyPose tests the paradigm on a right arm and hand that grasp, transport, and reorient objects to sampled goal poses.A separate tactile-conditioned controller is distilled from rubber-hammer teachers with scale randomization, then evaluated on four tool geometries and varied placements and goals.Source: F. Arm-Hand Embodiment Validation · p. 3
- The arm-hand reward progresses from fingertip approach and lifting to reducing maximum object-to-goal keypoint distance, with a sparse bonus inside tolerance.Action, object-velocity, and arm-table-collision penalties regularize behavior; downstream training omits the object-angular-velocity penalty.Source: F. Arm-Hand Embodiment Validation · pp. 3–4
G. Training Details
The experiments evaluate whether a distilled contact prior transfers across scales, shapes, control commands, and embodiments. The setup standardizes comparisons while varying the downstream task and ablations.
G. Training Details
- The in-hand controller uses 22-DoF position targets, a 16-D Gaussian latent action, and encoder, prior, and decoder MLPs with hidden sizes (512, 256, 128).Its three-frame tactile-proprioceptive history includes joints, previous targets, contact magnitudes, and contact locations; distillation uses teacher action matching, KL annealing, and temporal smoothing.Source: G. Training Details · p. 4
IV. EXPERIMENTS
- The experiments ask whether multi-scale sphere teachers extrapolate, tactile residual-latent control transfers to complex objects, signed-axis commands work in simulation and hardware, and the paradigm supports arm-hand goal reaching.The four questions cover prior coverage, complex-object transfer, commanded-axis rotation and hardware transfer, and arm-hand validation.Source: §IV · p. 4
A. Experimental Setup
- In-hand teachers use eight sphere scales from 0.3 to 1.0; evaluation includes unseen scales and shapes excluded from teacher training, ordinarily over 1,000 randomized attempts.The common simulation protocol uses the same dexterous hand model, initialization protocol, and termination rules within each benchmark.Source: A. Experimental Setup · p. 4
- Figure 2 depicts sphere-teacher rollouts and downstream arbitrary-pose rollouts across multiple object instances.The figure contrasts teacher sphere scales with downstream objects of different scales, curvatures, and faceted structures.Source: Figure 2 · p. 4, A. Experimental Setup · p. 4
- Downstream policies add residual latent codes to the frozen prior mean and decode them into 22-DoF hand commands; the main setting finetunes the decoder while strict reuse freezes it.These comparisons separate latent-prior, tactile-conditioning, and decoder-adaptation effects.Source: A. Experimental Setup · p. 4
- Scratch baselines and TacBPM share rewards, curricula, environment counts, PPO settings, training budgets, initialization, and evaluation protocols within each benchmark.Metrics include success, drop rate, episode length, and rotation progress; success requires reaching within 0.2 rad before timeout or drop.Source: A. Experimental Setup · p. 4
B. Sphere-Scale In-Hand-to-AnyPose Extrapolation
Multi-scale sphere specialists provide a prior that can transfer to unseen scales and shapes, though the results show limits beyond teacher support. Tactile conditioning improves performance under contact-regime shifts.
B. Sphere-Scale In-Hand-to-AnyPose Extrapolation
- Teacher rollouts span sphere sizes and orientations; downstream rollouts extend to non-spherical objects, while a remaining gap on sphere 1.2 and several novel shapes marks an extrapolation limit.Figure 4 shows representative arbitrary-pose rollouts across six object geometries.Source: B. Sphere-Scale In-Hand-to-AnyPose… · p. 4, Figure 4 · p. 5
- On seen scales, prior-decoder average success rose from 26.03% to 93.67%, then to 95.62% with tactile conditioning; on unseen scales and shapes, average success rose from 40.05% to 60.38%.Table I evaluates seen sphere scales, unseen sphere scales, and shapes excluded from teacher training.Source: Table I · p. 5, B. Sphere-Scale In-Hand-to-AnyPose… · p. 5
- Downstream latent commands often remain near the sphere-prior cloud, while boundary excursions mark task-driven deviations.The t-SNE projection visualizes sphere-prior samples alongside executed downstream task latents.Source: B. Sphere-Scale In-Hand-to-AnyPose… · pp. 4–5, Figure 3 · p. 5
- Raw-action PPO succeeds on the easiest mid-size sphere but is unstable across most other scales under the matched protocol.The passage identifies direct exploration in the 22-D action space as a bottleneck in this setting.Source: B. Sphere-Scale In-Hand-to-AnyPose… · p. 5
C. In-Hand-to-AnyPose Transfer on Complex Objects
TacBPM transfers to anisotropic complex objects under arbitrary-pose reorientation, where raw-action PPO rarely succeeds within the matched budget. Decoder finetuning improves several single-object results, while a frozen decoder performs best in the shared Multi setting.
1) Task and baselines:
- The task tests arbitrary target orientations across edges, elongated bodies, irregular shapes, and pose-dependent contact affordances, using matched protocols for all methods.The reward combines rotational progress and a goal-tolerance bonus with penalties for drift, large actions, non-nominal posture, torque, work, and falls.Source: 1) Task and baselines · p. 5
2) Transfer results:
- Raw-action PPO rarely succeeds (0.80–2.10% on single objects; 1.00% in Multi), while the single-scale prior reaches 65.80% in Multi, below both multi-scale TacBPM variants.Actor-level transfer reaches 40.00% in Multi, while action-space residual learning reaches 8.80% in Multi.Source: 2) Transfer results · p. 5
- Figure 5 plots episode reward and episode length across evaluated objects and Multi for the complex-object transfer ablations.Source: Figure 5 · p. 6
- Decoder finetuning gives the highest single-object success on block 03, block 05, bottle 01, and trash can 02, improving over the frozen decoder by 5.8, 6.7, 2.7, and 1.8 percentage points, respectively.The frozen decoder instead achieves the highest Multi success, suggesting strict decoder reuse can regularize a shared policy across broader object variation.Source: 2) Transfer results · p. 6
D. Axis-Conditioned Rotation
A single latent controller is evaluated on six signed Cartesian rotation commands. In simulation, TacBPM has the highest axis-average rotation across six objects, while tactile-free variants often outperform raw-action exploration.
1) Commanded-axis protocol:
- One policy follows six signed Cartesian-axis commands while maintaining stable contact and motion gaits, with rewards encouraging commanded-axis angular velocity and penalizing orthogonal rotation.The VAE prior is frozen and the decoder finetuned under object, contact, mass, actuation, noise, push, and gravity randomization.Source: 1) Commanded-axis protocol · p. 6
2) Simulation results:
- TacBPM obtains the highest axis-average rotation across six simulated objects, while raw-action exploration struggles on bottle 02, block 08, and block 14, where signed angle is near zero.The no-tactile variant often exceeds raw-action exploration, but the remaining gap reflects the value of contact-state information; variability includes off-axis motion or contact loss.Source: 2) Simulation results · p. 6, Table III · p. 6
- Figure 6 shows rollouts for all six signed axes, illustrating that the same policy must change direction and contact strategy without switching controllers.Source: Figure 6 · p. 6, 2) Simulation results · p. 6
3) Real-robot protocol and results:
Real-robot commanded-axis tests show TacBPM's strongest performance on most object-axis pairs, with failures including contact drift and drops. In arm-hand simulation, residual-latent control learns faster than raw-action exploration on novel tool geometries.
3) Real-robot protocol and results:
- Hardware episodes run tactile and proprioceptive feedback at 20 Hz, omit object-pose feedback from the in-hand action loop, and count success as over 180° commanded-direction rotation within 20 seconds.Each object-axis entry uses 10 episodes; drops or stuck states before the threshold count as failures.Source: 3) Real-robot protocol and results · p. 6
- On hardware, TacBPM performs strongest on most object-axis pairs, with pronounced improvements on the corner block, standard tennis, and unseen multiface object.The no-tactile policy struggles on lateral y-axis commands for corner block and small tennis; command switching before stable contact is re-established can cause a drop.Source: 3) Real-robot protocol and results · p. 7, Figure 7 · p. 7, Table IV · p. 7
1) Task and evaluation:
- Arm-hand evaluations use 1024 deterministic episodes per object, counting failed episodes as 20 cm position error and 20° rotation error.The simulation evaluation covers small hammer, blue brush, staples marker, and mallet hammer, transferred from the rubber-hammer teacher family.Source: 1) Task and evaluation · p. 7, Table V · p. 7
- For four novel tool geometries, residual-latent policies reach higher success and reward earlier than raw-action exploration, including on blue brush and staples marker.Figure 8 plots success rate and episode reward across the four geometries; raw-action exploration often fails to discover stable grasp-to-goal behavior within the same horizon.Source: 1) Task and evaluation · p. 7, Figure 8 · p. 7
2) Simulation evaluation and hardware demonstration:
In simulation, TacBPM improves arm-hand grasp-to-goal performance across four novel tool categories and accelerates optimization under the same protocol. The hardware demonstration shows sim-to-real transfer for the full grasp–lift–goal sequence, but is not a large-scale benchmark.
2) Simulation evaluation and hardware demonstration:
- The hardware rollout demonstrates transfer for the complete grasp–lift–goal sequence, but remains a qualitative result rather than a large-scale real-robot benchmark.Source: 2) Simulation evaluation and hardware… · p. 7
- TacBPM improves success on all four novel tool categories and reduces capped position and rotation errors on blue brush and staples marker, where matched raw-action PPO rarely reaches stable goal-pose behavior.The staples marker is the clearest raw-action failure case, at 0.49% success with errors near their caps.Source: 2) Simulation evaluation and hardware… · p. 7
- The results indicate faster optimization and a higher empirical performance ceiling under the same protocol, while the figure and videos provide complementary evidence of successful sim-to-real transfer.Source: 2) Simulation evaluation and hardware… · p. 8
V. CONCLUSION
- The conclusion reports effective reuse of the teacher-distillation and residual-latent paradigm across in-hand and arm-hand settings, with separately trained teachers and priors for the arm-hand study.Source: §V · p. 8