PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaDexTrack: Dynamic Modulation for Adaptive and Generalizable Dexterous Manipulation Tracking

Jianibieke Adalibieke, Qianwei Han, Xueyi Liu, Yuzhe Qin, Li Yi

arXiv:2610.33852v1cs.RO

TL;DR

Noisy synthesized or retargeted references and fixed-reference tracking make long-horizon, contact-rich hand–object interaction challenging. AdaDexTrack pairs a distilled generalist tracker with an in-loop modulator, and reports improvements across unseen trajectories and objects, including zero-shot sim-to-real transfer.

  • Problem

    Language-conditioned hand–object tracking must execute noisy synthesized or retargeted references over long horizons, where fixed-reference tracking allows small errors to accumulate.

    Source: §1 · pp. 1–2

  • Method

    AdaDexTrack distills RL-trained specialist trackers into a generalist and trains an aligned RL modulator to adjust references, object latents, and positional targets.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    AdaDexTrack outperforms prior SOTA on unseen-trajectory and unseen-object splits in average tracking error and success rate, and achieves zero-shot sim-to-real transfer.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The results support modulated tracking as a way to improve robustness and generalization across unseen trajectories and objects, including in zero-shot real-world transfer.

    Source: §1 · p. 2, §5 · p. 8

    In real-world settings, reliance on an explicit 6-DoF pose estimator makes the system struggle with symmetric objects whose pose is ambiguous.

    Source: §5 · p. 8

Abstract

from arXiv · show

Language is a natural way to command robots, but converting a single instruction into a long-horizon, contact-rich hand-object interaction remains challenging: synthesized references are noisy, human-to-robot retargeting introduces embodiment bias, and fixed-reference tracking lets small errors snowball. We address this with AdaDexTrack, a modulator-in-the-loop framework for language-conditioned manipulation tracking. A distilled generalist tracker serves as the skill carrier, while a tightly aligned modulator performs three feedback corrections: reference modulation (continual adjustment of what to track), object-latent modulation (online adaptation of the object representation to recruit suitable skills), and positional-target modulation (small state-dependent refinements for execution). The tracker is learned via large-scale specialist to generalist distillation on a corpus of language-conditioned hand-object trajectories; the modulator is trained with RL under the same task objective, ensuring tight coupling. Across large-scale evaluations, AdaDexTrack consistently outperforms prior SOTA on unseen-trajectory and unseen-object sets in both average tracking error and success rate, demonstrating robustness and generalization. We further show zero-shot sim-to-real transfer on real hardware, where adding the modulator yields substantial gains over a tracker-only variant. AdaDexTrack reframes language-conditioned dexterous manipulation as modulated tracking, replacing the open-loop, fixed-reference tracking with in-loop modulation that adjusts the reference, object latent, and positional target, yielding drift-resistant execution from noisy text references.

1. Introduction

Language-conditioned hand–object tracking must turn noisy references into stable, long-horizon execution, but fixed-reference tracking can let deviations accumulate. AdaDexTrack pairs a distilled generalist tracker with an aligned, in-loop modulator and reports gains across simulation and real-world evaluations.

1. Introduction

  • Fixed-reference tracking follows a noisy synthesized or retargeted reference without adapting to execution, allowing small errors to accumulate over long horizons.The paper describes this as a central challenge for contact-rich, language-guided manipulation.Source: §1 · pp. 1–2
  • AdaDexTrack combines a distilled generalist tracker, which carries learned skills, with a tightly aligned modulator that closes the loop around tracking.The tracker is distilled from RL-trained specialist policies that track synthesized text-guided hand–object trajectories.Source: §1 · p. 2
  • The modulator adjusts the reference, object latent, and positional targets online to re-align tracking, recruit suitable skills, and refine execution.It shares the tracker’s task objective and is trained with RL.Source: §1 · p. 2
  • AdaDexTrack outperforms prior SOTA on unseen-trajectory and unseen-object splits in average tracking error and success rate, with zero-shot sim-to-real transfer and gains over a tracker-only variant.The real-world comparison reports substantial gains on both unseen-trajectory and unseen-object splits.Source: §1 · p. 2

2. Related Works

Prior work uses hand–object references to pursue dexterous behavior, but reference generation and tracking each retain limitations. AdaDexTrack keeps the tracking interface while adding a tightly aligned modulator to adapt tracking online.

Dexterous Manipulation via HOI Reference

  • HOI references can come from teleoperation, human reconstruction, or text-to-motion synthesis, each with distinct scaling, labor, noise, or embodiment-gap trade-offs.Text-to-motion enables higher-scale language-conditioned generation despite synthesis and retargeting noise.Source: §2 · p. 2
  • Prior RL/IL trackers often learn per-trajectory policies or generalize narrowly; DexTrack broadens coverage through specialist distillation but retains a fixed test-time reference.The paper contrasts that fixed-reference setup with online modulation.Source: §2 · pp. 2–3

Skill Hierarchy in Dexterous Manipulation

  • Classical motion modulation often relies on hand-designed dynamical representations, while skill-hierarchy systems can leave tracking references and object representations outside the feedback loop.The paper characterizes many hierarchical stacks as loosely coupled and optimizing different objectives across levels.Source: §2 · pp. 2–3

3. Method

AdaDexTrack combines a distilled generalist tracker with a tightly aligned modulator that adjusts the reference, object latent, and positional targets during execution. The method builds the tracker from text-guided HOI references and uses feedback modulation to address drift and execution discrepancies.

3.1. Tracking Controller

  • Text-guided HOI references are synthesized and paraphrased, specialist RL trackers learn to follow them, and successful specialist trajectories are distilled into a generalist with behavior cloning.The paraphrases increase linguistic variety while remaining within DiffH2O’s annotation schema and generative distribution.Source: §3.1 · pp. 3–4

3.2. Tracking Modulator

  • The modulator continually adjusts near-future hand and object references, keeping them close to the original trajectory to counter accumulated tracking error and long-horizon drift.It uses near-future references at k ∈ {1, 2, 4, 12}.Source: §3.2 · p. 5
  • Object-latent modulation moves the tracker smoothly between specialist behaviors, making the latent a continuous control variable for interpolating between and switching across object anchors.The resulting continuous manifold expands a finite set of object-latent-conditioned skills.Source: §3.2 · p. 5
  • A small, bounded positional-target residual corrects execution-level discrepancies while leaving the tracker in charge of the skill’s low-frequency structure.The residual targets fast, local, high-frequency nonidealities.Source: §3.2 · p. 5

3.3. Sim-to-Real Transfer

  • For sim-to-real transfer, system identification tunes joint stiffness and damping to match real motion, then simulation randomizes robot and object dynamics, sensing, actions, and initial object pose.The pipeline also uses high-friction tape on fingertips and the object.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate AdaDexTrack against prior trackers on generalization, modulation ablations, training-data scale, and zero-shot real-world transfer. Results show gains across unseen splits and in real-world tests, while ablations examine the contribution of feedback channels.

4.1. Experiment Setup

  • The setup filters synthesized references for single-hand manipulation, collision avoidance, and workspace feasibility, then evaluates tracking with six error metrics and two success-rate variants.Training and testing include an unseen-trajectory split and a separate split holding out objects.Source: §4.1 · pp. 5–6

4.2. Generalizable Tracking via Adaptation

  • AdaDexTrack attains the highest success rates on both unseen-trajectory and unseen-object sets, outperforming all baselines.Its three-level feedback contrasts with fixed-reference DexTrack and the more loosely coupled ObjDex planner-over-controller.Source: §4.2 · pp. 6–7

4.3. Ablation Experiments

  • Adding modulator components consistently improves tracking performance on both unseen-trajectory and unseen-object sets.The tested variants add reference modulation, then positional-target modulation, then object-latent modulation.Source: Table 1 · p. 6, §4.3 · p. 7
  • Reference modulation updates the reference after drift, creating a low-error rejoin point that stabilizes long-horizon tracking.Object-latent modulation favors larger latents for open fingers during reach and smaller latents for grasp.Source: §4.3 · p. 7, Figure 4 · p. 8, Figure 5 · p. 8
  • Performance increases steadily with training-data size and surpasses the tracker-only baseline on both evaluated splits.The dataset is downsampled from the full set, with a separate model trained from scratch for each scale.Source: §4.3 · pp. 7–8, Figure 6 · p. 8

4.4. Sim-to-Real Experiments

  • Zero-shot real-world tests show higher completion rates than the tracker-only baseline on both unseen-trajectory and unseen-object sets.The evaluation excludes axially symmetric objects because FoundationPose has rotational ambiguity on them.Source: §4.4 · p. 8, Table 2 · p. 8

5. Conclusions and Limitations

AdaDexTrack combines a tracker distilled from many experts with three tightly coupled modulation interfaces to compose skills and suppress long-horizon drift. It surpasses baselines on unseen trajectories and objects and achieves zero-shot sim-to-real transfer, while its real-world use is limited by pose-estimation ambiguity on symmetric objects.

5. Conclusions and Limitations

  • AdaDexTrack uses a distilled expert tracker and reference, object-latent, and positional-target modulation to compose skills and suppress long-horizon drift.The tracker and modulator are trained under the same task objective.Source: §5 · p. 8
  • The method surpasses baselines on unseen trajectories and objects, improves with dataset scale, and achieves zero-shot sim-to-real transfer.In real-world settings, explicit 6-DoF pose estimation remains problematic for symmetric objects because their pose is ambiguous.Source: §5 · p. 8
  • A pose-free route proposed by the authors uses point clouds and symmetry-aware, SE(3)-equivariant embeddings for reference matching and latent modulation.Source: §5 · p. 8

A.1. Data Preprocessing

The appendix describes how the trajectory corpus is filtered and augmented, how robot references and object latents are formed, and how the transfer and policy-training pipeline is prepared. Specialist policies are grouped by semantic intent before generalist training.

A.1. Data Preprocessing

  • Filtering 505 base demonstrations and augmenting each with paraphrased descriptions yields 2,765 valid text–trajectory pairs spanning 50 objects.The filters retain single-hand, collision-free sequences within the robot’s reachable workspace; paraphrased descriptions are resynthesized with DiffH2O.Source: §A.1 · p. 12
  • Kinematic retargeting matches MANO and robot-hand keypoints by least squares, while a Chamfer-trained point-cloud autoencoder supplies the online-modulated object latent.Source: §A.1 · p. 12

A.2. Sim-to-Real Transfer

  • System identification fits each joint’s stiffness and damping by matching simulated and real responses to swept sinusoidal commands; training also randomizes dynamics, objects, observations, and actions.Figure 8 reports close agreement between simulated and measured joint responses.Source: §A.2 · pp. 12–13, Figure 8 · p. 13

A.3.1. Specialist Policy

  • Specialists are trained jointly on semantically grouped trajectory variants, reducing the number of policies while exposing each to multiple motion and language variants.The 2,765 references are grouped by their underlying base demonstration; each specialist trains with 8,192 parallel environments and converges in about three hours.Source: §A.3.1 · p. 14

A.3.2. Generalist Policy

  • The generalist policy is trained by behavior cloning with the same observation space as the specialists and converges in about two days on eight NVIDIA A10 GPUs.Source: §A.3.2 · p. 14

A.4. Tracking Modulator Training.

The modulator is trained with the generalist frozen and the specialist task reward, while the appendix specifies the training segment and outlines three baseline implementations. These baselines differ in whether they use fixed references, hierarchical wrist planning, or a single PPO policy.

A.4. Tracking Modulator Training.

  • The frozen generalist is paired with a modulator trained on the same reward, changing tracker inputs and adding a small residual to its action.The modulator is trained with 8,192 parallel environments and converges in about two days.Source: §A.4 · p. 14

B.1. Environment Setup

  • Training uses the 300 frames following the first frame where the minimum hand-mesh-to-object-center distance exceeds 20 cm.The simulator runs at 60 Hz with a 12 Hz control frequency.Source: §B.1 · p. 14

B.2.1. DexTrack

  • The DexTrack reproduction keeps references and object latents fixed at test time and uses time-indexed specialist actions as imitation targets alongside PPO.The reproduction omits homotopy-based demonstration mining because the original procedure is not publicly available.Source: §B.2.1 · pp. 14–15

B.2.2. ObjDex

  • The ObjDex reproduction uses a planner to predict wrist trajectories and a low-level controller that tracks them, without reference replanning or modulation.For unseen objects, the planner uses a seen surrogate object ID from the same category.Source: §B.2.2 · p. 15

B.2.3. Vanilla RL (PPO)

  • Vanilla RL is a single PPO policy trained across all training references with the specialist action space, observation space, and reward.Source: §B.2.3 · p. 15

B.3. Qualitative Results

The qualitative comparisons favor AdaDexTrack’s hand–object tracking, while the real-world setup excludes objects with continuous pose ambiguity unless texture stabilizes their frame. The textured symmetric-object evaluation reports that the full method substantially outperforms its no-modulator variant.

B.3. Qualitative Results

  • On unseen trajectories, AdaDexTrack gives the most faithful hand–object tracking; Vanilla RL and DexTrack often drift or fail, while ObjDex produces unnatural hand motions.Source: §B.3 · p. 15

C.1. Environment Setup

  • The real-world setup uses RGB-D sensing and FoundationPose, excludes axially symmetric objects with continuous rotational ambiguity, and includes discretely symmetric objects when pose estimates are stable.Added asymmetric texture breaks the visual symmetry of axially symmetric objects and enables a stable object frame.Source: §C.1 · p. 16, Figure 10 · p. 16

C.2. Metrics

  • Real-world Success Rate uses the simulation thresholds and reports average completion over trajectories.Source: §C.2 · p. 16

C.3. Textured Axially Symmetric Objects

  • On eight textured axially symmetric objects, the full method substantially outperforms the no-modulator variant across five unseen trajectories per object.The evaluation reports average completion rate over all trials; asymmetric texture stabilizes pose estimates by breaking rotational symmetry.Source: §C.3 · p. 16, Table 6 · p. 16

C.4. Real-World Evaluation Without Object Tape

Without object tape, AdaDexTrack still substantially outperforms the tracker-only variant in real-world completion rates. Removing tape increases success rates for both methods, which the authors attribute to less friction allowing grasp-adjustment micro-sliding.

C.4. Real-World Evaluation Without Object Tape

  • The evaluation removes tape, keeps the rest of the setup unchanged, and reruns the unseen-trajectory and unseen-object settings.Completion rates are reported for the resulting tests.Source: §C.4 · p. 16, Table 7 · p. 16
  • Removing tape increases success rates for both methods, while AdaDexTrack still substantially outperforms the variant without the modulator.The authors attribute the increase to untaped objects allowing more natural micro-sliding during contact.Source: §C.4 · p. 16
  • The authors attribute the higher success rates without tape to excessive friction hindering the small relative motions needed for grasp adjustment.The full method’s advantage over the tracker-only variant remains without artificial adhesion.Source: §C.4 · p. 16

C.5. Real-World Evaluation with Practical Object Geometry

Real-world tests with practical object-geometry inputs retain strong performance across both evaluation sets. Reconstructed geometry is comparable to ground truth, and the method remains robust with segmented partial point clouds; qualitative comparisons also show robustness against the tracker-only baseline.

C.5. Real-World Evaluation with Practical Object Geometry

  • Using geometry reconstructed from a single RGB image yields performance comparable to ground-truth geometry on the unseen-trajectory and unseen-object sets.InstantMesh reconstructions are used for policy inference and FoundationPose-based pose tracking, without object tape.Source: §C.5 · p. 17, Table 8 · p. 17
  • With segmented partial point clouds from single-view depth, AdaDexTrack remains robust and achieves strong performance on both evaluation sets.The setting is more challenging because of incomplete geometry and self-occlusion.Source: §C.5 · p. 17, Table 8 · p. 17

C.6. Qualitative Results

  • Figure 14 provides qualitative comparisons with the tracker-only baseline on the unseen-trajectory set, highlighting AdaDexTrack’s robustness.Source: §C.6 · p. 17

D.1. Object Latent Smoothness

Object latent modulation is intended as a continuous interface for composing skills rather than a discrete object identifier. Perturbation analysis finds locally smooth tracker responses, supporting interpolation between latent anchors and phase-dependent adjustments to grasping behavior.

D.1. Object Latent Smoothness

  • Object latent modulation is intended to recruit suitable behavior primitives continuously, rather than identify an object discretely.The tracker is conditioned on the object latent, which the modulator adjusts online for the current interaction phase.Source: §D.1 · p. 17
  • The analysis perturbs only the object latent while holding robot state, object pose, and tracking reference fixed, then measures the resulting action change.It samples 200 unseen-trajectory observations at five timesteps each, for 1,000 observations total.Source: §D.1 · p. 17
  • The response ratio remains bounded across samples and perturbation directions, indicating that the tracker responds smoothly to local object-latent changes.The analysis treats this local smoothness as support for meaningful interpolation between latent anchors.Source: §D.1 · p. 17, Figure 12 · p. 17
  • Small latent adjustments can recruit different grasping behaviors, such as a more open hand during approach and a tighter hand during grasp.The cited qualitative behavior uses the latent as a phase-dependent knob for hand openness without changing the reference or relying on large action corrections.Source: §D.1 · p. 17

Related papers

Loading 2610.33852v1…