PaperlayerView exact revision on arXiv

Source-linked AI summary

OccluDex: Hierarchical 3D Visuo-Tactile Representation Learning for Egocentric Dexterous Manipulation under Self-Occlusion

Ziheng Xu, Yueyuan Chen, Xinyuan He, Guoxing Liu, Yuanshuo Tan, Huiming Pan, Bin He, Shuo Jiang, Peter B. Shull

arXiv:2609.39017v1cs.RO

TL;DR

Egocentric hand–object occlusion limits visual evidence for estimating geometry and contact during dexterous manipulation. OccluDex learns a hierarchical 3D visuo-tactile representation and transfers its pretrained encoder to downstream control. It improves average success across two simulated tasks and transfers zero-shot to a physical hand on unseen objects.

  • Problem

    Dynamic self-occlusion limits changing visual evidence, while tactile cues are local and sparse, leaving a need for control-relevant representations that combine global geometry and contact information.

    Source: §I · pp. 1–2

  • Method

    OccluDex hierarchically encodes partial 3D geometry, fuses tactile contact tokens with high-level geometric features, and transfers its pretrained encoder as a frozen backbone for downstream PPO.

    Source: §I · p. 2

  • Results

    86.2% average success across two tasks, compared with 75.8% for VTDexManip, 73.0% for Base, and 65.6% for VTT-3D; real unseen-object success rates were 85.0% and 70.0%.

    Source: C. Comparison on Downstream Dexterous… · p. 6, §V · p. 8

  • Takeaways & Limitations

    OccluDex is presented as a promising perceptual foundation for robust and transferable egocentric dexterous control under dynamic self-occlusion.

    Source: §V · p. 8

Abstract

from arXiv · show

Reliable dexterous manipulation requires continuous estimation of object geometry and hand-object contact throughout interaction. With egocentric sensing, however, the manipulating hand frequently occludes task-relevant object surfaces and contact regions, reducing the visual evidence available for state estimation and thereby making robust closed-loop control and generalization to unseen object geometries particularly challenging. To address this, we present OccluDex, a hierarchical 3D visuo-tactile representation learning framework that integrates global geometric structure with local contact information for robust manipulation under dynamic self-occlusion during hand-object interaction. OccluDex adopts multi-scale masked autoencoding to progressively encode partial 3D geometry and fuses tactile contact tokens with high-level geometric features through cross-modal attention. The encoder is pretrained from synchronized human visuo-tactile demonstrations and transferred as a frozen perceptual backbone for downstream reinforcement learning. We evaluate OccluDex on a faucet rotation task, requiring one full clockwise handle revolution, and a tabletop object reorientation task, requiring a 180-degree tabletop object reorientation without toppling. In simulation experiments, OccluDex demonstrated 12.6% higher accuracy for unseen objects and 8.3% higher accuracy for previously seen objects than the strongest state-of-the-art baseline models. Physical experiments were further performed with a Shadow Hand to demonstrate successful zero-shot sim-to-real generalization on unseen physical objects. This results could enable humanoid egocentric object manipulation for seen and unseen objects even when the manipulating robotic hand occludes vision.

I. INTRODUCTION

Dynamic hand–object self-occlusion limits visual evidence and makes robust egocentric dexterous control challenging. OccluDex combines hierarchical 3D geometry and tactile contact information in a learned representation for manipulation under occlusion.

I. INTRODUCTION

  • Dynamic self-occlusion makes visual evidence limited and changing, while geometric completion alone does not provide the instantaneous contact information needed for dexterous control.The challenge is to learn a control-relevant representation directly from egocentric observations without explicit 3D reconstruction.Source: §I · p. 1
  • OccluDex hierarchically encodes point clouds, fuses tactile contact tokens with high-level geometric features through cross-modal attention, and jointly reconstructs masked geometric and tactile observations during pretraining.The pretrained encoder is frozen and transferred as the perceptual backbone for a downstream PPO policy.Source: §I · p. 2
  • The authors collect egocentric human manipulation demonstrations pairing partial 3D point clouds with tactile observations under natural hand–object occlusion.The paired signals provide geometric–contact supervision.Source: §I · p. 2
  • OccluDex improves dexterous manipulation and generalization across seen and unseen settings on Faucet Rotation and Tabletop Object Reorientation, with zero-shot transfer to a physical Shadow Dexterous Hand.The evaluation includes previously unseen physical objects; Fig. 1 summarizes the simulation and real-world settings.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORK

Related work addresses egocentric occlusion through external views or geometry reconstruction, while visuo-tactile methods combine geometric and contact information. Structured 3D visuo-tactile learning for multi-fingered dexterous manipulation remains underexplored.

A. Egocentric Perception for Dexterous Manipulation

  • Hand–object occlusion can make visual geometry incomplete or ambiguous, and existing approaches use external viewpoints or reconstruction despite continuously changing, limited egocentric evidence.The stated challenge is learning a control-relevant representation directly from egocentric observations without explicit 3D reconstruction.Source: A. Egocentric Perception for Dexterous… · p. 2

B. Visuo-Tactile Representation Learning

  • Tactile sensing provides local interaction information but limited global object geometry, motivating methods that combine visual observations and tactile feedback.Existing visuo-tactile approaches are predominantly based on image representations, with spatial structure encoded implicitly.Source: B. Visuo-Tactile Representation Learning · p. 2
  • Structured 3D visuo-tactile learning for multi-fingered dexterous manipulation remains underexplored, as prior 3D approaches have primarily studied parallel-jaw grippers.The passage distinguishes the simpler contact patterns and interaction dynamics of parallel-jaw grippers from multi-fingered hands.Source: B. Visuo-Tactile Representation Learning · p. 2

III. METHODS

OccluDex combines hierarchical 3D geometry with tactile contact tokens, fusing them at the highest semantic stage. Its geometric hierarchy aggregates local regions into coarser representations and propagates visibility consistently across scales.

III. METHODS

  • OccluDex aggregates local surface structures into higher-level spatial representations and fuses sparse tactile contact tokens with global 3D context at the highest semantic stage.During pre-training, a hierarchical decoder reconstructs jointly masked geometric and tactile observations; the decoder is then discarded and the frozen encoder transferred to a downstream PPO policy.Source: §III · pp. 2–3

A. Hierarchical Multi-scale Geometric Encoding

  • Recursive farthest-point sampling and k-nearest-neighbor grouping construct multi-scale geometric regions from the input point cloud.At deeper stages, token merging reduces token count while increasing feature dimensionality to represent larger regions with richer context.Source: A. Hierarchical Multi-scale Geometric… · pp. 3–4
  • Progressive masking samples visibility at the coarsest scale and propagates it toward finer scales through hierarchical neighborhood correspondences.This maintains consistent visibility across resolutions; the masking objective is not intended to simulate physical self-occlusion.Source: A. Hierarchical Multi-scale Geometric… · p. 3
  • Fig. 3 depicts recursive multi-scale grouping and visibility propagation from the highest semantic scale to finer scales.Visible tokens are grouped and merged according to hierarchical neighborhood indices, keeping them aligned with the same visible regions.Source: A. Hierarchical Multi-scale Geometric… · p. 3, Figure 3 · p. 4

B. Tactile Token Embedding

Tactile observations are encoded as channel-specific tokens that preserve localized contact information.

B. Tactile Token Embedding

  • Each of the 19 tactile channels is binarized using a fixed activation threshold, emphasizing contact activation while reducing sensitivity to force-magnitude variation.Source: B. Tactile Token Embedding · p. 4
  • Each tactile channel is mapped to a C-dimensional latent token with a sensing-location positional embedding and a learnable tactile-modality embedding.Source: B. Tactile Token Embedding · p. 4
  • The resulting 19-token sequence preserves sensor-specific contact information for subsequent visuo–tactile interaction.Source: B. Tactile Token Embedding · p. 4

C. Cross-modal Visuo–Tactile Fusion

OccluDex delays visuo–tactile fusion until geometric features have reached the highest semantic stage, where they provide global context for sparse tactile cues.

C. Cross-modal Visuo–Tactile Fusion

  • Tactile tokens are independently masked during pre-training, and only visible tactile tokens enter the encoder.Source: C. Cross-modal Visuo–Tactile Fusion · p. 4
  • Cross-modal fusion occurs at the highest semantic stage, where aggregated geometric context provides a stable spatial reference for sparse tactile cues.The final-stage geometric and tactile tokens and their positional embeddings are concatenated and processed by a cross-modal Transformer.Source: C. Cross-modal Visuo–Tactile Fusion · p. 4
  • A block-wise attention mask preserves geometric attention, suppresses interactions involving invalid geometric tokens, and permits attention among visible tactile tokens.Source: C. Cross-modal Visuo–Tactile Fusion · p. 4

D. Hierarchical Decoder and Reconstruction

The hierarchical decoder reconstructs masked geometric and tactile observations from encoded visible tokens. Geometry is recovered progressively at finer scales, while tactile predictions are produced by a lightweight reconstruction head.

D. Hierarchical Decoder and Reconstruction

  • The decoder inserts mask tokens and combines them with visible encoded tokens, then uses global self-attention to reconstruct masked observations.It starts from the coarsest geometric scale and uses S −1 lightweight Transformer stages.Source: D. Hierarchical Decoder and… · p. 4
  • Tactile features are mapped to masked tactile predictions after the first decoder stage, and tactile loss is computed only on masked tokens.The overall pre-training objective balances geometric and tactile reconstruction losses to encourage complementary representations.Source: D. Hierarchical Decoder and… · pp. 4–5
  • Geometric decoding propagates coarse features to finer centers and combines them with encoder features through skip connections.A lightweight reconstruction head maps decoded representations to masked local point patches, trained with symmetric Chamfer Distance.Source: D. Hierarchical Decoder and… · p. 5

IV. EXPERIMENTS

The experiments assess representation effectiveness, generalization, and real-world transfer across downstream dexterous manipulation tasks. The evaluation includes baseline comparisons, pre-training ablations, and transfer to a Shadow Dexterous Hand with previously unseen objects.

IV. EXPERIMENTS

  • The evaluation covers representation effectiveness, generalization, and real-world transfer on downstream dexterous manipulation tasks.The experimental setup is described before the metrics and baselines.Source: §IV · p. 5
  • Quantitative comparisons with representative visuo–tactile representation learning methods are followed by ablations of the proposed pre-training components.The comparisons and ablations are assigned to Sections IV-C and IV-D, respectively.Source: §IV · p. 5
  • The final evaluation tests sim-to-real transfer on a Shadow Dexterous Hand using previously unseen objects.This is presented as the real-world transfer evaluation in Section IV-E.Source: §IV · p. 5

A. Experimental Setup

OccluDex is pretrained using synchronized egocentric human visuo–tactile demonstrations and evaluated in simulation with a Shadow Dexterous Hand. The setup covers two manipulation tasks with substantial hand-induced self-occlusion.

A. Experimental Setup

  • Three human subjects contributed 1,690 sequences and 409,035 synchronized visuo–tactile frames across six representative manipulation tasks.The collection paired head-mounted RGB-D observations with contact responses from a 19-channel WiseGlove, synchronized at 30 Hz.Source: A. Experimental Setup · p. 5
  • Downstream reinforcement learning uses a simulated Shadow Dexterous Hand in Isaac Gym for Faucet Rotation and Tabletop Object Reorientation.Both tasks involve substantial hand-induced self-occlusion during manipulation and operate at 60 Hz.Source: A. Experimental Setup · p. 5
  • At each control step, the policy receives proprioceptive state, binary tactile observations, and a partial point cloud.The inputs are 48-D, 19-D, and 4,096 points, respectively; the egocentric camera resolution is 424 × 240.Source: A. Experimental Setup · p. 5

B. Metrics and Baselines

OccluDex achieves the highest reported average success across the two manipulation tasks and leads on both seen and unseen evaluations. Policies are evaluated over repeated episodes, with comparisons against visuo–tactile and proprioception-only baselines.

B. Metrics and Baselines

  • Policies are evaluated over 100 episodes per random seed in seen and unseen settings, with mean and standard deviation reported across three seeds.The compared methods include VTDexManip, VTT-3D, and a proprioception-only Base baseline, using the same PPO training protocol and evaluation procedure.Source: B. Metrics and Baselines · pp. 5–6

C. Comparison on Downstream Dexterous Manipulation

  • Training curves report success rate in the top row and mean reward per step in the bottom row, with solid lines showing three-seed means and shading showing standard deviation.The accompanying prose says the training dynamics reflect facilitated downstream policy optimization on both tasks.Source: Figure 4 · p. 6, C. Comparison on Downstream Dexterous… · p. 6
  • On Faucet Rotation, OccluDex achieves 99.9% success on seen instances and 86.6% on unseen scales, outperforming all baselines in both settings.The reported average across seen and unseen evaluations is 93.2% success.Source: C. Comparison on Downstream Dexterous… · p. 6
  • On Tabletop Object Reorientation, OccluDex reaches 75.1% success on seen objects and 83.3% on held-out objects, exceeding VTDexManip at 76.9%, VTT-3D at 72.1%, and Base at 63.9%.The passage identifies the held-out objects as the unseen-object evaluation.Source: C. Comparison on Downstream Dexterous… · p. 6
  • 86.2% average success across the two downstream tasks compares with 75.8% for VTDexManip, 73.0% for Base, and 65.6% for VTT-3D.The passage reports consistent gains across distinct tasks and evaluation settings.Source: C. Comparison on Downstream Dexterous… · p. 6

D. Ablation Study

OccluDex’s ablations show that tactile feedback and 3D geometry both contribute to manipulation performance, while frozen pretraining reduces downstream training cost. Real-world trials demonstrate closed-loop success on unseen physical objects in both tasks.

D. Ablation Study

  • Removing tactile input caused the largest degradation on both tasks, while adding point-cloud observations improved the tactile-only variant.Point-cloud-only success dropped by 45.7% on seen and 48.0% on unseen Faucet Rotation, and by 15.6% on seen and 17.3% on held-out tabletop objects; adding point clouds to tactile-only improved success by 24.3% and 6.7% for seen and unseen Faucet Rotation, and 9.6% and 4.7% for seen and held-out tabletop objects.Source: D. Ablation Study · p. 6
  • Frozen pretraining reduces downstream reinforcement-learning cost and outperforms the feasible simplified from-scratch reference; full-scale joint training was impractical under the hardware constraints.The feasible reference removed the hierarchical design and used a single-stage fusion architecture.Source: D. Ablation Study · p. 6

E. Real-World Experiments

  • The physical platform combines 19 tactile sensors, a 4,096-point cloud, and 24 joint positions and velocities as policy inputs.The policy runs closed loop at 10 Hz and outputs 20 normalized joint-position targets.Source: E. Real-World Experiments · pp. 6–7
  • OccluDex succeeded in 17/20 Faucet Rotation trials and 14/20 Tabletop Object Reorientation trials on unseen physical objects, without real-world finetuning.These correspond to 85.0% and 70.0% success rates, respectively; each task used two objects and 10 trials per object.Source: E. Real-World Experiments · p. 7

V. CONCLUSION

OccluDex combines global 3D geometry and local tactile contact cues through masked pretraining, then transfers the encoder to PPO-based manipulation. It achieves the highest average success rate across two tasks and transfers zero-shot to unseen physical objects.

V. CONCLUSION

  • OccluDex integrates global 3D geometry and local tactile contact cues through masked pretraining, transferring the learned encoder to downstream PPO-based manipulation.The framework targets dexterous manipulation under severe self-occlusion.Source: §V · p. 7
  • 86.2% average success across Faucet Rotation and Tabletop Object Reorientation was the highest reported, while zero-shot physical-object success was 85.0% and 70.0%, respectively.The physical results were on previously unseen objects.Source: §V · p. 8
  • Future work will explore denser tactile sensing and finer-grained manipulation requiring more precise contact perception and control.The paper specifically mentions lateral fingertip contacts and scalable fusion of dense tactile observations with 3D geometry.Source: §V · p. 8
Loading 2609.39017v1…