DexNinja: Learning Robust Dexterous Cutting Policy with a Real-to-Sim-to-Real Data Engine

2.2 Dexterous manipulation and Teleoperation

• Dexterous hands offer stabilization and force modulation, while tactile cues support visual–tactile reasoning during contact-rich cutting.Prior work spans grasping, task-specific skills, video learning, interaction representations, and tactile sensing [54] [56] [9] [59] [60] [61] [62] [63].Source: §1 · p. 2, §2.2 · p. 3

1 Introduction

• DexNinja reconstructs real instances and generates category-level physical variations to create dense, realistic training data for topology-changing manipulation.The framework targets sim-to-real transfer and generalization to unseen objects with shifts in physical parameters.Source: §1 · p. 2

2.1 Data generation and differentiable simulation

• For cutting, trajectory or pose generation alone differs from densifying mass, geometry, and friction around reconstructed objects in a differentiable loop.The method also emphasizes topology consistency in its simulator-based cutting setup.Source: §2.1 · p. 3

3 Problem Formulation

• The task uses one arm and dexterous hand to hold a knife while the other hand interacts with deformable food.The observation includes scene point clouds, fingertip tactile measurements, and robot proprioception.Source: §3 · p. 3
• Food instances are represented with interpretable mass, friction, geometry, and elasticity parameters, while the knife is modeled separately as a known rigid tool.The elasticity representation uses Young’s modulus and Poisson’s ratio; the simulator converts these to Lamé parameters.Source: §3 · pp. 3–4
• Control is split between a visual-geometry-conditioned kinematic policy and a tactile- and proprioception-feedback policy for contact-sensitive hand control.The kinematic component maintains desired blade–fruit geometry, while the feedback policy uses tactile and proprioceptive observations.Source: §3 · p. 4

Stage I

• Real demonstrations provide synchronized visual, tactile, and proprioceptive trajectories used to reconstruct food geometry and estimate its pose.Assets are aligned at metric scale, registered to an object-centric canonical template, and paired with a fixed end-effector–knife transform.Source: §4 · pp. 4–5

Stage II

• Simulation rollouts vary reconstructed food geometry and physical parameters, perturb knife motions, and record aligned multimodal observations and actions during deformable cutting.Inverse kinematics converts knife-path targets into feasible joint-space trajectories; the simulator captures contact evolution, material separation, and tactile feedback.Source: §4 · p. 5
• Additional in-category assets and physically consistent rollouts form an augmented dataset that combines simulated data with the real demonstration.Real-world execution filters trajectories for collision and feasibility, then applies targeted fruit-pose randomization to address residual sim-to-real differences.Source: §4 · p. 5

Stage III

• The ACT-style policy separates arm control for global approach and alignment from hand control for grasping and contact regulation.Effector-specific tactile token groups encode localized inputs for the corresponding hand or knife.Source: §4 · p. 5
• Action chunks adapt to manipulation stage: shorter horizons serve contact-sensitive motion, while longer ones serve coarse free-space motion.The real system uses nominal horizons of 64 for approach and grasp, 128 for knife positioning, and 16 for cutting with tactile feedback.Source: §4 · pp. 5–6, Figure 3 · p. 5

5 Continuation on a Shape Manifold for Data Augmentation

• The method samples nearby instances from a learned category shape family and transfers a successful nominal cutting motion to them by local continuation.A local shape manifold means small latent perturbations induce small geometry changes; the family is anchored by a reconstructed real instance.Source: §5 · p. 6, Data augmentation pipeline · p. 6

Data augmentation pipeline

• The continuation argument assumes the deployable-observation-to-action mapping is differentiable near a successful anchor, so small observation changes admit proportionally small action corrections.This local regularity is the stated basis for stable continuation; the shape generator is likewise locally regular around the nominal latent code.Source: Data augmentation pipeline · p. 6
• A local corrective policy is fitted around each demonstrated observation–action pair, with simulation testing whether candidate actions pass cutting and collision checks.Its observation distribution includes small state perturbations, physics randomization, and local shape variation; the correction stays close to the demonstrated action while compensating for deformation and partial separation.Source: Data augmentation pipeline · p. 7
• Augmented episodes are generated by rolling out local corrective policies and retaining successful outcomes.The category-conditioned prior anchors variation to real captures, and sampled rollouts must pass inverse-kinematics, collision, and cutting-success checks.Source: Data augmentation pipeline · p. 7
• Continuation concentrates augmented samples in regions that remain physically feasible under geometry variation and cutting-induced topology changes.This contrasts with naive perturbation, which does not provide the same feasibility focus.Source: Data augmentation pipeline · p. 7
• A single teleoperated episode covers only one instance, grasp, and approach, so simulation broadens that narrow slice with consistent rollouts over category-prior instances.Observations are restricted to deployable sensing modalities.Source: Data augmentation pipeline · p. 7

6.1 Task setup and evaluation protocol

• The protocol evaluates food-halving on four food types, with five real demonstrations and 50 simulated sequences per instance, and reports progress, success, and integrity.Evaluation uses 25 instances per category and randomizes food placement within a predefined reachable workspace.Source: §6.1 · pp. 7–8, §6 · p. 7, §6.2 · p. 8

6.2 Sim-to-real gap and the effect of mixing sim and real data

• 76% progress, 50% success, and 60% integrity for Sim+Real (Augmented) exceed Sim-only and Real-only on held-out real-world setups.The comparison aggregates 100 real trials across four foods and 25 instances per food.Source: Table 1 · p. 8, §6.2 · p. 8

6.3 Complex Motion and Heterogeneous Objects

• Lemon-cutting success falls from 8/10 single-cut trials to 7/10 two-cut and 3/10 multi-cut trials, as accumulated errors and harder contact dynamics challenge execution.The lemons test multi-cut behavior on objects with complex deformation and heterogeneous internal structure.Source: §6.3 · pp. 8–9, Figure 5 · p. 8

6.4 Component ablations and failure analysis

• Matched-budget ablations show trajectory refinement's removal causes the largest Progress and Success drops, while removing tactile observation most strongly affects Integrity.Reconstruction-only training reduces geometric coverage.Source: §6.4 · p. 9, §6.3 · p. 9
• Among 50 failed Sim+Real trials, grasp/contact control and cutting-physics mismatch are the main failure sources; the asset funnel accepts 1,200 of 1,670 proposals.Cutting-physics mismatch means stable registration, grasp, and trajectory tracking but a material response not captured by simulation.Source: §6.4 · p. 9, Table 3 · p. 9, Table 4 · p. 9

7 Conclusion

• DexNinja densifies training from teleoperated demonstrations through reconstructed assets, physically interpretable parameter estimates, and physics-aware randomization and instance generation.Its differentiable simulator couples robot dynamics, tactile contact, and deformable cutting for rollouts and gradient-based fitting.Source: §7 · p. 9
• The evaluated policies improve robustness to unseen food shapes and sizes when real demonstrations are combined with DexNinja's augmented pipeline.The policy uses separated arm/hand control and adaptive action chunking.Source: §7 · p. 9

8 Limitations

• Deployment flexibility may be limited by the mounted camera, and current category-specific policies require new assets and retraining or fine-tuning for new categories.The authors identify wrist-camera deployment as future work.Source: §8 · p. 9

1.1 Stage I: Teleoperation and Object Frame

• A teleoperated cutting trial records visual observations, joint-angle actions, food pose, and geometry to extract a transferable knife trajectory in the food frame.The relative knife-object motion is intended to transfer across reconstructed food instances.Source: §1.1 · p. 16

1.2 Stage II: Differentiable Cutting Simulator Details

• DexNinja reconstructs task assets, varies strawberry shape and physical parameters, perturbs object-centric knife trajectories, and maps them through grasp geometry and inverse kinematics into simulated rollouts.The object-centric template is refined against metric-scale reconstructed geometry; sampled trajectories vary start offsets, approach angles, and small pose noise.Source: §1.2 · p. 16
• The differentiable simulator optimizes knife trajectories against ground interaction and cutting progress, with a smooth collision penalty for penetration beyond tolerance m.The signed-distance convention is positive outside the ground, zero on its surface, and negative inside; m is a numerical penetration tolerance.Source: §1.2 · pp. 16–17

1.3 Tactile Simulation During Cutting

• Tactile adaptation data come from simulated grasp-and-cut sequences, where tactile cues of incipient slip prompt small corrective finger motions that maintain contact while permitting deformation.The simulator estimates fingertip contact from particle penetration and relative tangential velocity, then spatially bins forces into tactile observations.Source: §1.3 · p. 17

1.4 Scene level and hardware registration

• A knife-based registration bridge passes local cutting contact points, forces, and moments to the scene-level robot simulator, avoiding a full reimplementation of deformable cutting there.The bridge is modular across objects, tools, and robot embodiments, though it adds per-step data transfer.Source: §1.4 · p. 18

2 Training Details

• Separate kinematic and tactile action heads address mismatched arm and hand action scales, which can otherwise destabilize learning or leave the hand under-actuated.After contact, pose bias, friction variation, and compliance errors can lead to slip, over-compression, or failed cutting.Source: §2 · pp. 18–19

2.1 Stage-conditioned action chunking

• Stage-conditioned action chunks use longer horizons for free-space motion and shorter horizons for tactile-guided cutting, matching coarse movement with smoothing and contact control with frequent replanning.The nominal horizons are 64 for approach, 128 for knife positioning, and 16 for cutting.Source: §2.1 · p. 19

2.2 Group-wise action normalization and execution

• Group- and stage-specific action statistics and step sizes keep arm- and hand-scale actions from sharing one global normalization range, while command projection enforces robot safety constraints.The kinematic step size is smaller than the tactile step size for fine arm/tool corrections near contact.Source: §2.2 · p. 19

2.3 Chunked behavior-cloning objective

• The chunked behavior-cloning objective weights kinematic actions more during approach and knife positioning, then emphasizes tactile actions during cutting while retaining low-rate kinematic correction.Executed joint increments are also regularized.Source: §2.3 · p. 20

2.4 Sim2Real deployment

• Deployment synchronizes visual, tactile, and proprioceptive observations and uses geometric proximity and tactile onset to trigger stage transitions; sustained contact loss returns control to knife positioning.Commands pass through the safe-command projection during execution.Source: §2.4 · p. 20

3.1 Two-Embodiment Setup Verification

• The same object-centric pipeline and kinematic–tactile action decomposition are used on both embodiments, changing only calibration, kinematics, inverse kinematics, joint limits, and action scaling.The second setup replaces the primary DR3/DR5 arms and LinkerHand L10 hands with Franka arms and LinkerBot L20 hands.Source: §3.1 · p. 20, Figure 1 · p. 21

3.2 Embodiment-Specific Control and Safety Filters

• The object-centric blade–object trajectory transfers to another robot by re-estimating the end-effector–knife transform and resolving inverse kinematics for that embodiment.Safety filters discard or repair knife-pose targets that violate collision constraints, exceed joint limits, or approach kinematic singularities.Source: §3.1 · p. 21, §3.2 · p. 22

• For each real capture, the pipeline identifies the food category and extracts structured shape, surface, color, and ripeness attributes to condition generation.These attributes are intended to support consistency between simulation and real domains.Source: §4 · p. 22
• A constrained description guides inpainting toward a centered, single-object image with transparent background while preserving the captured attributes.A vision-language judge rejects images with issues such as multiple objects, nontransparent backgrounds, attribute mismatch, or artifacts.Source: §4 · p. 22
• Retained images are lifted to a 3D Gaussian Splatting asset or mesh, rendered from multiple views, and checked for consistency before admission.The process includes semantic and multi-view checks for identity, geometry, and appearance stability.Source: §4 · p. 22
• Curated assets pair a transparent single-object image with a validated 3D asset and structured attributes, forming a category pool for simulation rollouts and continuation around real anchors.Simulation admission additionally requires metric-scale, geometry-validity, simulator-compatibility, and stable-initialization checks.Source: §4 · pp. 22–23

5.1 Comparison to baseline

• 57.5% success for DexNinja versus 30.0% for ROBONINJA across 40 real-world trials, a +27.5 percentage-point improvement.DexNinja improved on Strawberry, Potato, and Banana, while ROBONINJA did better on Apple.Source: §5.1 · pp. 23–24, Table 2 · p. 24
• The ROBONINJA comparison is a protocol-matched reproduction using an approximate contact/force signal derived from tactile readings.The original force-sensor model and calibration were not fully recoverable, so its noise, bandwidth, and calibration may differ.Source: §5.1 · p. 24

5.2 Simulation scaling and generation cost

• More simulated episodes generally improved real-world success, but gains diminished as common interaction modes became covered.A single-cut episode took about 10 minutes and 20 GB of GPU memory; an M-cut episode required approximately (10 + 10M) GB and M/6 GPU-hours.Source: §5.2 · pp. 24–25, Table 3 · p. 24

5.3 System-level ablations

• The full system performed best with separate kinematic and tactile action heads plus adaptive, stage-conditioned action chunking.A shared head creates a scale mismatch between arm and finger motion, while fixed chunks poorly match approach and contact-rich cutting.Source: §5.3 · p. 25, Table 4 · p. 25

5.4 Tactile regulation versus forceful grasping

• Tactile control regulates pressure and shear within the narrow regime between slipping and over-compression, rather than simply increasing grasp force.High-force guava contact produced broader, stronger tactile responses and visible compression than nominal contact.Source: §5.4 · p. 25, Figure 3 · p. 26

5.5 Summary of additional findings

• The authors conclude that simulation gains can saturate, so future data generation should prioritize interaction-mode diversity over episode count alone.The system-level ablations also support separate kinematic and tactile heads with adaptive action chunking for stable real-world execution.Source: §5.5 · pp. 25–26, §5.2 · p. 25

Cross-embodiment deployment

• Figure 4 presents representative executions across food categories using two embodiment settings.Background colors identify banana, strawberry, apple, and potato.Source: §6 · p. 26, Figure 4 · p. 27

Multi-cut experiments

• Three-cut trials required repeated fruit adjustment and re-alignment; failures often occurred when accumulated pose errors or contact disturbances prevented repositioning.Two-cut execution remained similar to the single-cut procedure.Source: §6 · pp. 26, 28, Figure 5 · p. 28

Rollout phases

• Real-world rollouts progress from a conservative-clearance approach to small contact corrections and then smooth, monotonic slicing.Contact initiation typically uses millimeter-scale translations and a slight pitch adjustment.Source: §6 · p. 26

Category-specific behavior

• Observed cutting behavior varied by food: bananas used shallower, shorter strokes, strawberries gentler contact, and potatoes slower progression with sustained blade orientation.These qualitative patterns correspond to differences in compliance, surface delicacy, and cutting resistance.Source: §6 · p. 27