PaperlayerView exact revision on arXiv

Source-linked AI summary

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

Jusuk Lee, Sungha Kim, Yeonsoo Park, Jonguk Cheon, Yoonkyo Jung, Yongjun You, H. Jin Kim, Jia-Bin Huang, Furong Huang, Youngseok Jang, Seungjae Lee

arXiv:2610.12470v1cs.ROcs.CV

TL;DR

Dexterous policies learned from a single human video often inherit demonstrated motions, while unguided RL struggles with exploration in complex, multi-stage tasks. Dex-One2Many uses stage-wise scene graphs to guide RL from simulation to real-world deployment. Across five real-world tasks, it achieves 75% mean success on unseen configurations, exceeding the next-best baseline by 71 percentage points.

  • Problem

    Single-video methods often tie dexterous policies to demonstrated motions, while unguided RL struggles to explore high-dimensional, contact-rich tasks.

    Source: §1 · p. 2

  • Method

    Dex-One2Many abstracts a single human video into stage-wise scene graphs that guide RL through diverse reset states and predicate-derived rewards.

    Source: §1 · p. 2

  • Results

    Across five real-world tasks, Dex-One2Many achieves 75% mean success on unseen configurations, exceeding the next-best baseline by 71 percentage points.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results suggest that extracting task structure while allowing RL to discover how to act provides a route to generalizing beyond the demonstrated execution.

    Source: §5 · p. 11

    The real-to-sim pipeline reconstructs only rigid objects; articulated objects require manually built assets because a single video may not reveal their underlying articulation structure.

    Source: §5 · p. 11

Abstract

from arXiv · show

While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.

1. Introduction

Single-video methods often tie dexterous policies to demonstrated motions, while unguided RL struggles with high-dimensional exploration. Dex-One2Many uses stage-wise scene graphs to guide RL and reports strong real-world generalization to unseen configurations.

1. Introduction

  • Single-video methods often tie policies to demonstrated motions, limiting generalization to unseen initial poses, goal poses, and grasps.Unguided RL, in turn, struggles to discover multi-finger, contact-rich interactions in high-dimensional state spaces.Source: §1 · p. 2
  • Stage-Wise Scene Graph Abstraction preserves task-defining relations and their order while leaving object poses, goal poses, and grasps free to vary.Each stage represents a set of states that preserve the task rather than one recorded state.Source: §1 · p. 2
  • Scene-Graph-Grounded RL samples diverse reset states for each stage and uses predicate-derived dense rewards, so policies learn task structure rather than the recorded execution.Starting episodes at every stage lets the policy practice later stages before mastering earlier ones.Source: §1 · p. 2
  • Dex-One2Many transfers zero-shot from simulation to a real multi-fingered hand on configurations with initial and goal poses absent from the video.The reported real-world success rate is 60–85%, while baselines remain at or below 15%.Source: §1 · pp. 2–3
  • 75% mean success on unseen configurations across five real-world tasks exceeds the next-best baseline by 71 percentage points.The policies are learned from a single human video.Source: §1 · p. 3

2. Related Work

Prior single-video approaches commonly preserve execution details, while relational task specifications offer a more abstract representation. Dex-One2Many uses stage-wise scene graphs to constrain training states and guide learning without inheriting the demonstrated motion.

Task Structure Extraction from Human Videos

Learning Dexterous Manipulation from a Single Human Video

  • Prior dexterous-video methods use trajectories, retargeted motions, or contacts as execution guidance, which makes exploration tractable but limits generalization largely to local variations around the demonstrated configuration (Lum et al., 2025, Han et al., 2026, Zhu et al., 2026).Demonstration-based resets likewise inherit the demonstrated execution; OmniReset (Yin et al., 2026) and Video2Policy (Ye et al., 2025) reduce this dependence but rely on user-specified task information or video-derived object states and have been demonstrated with parallel-jaw grippers.Source: §2 · pp. 3–4
  • Dex-One2Many grounds resets, observations, and rewards in a stage-wise scene graph derived from the video, deriving task structure without imitating execution.Its overview also shows zero-shot transfer to unseen initial poses, goal poses, and grasps on real hardware.Source: §2 · p. 4, Figure 2 · p. 4

3. Method

Dex-One2Many abstracts a human video into ordered scene-graph stages, then grounds policy observations, rewards, and reset states in those graphs. The policy is trained in simulation and transferred zero-shot using perception and control measures designed for deployment.

3. Method

  • The method formulates policy learning as an MDP grounded in a scene graph, with observations, rewards, and initial states constructed from that graph before zero-shot transfer to a real robot.The method overview describes a pipeline from a single demonstration through graph-derived resets and policy learning to real-robot transfer.Source: §3 · pp. 4–5

3.1. Task Abstraction as a Stage-wise Scene Graph

  • Stage-Wise Scene Graph Abstraction preserves the required relations and their order while allowing object poses, goal poses, grasps, and trajectories to vary.Nodes identify the hand, task-relevant objects, and functional parts; edges encode relations characterizing each stage.Source: §3.1 · p. 5
  • A VLM identifies entities and predicate relations over time, introducing a new stage when relations change, such as when pre_grasp becomes grasp.A predefined predicate vocabulary includes relations that must hold, such as grasp, contact, and inside, and proximity relations indicating what should be established next.Source: §3.1 · p. 5

3.2. Scene-graph-grounded Reinforcement Learning

  • Graph nodes and edges supply reusable policy observations, while predicate rewards use geometric distances or physical measures and are assembled as each stage adds a predicate.For grasp, the reward uses a force-closure measure augmented with a grasp-quality measure for stability.Source: §3.2 · pp. 5–6
  • Each stage graph constrains reset-state sampling; episodes start uniformly across stages, and diverse synthesized multi-finger grasps are retained only after collision and stability checks.Final-stage states are perturbed from goal-satisfying states to provide near-goal learning signal, allowing practice of later interactions without first reaching them from the task start.Source: §3.2 · pp. 6–7

3.3. Zero-shot Sim-to-Real Transfer

  • The policy is trained with an asymmetric actor–critic and randomized physical properties, while noisy and delayed observations help address perception differences for zero-shot transfer.At deployment, object poses are estimated from two RGB-D cameras; a QP controller maps desired palm and fingertip velocities to joint velocities while enforcing kinematic and safety constraints.Source: §3.3 · p. 7

4. Experiments

Experiments test generalization to unseen initial and goal configurations, applicability across dexterous embodiments, and critical design choices across five direct-manipulation and tool-use tasks.

4.1. Experimental Setup

  • Five tasks span direct manipulation and tool use; real-world tests use a UR3 with a 20-DoF Wuji 1 hand and compare against Do as I Do and CHORD.The tasks are Doll, Can, Stamp, Hammer, and Sweep.Source: §4.1 · pp. 7–8

4.2. Does Dex-One2Many Generalize Beyond the Human Video?

  • 85–100% simulation and 60–85% real-robot success for Dex-One2Many on Unseen configurations, versus at most 15% for baselines in both settings.On Can with the goal fixed, Dex-One2Many also succeeds across a broader range of initial positions than baselines. Its scene graphs generate diverse, task-consistent training states beyond the demonstrated execution.Source: Figure 4 · p. 8, §4.2 · pp. 8–9, Figure 5 · p. 9

4.3. Does Dex-One2Many Apply Across Diverse Dexterous Embodiments?

  • 94.6% to 96.8% mean success across four tested embodiments; learned grasps differ by embodiment and adapt to object configuration.Table 1 reports task-wise success rates and means across five tasks; Figure 6 illustrates grasp strategies on Doll.Source: §4.3 · p. 9, Table 1 · p. 9, Figure 6 · p. 10

4.4. Which Design Choices Are Critical to Dex-One2Many?

  • Removing stage-wise resets drops success from over 90% to about 16% after 2k updates; removing predicate rewards leaves success at 0%; omitting qcmd yields about 48%.The reported ablations are on Sweep, measured on full-task episodes from empty-hand starts.Source: §4.4 · pp. 10–11
  • 49,152 environments reach over 90% success, while 24,576 or 12,288 remain below 25% within the same number of PPO updates.The paper reports that parallel experience collection benefits learning under its diverse reset distribution.Source: §4.4 · p. 11

5. Conclusion

Dex-One2Many learns dexterous manipulation from a single human video by preserving task structure while letting policies discover how to act. The conclusion reports generalization to unseen configurations and zero-shot transfer, while noting limitations in pose-dependent deployment and articulated-object reconstruction.

5. Conclusion

  • Dex-One2Many preserves task structure while leaving poses, grasps, and motion paths free to vary, supporting generalization beyond the video and zero-shot transfer to a real multi-fingered hand.Across five manipulation tasks, the approach uses diverse stage-wise reset states and predicate-derived rewards.Source: §5 · p. 11
  • Real deployment depends on estimated 6D object poses, and pose noise or simulated-real geometry differences can degrade contact-rich performance.The pipeline reconstructs only rigid objects; articulated-object reconstruction would require manually built assets, and a single video may not reveal joint structure.Source: §5 · p. 11

A. Additional Related Works

  • Unlike demonstration-seeded RL methods that specify actions, progress targets, or reward programs, Dex-One2Many uses predicate-level relation changes to define stages and diverse intermediate reset states.These reset states allow episodes to start from intermediate task stages, reducing exploration burden.Source: A. Additional Related Works · p. 19
  • Where prior grasp methods target robust grasping across objects, Dex-One2Many uses grasp synthesis for diverse resets and trains RL to acquire grasps and complete later manipulation stages.The cited methods include TaskDexGrasp and Dexterous Functional Grasping (Chen et al., 2024, Agarwal et al., 2023).Source: A. Additional Related Works · p. 19
  • SimToolReal tracks a human-video 6D tool trajectory at execution, unlike this task setting; Play2Perfect and ADEPT study pretraining for downstream manipulation.The text says these settings are not directly comparable and that the pretraining approaches complement Dex-One2Many.Source: A. Additional Related Works · p. 19
  • Dex-One2Many augments a force-closure reward with a weakest-direction grasp-quality measure to encourage stable grasps within a multi-stage manipulation task.The passage distinguishes this use from grasp-quality rewards for in-hand manipulation.Source: A. Additional Related Works · p. 19

B. Task Abstraction as a Stage-wise Scene Graph

The task-abstraction section describes a stage-wise scene-graph representation and its predicate vocabulary. The vocabulary covers tabletop relations, with approach forms for geometric predicates.

B. Task Abstraction as a Stage-wise Scene Graph

  • The section describes the predicate vocabulary, a single VLM query that produces stage-wise scene graphs, object-asset reconstruction, physical-property estimation, and functional-region localization.The pretrained models used in these steps are summarized in Table 2.Source: B. Task Abstraction as a Stage-wise… · p. 20, Table 2 · p. 20
  • The notation defines d(p, S) as the Euclidean distance from point p to set S, with DA and BA denoting entity A’s top support disk and interior box.For relation c, ϵc is its satisfaction tolerance and φc is the outer sampling bound of its pre-relation.Source: B. Task Abstraction as a Stage-wise… · p. 20
  • For pre_grasp, dtip is the fingertip-to-grasp-region-center distance, ℓg and ug are its sampling bounds, GA is the verified grasp set, and hB is the hand configuration.Ptip B denotes fingertip positions.Source: B. Task Abstraction as a Stage-wise… · p. 20

B.1. Predicate Vocabulary

  • Table 3’s vocabulary covers common tabletop-manipulation relations, and each geometric predicate has a pre- form for approaching the relation before it is established.Paired pre-relation and terminal predicates share distance metrics and differ in constraints, except for pre_grasp/grasp.Source: §B.1 · p. 20, Table 3 · p. 20

B.2. VLM Query

The VLM-query section describes extracting task structure from each demonstration with a shared prompt and predicate vocabulary. Its output records objects, roles, task stages, and relations, with a reported query cost under $0.05 per task.

B.2. VLM Query

  • Each task uses one human demonstration video, with frames ordered over time, as the input to a Gemini 3.1 Pro query using the entire video.Videos are recorded at 640 × 480 and sent inline when under 20 MB, or through the file-upload API when larger.Source: §B.2 · p. 21, Figure 8 · p. 21
  • A shared prompt and fixed predicate vocabulary are used across tasks, with the video as the only task-specific input.The VLM returns JSON following a fixed response schema.Source: §B.2 · pp. 21, 23
  • A query takes 12–21 seconds and costs less than $0.05, while extracting all five task graphs takes under 90 seconds and $0.20 in total.Table 6 is the per-task query-cost table.Source: §B.2 · p. 21, Table 6 · p. 26
  • The five-task suite includes pick-and-place, stamping, hammering, and sweeping, with the predicate chains and object roles summarized in Table 4.The supplied examples include placing a Pikachu in a pot, a can in a bucket, and a stamp on paper.Source: Table 4 · p. 22, §B.2 · p. 22
  • The output schema identifies manipulable objects and assigns roles such as held, site, acted_on, or fixture, while recording task-relevant parts and tool-use direction.The prompt distinguishes a site that receives an object during manipulation from a fixture already holding one at the start.Source: Table 5 · p. 22, §B.2 · p. 23
  • The scene graph encodes an initial state, goal condition, and temporally ordered stages using predicates, with approach phases derived from established relations.For example, the Sweep output sequences grasping the brush, contacting the toy, and placing the toy inside the dustpan.Source: §B.2 · pp. 24–25
  • The prompt distinguishes an object placed inside an open container from one pressed into another object to form a rigid assembly.The former uses inside; the latter uses seat.Source: §B.2 · p. 25

B.3. Asset Preparation and Functional Regions

Asset preparation reconstructs task-relevant objects from the demonstration’s first frame, then prepares meshes for simulation and grasp synthesis.

B.3. Asset Preparation and Functional Regions

  • Task-relevant objects are reconstructed from the first demonstration frame, while the full video is used only for the VLM query.A VLM visual phrase prompts SAM 3.1 to return an instance mask; RGB, the mask, and the depth-derived metric point map are then used by SAM 3D Objects to reconstruct a mesh.Source: §B.3 · p. 26
  • Some objects use available CAD meshes instead of reconstructed meshes.The meshes are post-processed into watertight simulation meshes and convex pieces for grasp synthesis.Source: §B.3 · p. 26
  • Convex pieces serve as collision geometry for grasp synthesis, while watertight meshes serve as collision meshes in Isaac Sim.Table 7 summarizes mesh post-processing.Source: §B.3 · p. 26, Table 7 · p. 26

B.4. Physical Properties and Functional Region Localization

The appendix describes how Dex-One2Many builds scene-graph-grounded observations and rewards, then uses stage graphs to generate reset states. Physical properties and functional regions supply simulation parameters and task-relevant geometry.

B.4. Physical Properties and Functional Region Localization

  • Object properties are estimated separately from VLM semantics, while mesh-based localization identifies grasp regions and tool working surfaces for grasp synthesis and rewards.During RL, mass and static friction are sampled within estimated ranges; dynamic friction is set to 0.9 times static friction.Source: §B.4 · pp. 26–27

C.1. Observations

  • Policy observations concatenate role-specific hand, object, and functional-part features with predicate-specific relative 6D poses between connected entities.Direct manipulation includes the manipulated and receptive objects; tool use treats the held object as the tool and adds the acted-on object.Source: §C.1 · p. 28, Figure 9 · p. 28

C.2. Rewards

  • Grasp rewards assess force closure in 6D wrench space and favor support along the weakest wrench direction, while spatial-predicate rewards pay only for new distance progress.After progress stops, the progress reward becomes zero; contact rewards also include alignment with the desired interaction direction.Source: §C.2 · pp. 28–29

C.3. Reset Generation

  • Stage graphs generate diverse reset states by sampling entity placements that satisfy their predicates, instantiating grasps, and retaining states verified as usable and physically valid.The resulting reset distribution contains 10,000 verified states per stage.Source: §C.3 · p. 30, Figure 10 · p. 30

C.3.1. Predicate Samplers and Placement Order

Predicate samplers construct stage states by placing entities in dependency order, while grasp synthesis supplies candidate hand–object configurations for stages that require grasping.

C.3.1. Predicate Samplers and Placement Order

  • Predicate samplers place each source node in a region defined by its relation to an already placed target, with the anchor placed first and unconstrained nodes sampled last.For a final stage, the sampler starts from a goal-satisfying state and perturbs it into a near-goal state.Source: §C.3.1 · pp. 30–31

C.3.2. Grasp Synthesis

  • Dexonomy generates diverse grasps across 33 types, restricted to the object’s grasp region when available, and stores finger-closing commands for reset initialization.Pre-grasp states interpolate finger joints toward open and retract the hand 1–5 cm along its approach axis.Source: §C.3.2 · p. 31

C.3.3. Transition Feasibility and Physical Validity

Reset candidates are checked for whole-task feasibility and physical validity, and later stages constrain how earlier states are generated. A deployment-time QP can add hardware-specific constraints without retraining the task policy.

C.3.3. Transition Feasibility and Physical Validity

  • Grasps are ranked by pass rate across all grasp-containing stages, so the retained grasp set favors candidates that remain compatible with the whole task rather than stability at pickup alone.The top four grasp types form the grasp set; candidates unable to attain the goal relation are excluded.Source: §C.3.3 · p. 31
  • Stages are generated backward from the goal, carrying verified anchor poses and grasps into the preceding stage so each earlier reset is built around a feasible later state.Candidate states also undergo collision and reset-stability checks using maximum mass and minimum friction from the randomized range.Source: §C.3.3 · pp. 31–32, Figure 11 · p. 32

C.4. Hierarchical decomposition with a QP low level.

  • A deployment-time hand–table clearance constraint in the QP prevents the Wuji 1 coupler from disengaging during real execution without additional policy training.The task-space policy remains unchanged while the QP modifies feasible joint motion.Source: §C.4 · pp. 33–34, Figure 12 · p. 33

C.5. Simulation Training Details

Policies are trained in simulation with shared optimization settings and randomized physical properties, observation noise, and delays. Real-world perception estimates task-object poses from two RGB-D camera views and fuses them according to depth-based reliability.

C.5. Simulation Training Details

  • Simulation runs at 120 Hz while policies act at 10 Hz, and PPO uses an asymmetric actor–critic with a recurrent actor and privileged-state critic.The same optimization and network configuration is used across tasks and embodiments.Source: §C.5 · p. 34
  • Training varies physical properties and adds actor-observation noise plus observation and action delays to address sim-to-real differences.Source: §C.5 · p. 34, Table 10 · p. 35

D. Real-World Perception

  • Two RGB-D camera pose estimates are fused using how well each rendered mesh explains its depth image, reducing the weight of occluded or truncated views.When both cameras are excluded, the pose is extrapolated with a constant-velocity model for up to 0.5 seconds and then held.Source: D. Real-World Perception · pp. 35–36

E. Additional Experimental Results

The additional experiments examine initial-position generalization, grasp diversity, and successful rollouts across dexterous embodiments. They report broader success over unseen initial positions and grasp strategies that vary with embodiment and object configuration.

E.1. Generalization to Unseen Initial Object Positions

  • With goals fixed, Dex-One2Many succeeds over a substantially broader range of initial object positions than baselines across five tasks.Figures 13–17 visualize initial-position generalization on Doll, Can, Stamp, Hammer, and Sweep.Source: §E.1 · p. 36, Figure 13 · p. 36, Figure 14 · p. 37

E.2. Diverse Grasp Strategies Across Embodiments and Configurations

  • Learned grasp types vary across embodiments and object configurations rather than remaining fixed to the human demonstration.The reset pipeline samples diverse multi-finger grasps from a 33-type taxonomy; representative strategies are shown for Doll, Can, and Stamp.Source: §E.2 · p. 41, Figure 18 · p. 41, Figure 19 · p. 42

E.3. Simulation Rollouts Across Diverse Dexterous Embodiments

  • Across four dexterous embodiments, separately trained policies show successful rollouts for all five tasks from distinct task configurations.Each task-and-embodiment example includes three rollouts; rows represent configurations and columns show execution over time.Source: §E.3 · p. 44, Figure 21 · p. 45, Figure 22 · p. 46

Related papers

Loading 2610.12470v1…