PaperlayerView exact revision on arXiv

Source-linked AI summary

DexAgent: An Agentic Human2Sim2Robot Framework for Dexterous Manipulation with Self-Evolving Tool Library

Youhui Wang, Yunzhu Li, Li Fei-Fei, Jiajun Wu, Huang Huang

arXiv:2609.35318v1cs.RO

TL;DR

Existing Human2Sim2Robot pipelines struggle with diverse object properties and interactions. DexAgent adapts a verification-guided, four-stage workflow to generate robot-training data from one human video. Across eleven real-world tasks, its trained policies average 63.6% success, approximately 3.5× the success rate of SPIDER.

  • Problem

    Predefined Human2Sim2Robot procedures struggle with diverse object properties and interactions, especially for articulated and deformable objects.

    Source: §I · p. 1

  • Method

    DexAgent adapts a four-stage video-to-robot-data workflow by selecting or developing task-specific skills and refining stage outputs with verifiers.

    Source: §I · pp. 1–2

  • Results

    Across eleven real-world tasks, DexAgent-trained policies average 63.6% success, approximately 3.5× the rate of SPIDER.

    Source: B. Results Analysis · p. 7

  • Takeaways & Limitations

    Experiments demonstrate improved data quality and real-world policy performance across eleven dexterous tasks, with lower processing costs as the library grows.

    Source: §VI · p. 7

    Forcesensitive interactions can fail despite passing simulation checks, and undetected errors can propagate through the library.

    Source: §VI · p. 7

Abstract

from arXiv · show

Human videos offer a scalable source of demonstrations for dexterous robot manipulation. However, existing human-to-simulation-to-robot (Human2Sim2Robot) pipelines rely on predefined procedures that struggle to accommodate diverse object properties and interactions, particularly those involving articulated and deformable objects. We introduce DexAgent, an agentic Human2Sim2Robot framework that converts a single egocentric human video and a task prompt into physically grounded robot trajectories for policy training. It operates through four stages: semantic understanding of human videos, property-based simulation reconstruction, robot trajectory optimization, and robot data generation. At each stage, DexAgent adapts its approach to the task and object properties by selecting suitable skills from its tool library or developing new ones when needed. Property-specific verifiers assess stage outcomes for physical validity and task-specific requirements and provide feedback for refinement, preventing error propagation through the workflow. This adaptive, verification-guided process allows DexAgent to process diverse objects and long-horizon tasks. In the final stage, DexAgent varies object and robot states in simulation to generate diverse robot trajectories from a single human video, then retextures the rendered observations to facilitate sim-to-real transfer. Newly developed skills and verifiers are retained in its tool library, making it self-evolving to accumulate reusable capabilities. This reduces processing time as DexAgent encounters more human videos. Across eleven real-world tasks, policies trained with DexAgent-generated data achieve a 3.5x higher success rate than competing baselines. Project website: https://dexagent123.github.io/.

I. INTRODUCTION

Predefined Human2Sim2Robot procedures struggle with diverse object properties and long-horizon interactions, while DexAgent adapts its skills and verification across tasks. The paper reports higher policy success over competing baselines and positions the framework against prior approaches that remain difficult to extend or depend on robot demonstrations.

I. INTRODUCTION

  • Policies trained with DexAgent-generated data achieve a 3.5× higher success rate than competing baselines across eleven real-world tasks.The evaluated tasks involve rigid, articulated, and deformable objects.Source: §I · p. 2
  • DexAgent converts one human video into physically grounded robot trajectories by selecting or developing property-specific skills for diverse dexterous tasks.Its four-stage workflow adapts semantic understanding, reconstruction, trajectory generation, and data generation to task and object properties.Source: §I · pp. 1–2
  • Stage verifiers provide feedback for refinement, while retaining newly developed skills and verifiers makes the library self-evolving and reduces later processing time.The verification-guided process limits error propagation through the workflow.Source: §I · p. 2

II. RELATED WORK

  • R3M [9], EgoVLA [10], EgoMimic [11], and EgoScale use human-video data but still rely on robot demonstrations for training or adaptation.Human2Sim2Robot approaches also face difficulty extending to diverse object types and long-horizon tasks, which DexAgent addresses through property-specific skills and verification.Source: §II · p. 2

III. METHODOLOGY

DexAgent processes a human video and task prompt through semantic understanding, simulation reconstruction, trajectory generation, and data generation. Each stage uses or develops tools suited to the task and advances only after verification, within a stage execution budget.

III. METHODOLOGY

  • Each stage selects or develops skills, refines results using verifier feedback, and passes only verified outputs forward unless its execution budget is exhausted.Skills and verifiers are stored in the tool library for reuse.Source: §III · pp. 2–3, A. Skills and Verifiers · p. 3

B. Semantic Understanding

  • Semantic understanding records object types and required properties alongside decomposed subtasks, acting hands, objects, and goals in a shared JSON asset.The object properties guide reconstruction, while the subtask goals inform trajectory optimization.Source: B. Semantic Understanding · p. 3

C. Property-Based Simulation Reconstruction

  • Reconstruction verifiers are selected for object properties, and reusable skills are selected or created to build assets that jointly pass those checks.Checks cover rigid-object silhouette, scale, mass, and stability; articulated-object joints; and deformable-object topology.Source: C. Property-Based Simulation… · p. 3
  • DexAgent places verified assets in a simulation scene with the robot setup and camera aligned to the original video.Placement is optimized against the overlap between rendered and human-video object masks.Source: C. Property-Based Simulation… · pp. 3–4

III. METHODOLOGY

  • The four-stage workflow ends by augmenting and retexturing robot data, with Stage 3 combining human-motion-guided optimization and code-generated motion.The stages move from task decomposition and property identification to simulation reconstruction, robot trajectories, and policy-training data.Source: §III · pp. 2–3, Figure 2 · p. 4

D. Trajectory Optimization

Trajectory optimization generates a robot episode from task-specific subgoals and verified hand poses. It combines pose-selection skills with motion generation, then checks physical consistency across the complete trajectory.

D. Trajectory Optimization

  • DexAgent defines a success condition for each subtask, such as a target object position or articulated joint state.These conditions specify whether a generated motion accomplishes its subtask.Source: D. Trajectory Optimization · p. 4
  • Pose skills can retarget demonstrated poses, reproduce demonstrated contacts, or generate task-suited grasps, and verifiers test whether each pose supports the required interaction.Failed poses are refined or replaced; only verified poses proceed to trajectory generation.Source: D. Trajectory Optimization · p. 4
  • Inverse kinematics maps verified hand poses to arm configurations, and generated code connects them with optional human-motion priors.Subtask motions are combined into an episode, earlier motions may be revised to support later subtasks, and whole-trajectory verifiers check motion limits and physical consistency.Source: D. Trajectory Optimization · p. 4

E. Data Generation and Policy Training

DexAgent augments and retextures verified trajectories to produce policy-training data from a single human video. Reconstruction and trajectory evaluations report gains over baselines, while simulation rollouts test both physical validity and motion fidelity.

E. Data Generation and Policy Training

  • DexAgent varies initial object and robot states, adapts trajectories, and retextures rendered observations to create diverse training episodes from a single video.Augmented episodes pass the same task-specific pose and whole-trajectory checks as the original episode.Source: E. Data Generation and Policy Training · pp. 4–5

IV. DATA QUALITY EXPERIMENTS

  • The data-quality experiments use MuJoCo to compare reconstruction and robot-trajectory quality against baselines.The trajectory evaluation measures physics-check success and time-averaged object-pose deviation from the demonstration.Source: §IV · p. 5, B. Robot Trajectory Generation Quality · p. 5

A. Human–Object Interaction Reconstruction

  • For articulated objects, DexAgent improves F-5 and F-10 by 17.5% and 11.5%, respectively, and reduces Chamfer distance by 10.7% against the strongest baseline.For rigid objects, the corresponding reported gains are 15.3% and 5.5% for F-5 and F-10, with a 40.8% Chamfer-distance reduction.Source: A. Human–Object Interaction… · p. 5

B. Robot Trajectory Generation Quality

  • DexAgent achieves the highest trajectory success rate, with a 5.8% relative improvement over the strongest baseline, and reduces rotational error by 20.0% while matching positional error.Simulation rollouts count clips as successful when mean object-position error is below 0.1 m and mean rotation error is below 0.5 rad.Source: B. Robot Trajectory Generation Quality · p. 5

V. PHYSICAL EXPERIMENTS

The physical experiments evaluate trajectory replay and learned-policy performance on a shared set of eleven dexterous tasks. The setup controls baseline pipeline components and uses common data and visual-processing protocols.

V. PHYSICAL EXPERIMENTS

  • The trajectory-replay benchmark gives each method the same human demonstration video, while DexAgent reconstructs the scene and generates a verified trajectory.Depending on the task, DexAgent uses MuJoCo or Isaac Sim.Source: §V · p. 5

A. Evaluation Setup

  • DexAgent takes 2.1 hours per sample in a fresh run, versus 3.3 hours for GPT 6: Astra and 3.7 hours for V2D.The figure reports reductions of 36.4% versus GPT 6: Astra and 43.2% versus V2D.Source: Figure 4 · p. 6
  • Policies are evaluated over ten randomized-state trials per task, using 500 robot episodes for fine-tuning and reporting the fraction of successful trials.Replay uses ten executions of each generated trajectory; a task counts as successful if at least one succeeds.Source: A. Evaluation Setup · p. 6
  • Baselines retain their supported reconstruction or retargeting components, with common implementations filling missing stages and shared data augmentation and visual processing.All methods train the same policy model; GPT-6: Astra is evaluated zero-shot without the DexAgent harness.Source: A. Evaluation Setup · p. 6
  • Eleven tasks span rigid, articulated, deformable, tool-use, and long-horizon manipulation.Examples include cup placement, bottle opening, rope knotting, drawing and wiping, and computer installation.Source: A. Evaluation Setup · p. 6

B. Results Analysis

DexAgent’s generated data supports stronger physical policy performance across the evaluated tasks, while its reusable library reduces processing time as videos accumulate. The experiments also identify verification and simulation-to-reality limits.

B. Results Analysis

  • DexAgent achieves successful trajectory replay on all eleven tasks, compared with six for SPIDER.The passage attributes this pattern to greater physical executability and broader task coverage than other baselines, which struggle with deformable, articulated, or long-horizon tasks.Source: B. Results Analysis · p. 7
  • 63.6% average success rate across eleven tasks is approximately 3.5× SPIDER and 3.7× V2D.Drawer success is 80% versus the highest baseline’s 50%, and rope-knot success is 50% while baselines score zero.Source: B. Results Analysis · p. 7
  • By sample 78, the library contains 85 skills and 168 verifiers that later videos can reuse.The videos were ordered with recurring-object cases last to evaluate asset reuse.Source: B. Results Analysis · p. 7
  • Reuse reduces processing time from 4.3 hours for the first video to 2.2 hours with skill reuse and 13.1 minutes with asset reuse.DexAgent produces a verified episode approximately 17 times faster than V2D and 15.1 times faster than GPT-6 Astra when using its tool library and asset reuse.Source: B. Results Analysis · p. 7

VI. CONCLUSION

  • The framework iterates stages until verifiers pass and retains newly developed skills and verifiers for reuse across videos.The conclusion reports improved data quality and real-world policy performance across eleven tasks, with processing costs falling as the library grows.Source: §VI · p. 7
  • Remaining limitations include semantic and geometric errors, incomplete verification, and simulation-to-reality gaps.Undetected errors can propagate through the library, and force-sensitive interactions may fail despite passing simulation checks.Source: §VI · p. 7

APPENDIX

The appendix illustrates how a single human demonstration becomes varied robot training data and shows semantic outputs that encode task steps, hand roles, object properties, and observable goals.

APPENDIX

  • The appendix shows an egocentric scene with the demonstrator removed and a retextured robot rendered in, alongside left- and right-wrist views and randomized examples.The views are used for policy training, and different random seeds illustrate data diversification.Source: Figure 6 · p. 10

A. Additional Robot Data Generation Examples

  • DexAgent samples 1,500 initial object configurations, rejects infeasible cases, and selects 500 successful seeds for training.Each accepted configuration is rendered from egocentric and both wrist views; more seeds are sampled if fewer than 500 pass.Source: A. Additional Robot Data Generation… · p. 10

B. Example Outputs from the Semantic Understanding Stage

  • Stage 1 semantics represent task-relevant objects and properties, ordered subtasks, acting hands and roles, and observable success conditions.These conditions guide and verify later stages.Source: B. Example Outputs from the Semantic… · p. 10

1) Drawing and Wiping:

  • The drawing-and-wiping example specifies writing digits and selectively erasing 4, with contact properties describing persistent marker strokes and eraser removal.Its subgoals assign the writing and wiping actions to different hands while preserving digits 1–3.Source: 1) Drawing and Wiping · p. 11

2) Open the Bottle:

The bottle example encodes an articulated cap-and-tube interaction as ordered subtasks, while distinguishing the intended pouring outcome from what the video visibly confirms.

2) Open the Bottle:

  • The bottle is represented as an articulated object with a body, mouth, screw cap, and attached tube.The cap turns relative to the body before detaching, and the tube must clear the mouth.Source: 2) Open the Bottle · p. 12
  • The task sequence includes loosening the cap with both hands and lifting off the cap assembly.The left hand resists body rotation while the right turns and readjusts its grip.Source: 2) Open the Bottle · p. 12
  • The pouring subgoal is to transfer water into a mug, but the observed video shows a pouring posture without clearly resolved liquid transfer.The left hand tilts the bottle toward the mug while the right remains by the cap.Source: 2) Open the Bottle · p. 12

3) Place Toy in Drawer:

The example models placing a small plush toy in a drawer as a sequence of subgoals, using an articulated drawer representation and specifying each action’s intended outcome.

3) Place Toy in Drawer:

  • The drawer is represented as an articulated object with a single-DOF prismatic joint connecting its sliding tray to the housing.The tray moves outward to open and inward to close, with motion perpendicular to the sliding axis constrained.Source: 3) Place Toy in Drawer · p. 13
  • The scene description identifies the drawer housing and front panel as gray, the tray interior as dark gray, and the handle as black.The handle is described as a front-mounted pull handle.Source: 3) Place Toy in Drawer · p. 13
  • The observed opening travel is estimated at approximately one housing depth, explicitly as a rough visual estimate rather than a measured mechanical limit.Source: 3) Place Toy in Drawer · p. 13
  • The task sequence opens the drawer, lifts the toy, moves it over the tray, releases it inside, and closes the drawer.The stated goals include making the drawer interior accessible, clearing the toy from the tabletop, and resting it unsupported in the tray’s front region.Source: 3) Place Toy in Drawer · p. 13

C. Additional Simulation Reconstruction and Robot Trajectory Optimization Examples

Additional examples show simulation reconstruction and robot trajectory optimization on tasks outside the main paper, tracing demonstrations through reconstructed scenes to optimized robot motions.

C. Additional Simulation Reconstruction and Robot Trajectory Optimization Examples

  • Each example traces a task from its recorded human demonstration, through the Stage 2 reconstructed scene, to the Stage 3 optimized robot trajectory.Figure 7’s first column shows the demonstration, its second the reconstructed scene, and columns 3–5 representative trajectory frames t0–t3.Source: C. Additional Simulation… · p. 13, Figure 7 · p. 14
  • The examples cover tasks not included in the main paper and span rigid, articulated, and deformable objects.Source: C. Additional Simulation… · p. 13

D. Current Library Tools Breakdown

The current tool library comprises skills and verifiers spanning DexAgent’s first three stages, organized by function and retained as the system processes additional demonstrations.

D. Current Library Tools Breakdown

  • The current library contains 103 skills and 188 verifiers accumulated from the EgoDex experiment and task samples.The tools span Stages 1–3 and are grouped into six functional categories each.Source: D. Current Library Tools Breakdown · p. 15
  • The six skill categories cover trajectory and execution, object reconstruction, perception and grounding, scene runtime and rendering, grasp and poses, and measurements.Examples include subgoal chaining, reconstructing drawers, extracting hand motion, generating valid hand configurations, and measuring obstacle clearance.Source: D. Current Library Tools Breakdown · p. 15
  • The six verifier categories check task completion, grasp stability, reconstructed mechanisms, episode consistency, task-specific scene conditions, and scene compilation validity.Examples include testing grasps under lifting or shaking and detecting penetration, unsupported objects, or missing mechanisms.Source: D. Current Library Tools Breakdown · p. 15
  • Newly developed tools are retained as DexAgent processes demonstrations, with the library expected to grow through reusable capabilities and checks.The authors expect these tools and verifiers to generalize to new samples and tasks.Source: D. Current Library Tools Breakdown · p. 15
Loading 2609.35318v1…