Source-linked AI summary
Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation
Bing Cheng, Yiran Mao, Zewei Ye, Yifan Jin, Xinyi Chen, Ruifeng Tu, Shuhuai Wang, Tao Lin, Zhimu Zhou, Yuxin Du, Fangwei Zhong, Junchi Yan, Bo Zhao
TL;DR
Existing benchmarks often provide many target demonstrations and leave source–target relatedness uncontrolled, making transfer hard to distinguish from memorization. Evo-DexBench evaluates few-shot transfer between tasks sharing functional interactions; functionally matched Core data tends to yield larger gains, while the best mean success rate is 38.5%.
Problem
Existing evaluations often provide many target demonstrations or leave source–target relatedness uncontrolled, making it difficult to assess transfer separately from memorization.
Source: §1 · p. 2Method
Evo-DexBench pairs related Core tasks with a Gen target sharing their functional interaction and gives the target only a few demonstrations.
Source: §1 · p. 2, §3.1 · p. 4Results
Functionally matched Core data tends to yield larger gains than mismatched data, and the best mean success rate is 38.5%.
Source: §1 · p. 3Takeaways & Limitations
Evo-DexBench provides a benchmark for evaluating dexterous policies' generalization across related tasks.
Source: §5 · p. 10Evaluations are primarily based on a specific robot embodiment and do not systematically examine cross-embodiment transfer.
Source: §5 · p. 10
Abstract
from arXiv · showhide
General-purpose dexterous manipulation requires policies to transfer knowledge from related tasks to new tasks with only a few target demonstrations. Existing benchmarks often provide too many target demonstrations. They also leave the differences between source and target tasks uncontrolled. As a result, they cannot clearly distinguish knowledge transfer from task memorization. Our key insight is the Core-Gen principle. Core tasks are related tasks that share a functional interaction and provide source knowledge, while a Gen task is a novel target within the same functional family that has only a few demonstrations. This setup tests whether a policy can reuse functional interaction knowledge. Based on this principle, we propose Evo-DexBench, which controls both source-task relatedness and target-task exposure. It comprises 30 tasks in six functional families and a VR teleoperation pipeline. The resulting dataset covers three everyday scenes per task. It contains 1,248 multimodal demonstrations collected with simulated Shadow Hands equipped with 123 taxels. Experiments show that even the best-performing method achieves an average success rate of only 38.5%, highlighting the challenges existing methods face in transferring functional interaction knowledge. Further analysis shows that, across the two tasks, joint Core+Gen training improves performance by 12% and 38%, respectively, over training on Core followed by fine-tuning on Gen. This indicates that joint training makes better use of Core knowledge and that functionally matched Core data yields greater gains in success rate than mismatched data. We release the Evo-DexBench dataset and benchmark to support the evaluation of dexterous manipulation policies' generalization across related tasks.
4 CORE + 1 GEN Dual-Arm Single-Arm
Evo-DexBench organizes tasks around shared functional interactions, with Core tasks providing related source experience and Gen tasks serving as targets. Tasks also vary across scenes and functional categories.
4 CORE + 1 GEN Dual-Arm Single-Arm
- Examples pair pressing a button with pressing a monitor power button, and configuration with holding and closing.Source: §4 · p. 1
- The listed task categories include structured configuration, button-driven rotational manipulation, access-and-transfer, and articulated configuration control.Source: §4 · p. 1
- Each of six functional task categories contains four Core tasks and one Gen task, with tasks switching among three scenes in Kitchen or Studio environments.Scene switching occurs during data collection and evaluation.Source: Figure 1 · p. 1
1 Introduction
Dexterous manipulation policies must transfer functional interactions to new tasks with few demonstrations, but existing evaluations do not reliably isolate that ability. Evo-DexBench controls source-task relatedness and target exposure to test transfer, and current methods remain limited.
1 Introduction
- Existing evaluations can confound transfer with memorization by providing many target demonstrations or using source and target tasks with uncontrolled relatedness.Dexterous success also depends on finger configurations and contact relationships, not only end-effector trajectories.Source: §1 · p. 2
- Evo-DexBench varies objects and scene conditions while giving policies few target demonstrations, testing whether related source-task experience transfers.It also examines whether experience with more related tasks leads to better performance.Source: §1 · p. 2
- The benchmark follows a Core-Gen design and pairs shared-functional-interaction tasks with three everyday scenes per task, collecting 1,248 multimodal demonstrations with simulated Shadow Hands equipped with 123 taxels.The integrated simulation environment and VR teleoperation pipeline support evaluation.Source: §1 · p. 2
- Table 1 compares embodied-manipulation benchmarks, while prior suites span functional grasping, unseen-object manipulation, and bimanual coordination.Source: Table 1 · p. 3
- The best mean success rate is only 38.5%, and functionally matched Core data tends to yield larger gains than mismatched data.These results indicate that current methods still struggle to extract and reuse functional interaction knowledge.Source: §1 · p. 3
2 Related Work
Prior manipulation benchmarks cover unseen environments or held-out tasks, but generally do not control whether source and target tasks share functional interactions. Evo-DexBench evaluates few-shot transfer from related source tasks to novel targets.
2 Related Work
- Existing evaluations include unseen-environment tests, held-out task splits, and task suites with knowledge partitions, but define splits by task identity rather than shared functional interaction.Some approaches adapt from rewards alone or use abundant demonstrations, including 50 per task in LIBERO.Source: §2 · p. 3
- Dexterous benchmarks cover functional grasping, unseen-object manipulation, and bimanual coordination, but typically test predefined task completion rather than contact-level interaction transfer.Source: §2 · p. 3
- Recent dexterous policies pursue reusable behavior across tasks, objects, and embodiments, yet benchmarks still struggle to distinguish transferable interaction knowledge from target-task memorization.Evo-DexBench addresses this by testing transfer from multiple related source tasks with few target demonstrations.Source: §2 · pp. 3–4
- The framework combines six task categories, Kitchen and Studio scenes, 123 hand taxels, and layout, camera, lighting, and physical-property variation while decoupling task logic from scene layout.Policies and keyboard/mouse or Quest 3 teleoperation use a unified interface.Source: Figure 2 · p. 4
3 Benchmark Design
Evo-DexBench evaluates whether policies transfer functional interaction knowledge from related Core tasks to a new Gen task with only a few target demonstrations. Its design combines this controlled task pairing with a VR demonstration pipeline and shared simulation, observation, action, and tactile interfaces.
3.1 Design Principles and Task Composition
- Core-Gen pairs functionally related Core tasks with a Gen target sharing their interaction, using few target demonstrations to test transfer rather than memorization.The benchmark also assesses generalization across scene variations.Source: §3.1 · p. 4, §3 · p. 4
- The benchmark organizes 30 tasks into six affordance families, each with four Core tasks and one Gen task; the first three use single-arm setups and the last three bimanual setups.Access-and-transfer and structured configuration tasks are long-horizon.Source: §3.1 · pp. 5–6
3.2 Data Collection Pipeline
- Quest 3 WebXR hand tracking supports simulated teleoperation, mapping operator wrist motion to the Panda arm and hand keypoints to Shadow Hand joint targets.The hand targets use a dexterous-hand retargeting method [39].Source: §3.2 · p. 6
- The dataset contains 1,248 successful trajectories across three randomly selected scene layouts per task, with 50 demonstrations per Core task and eight per Gen task.Canonical HDF5 preserves complete multimodal information, while LeRobot-v3 is provided for direct policy training.Source: §3.2 · p. 6
3.3 Implementation Details
- Studio and Kitchen scenes each have three layouts that vary furniture, object placement, and visual background, with tasks executed across everyday environments.The composable design reuses task logic across layouts and scenes across tasks.Source: §3.3 · p. 6
- Implementation uses role-based scene assets, three camera views, and a 123-dimensional normal-force signal from each Shadow Hand.The context and interaction cameras are manually calibrated for each task.Source: §3.3 · pp. 6–7
4 Experiments
The experiments compare policy architectures, few-shot adaptation strategies, functional transfer, and tactile sensing across Evo-DexBench tasks. They find that matched Core data and tactile input can improve Gen-task performance, while benefits vary by policy, family, and task.
4.1 Experimental Setup and Protocol
- Policies are evaluated on three one- or two-handed task families, with four Core tasks per family, eight Gen demonstrations, and 100 episodes per task.Episode sets include nominal and camera, lighting, physical, and spatial variations, with shared reset schedules and frozen policy weights.Source: §4.1 · p. 7
4.2 Baseline Policy Performance
- No single policy dominates across 15 tasks; DP-C, 𝜋0.5, and KineDex form the leading aggregate group, while SmolVLA is competitive but lower and DP-T trails.Family-level winners differ: KineDex leads Button-driven manipulation, 𝜋0.5 leads Rotational manipulation, and DP-C leads Articulated configuration control.Source: §4.2 · pp. 7–8, Table 3 · p. 8, Figure 5 · p. 8
4.3 Few-Shot Adaptation with Core Data
- Joint Core + Gen training performs best on both few-shot targets, while sequential fine-tuning from Core checkpoints provides limited or negative transfer.The comparison tests whether Core data is more useful as initialization or when retained throughout target adaptation.Source: §4.3 · p. 9, Table 4 · p. 9
4.4 Functional Transfer across Task Families
- Matched Core data provides the largest transfer gains, and broader coverage within a matched family generally outperforms narrower subsets.For DishwasherKnob, the matched Rotational source provides the strongest overall transfer across three models; source-task choice can still affect transfer quality.Source: §4.4 · pp. 9–10, Figure 6 · p. 9, Table 5 · p. 10
4.5 Tactile Sensing Ablation
- Adding tactile sensing improves success on seven of ten Rotational and Articulated tasks, with positive mean gains in both families.The largest improvement is on StaplerPaperMark; gains vary by task and are particularly pronounced for contact-rich tasks.Source: §4.5 · p. 10
5 Discussion and Conclusion
Evo-DexBench evaluates dexterous policies using Core-Gen pairs that share functional interaction requirements, testing transfer to new objects and tasks. Current evaluations are primarily limited to a specific robot embodiment and do not systematically examine cross-embodiment transfer.
5 Discussion and Conclusion
- Evo-DexBench uses Core-Gen task pairs sharing functional interaction requirements to assess transfer to new objects and tasks.The authors report that representative methods remain limited in extracting and reusing functional interaction knowledge.Source: §5 · p. 10
- Evaluations are primarily based on a specific robot embodiment and do not systematically examine cross-embodiment transfer.The authors identify evaluation across multiple robot embodiments as future work.Source: §5 · p. 10
A Complete Benchmark Specification
Evo-DexBench specifies related Core tasks and few-demonstration Gen targets, with task-specific progress criteria and controlled scene and environment variation. It also defines tactile observations and a tactile-feedback policy interface.
A Complete Benchmark Specification
- 30 tasks across six functional families pair four 50-demonstration Core tasks with one eight-demonstration Gen task per family.Core tasks share a functional interaction; the Gen task applies it in a new task context.Source: A Complete Benchmark Specification · p. 14
- Stage thresholds use simulator state and contact measurements, with terminal conditions defining 100% success and held conditions filtering transient contacts.Earlier milestones can be reached independently and distinguish progress from arbitrary motion.Source: A Complete Benchmark Specification · p. 14
B.1 Scene-layout variation
- Each task has three independently constructed layouts that vary visual context and scene composition, but are not a held-out scene-generalization split.The figure shows 90 task–layout combinations, with each task’s layouts arranged left to right.Source: §B.1 · p. 18
B.2 Generalization axes
- Generalized episodes perturb exactly one axis at a time, and each task is evaluated in 100 episodes across nominal and four single-factor conditions.Policies share condition, seed, layout, and reset schedules; metric or angular offsets are expanded to 1.5× their collection range for evaluation.Source: §B.2 · p. 24
C.1 Taxel layout and representation
- Each Shadow Hand has 123 taxels, and the policy-facing tactile observation is a non-negative scalar normal force in Newtons per taxel.The layout assigns 100 taxels to fingers and 23 to palm/sub-palm regions.Source: §C.1 · pp. 24–25
C.2 From contacts to tactile force
- Contact impulses are assigned to eligible taxels, accumulated over each control period, and converted to non-negative average compressive normal force.Eligible contacts must match the parent link and taxel patch geometry; weighted assignment distributes rather than duplicates covered impulse.Source: §C.2 · p. 25
C.3 KineDex interface and No-Touch control
- Full KineDex predicts 30 motion values and five finger-force targets per hand, then uses bounded force feedback to adjust flexion commands.No-Touch is separately trained with zero tactile inputs and bypasses force targets and the deployment-time force controller.Source: §C.3 · p. 25
D Detailed Data, Training, and Evaluation Settings
Demonstrations are stored in a policy-independent format with synchronized multimodal observations and actions. Policies share a compact semantic state/action interface, with separate checkpoints for single-arm and bimanual data.
D.1 Dataset and common policy contract
- Each Core task contributes 50 successful trajectories and each Gen task eight, recorded at 30 Hz with synchronized observations, actions, instructions, task-stage state, and tactile measurements when enabled.The simulator runs at 120 Hz, so four physics steps contribute to each policy observation and action; only successful trajectories enter training sets.Source: §D.1 · pp. 25–26
- The common interface uses 31-D state and 30-D action for one arm–hand system, or 62-D state and 60-D action for bimanual control.Single-arm and bimanual data use separate checkpoints, and normalization statistics are recomputed for each exact training mixture.Source: §D.1 · p. 26
D.2 Policy implementations and adaptations
The benchmark retains each policy’s released architecture and learning objective while adapting interfaces to a shared data and action contract. Training protocols balance Core and Gen data and compare target-only, sequential-transfer, and joint-training strategies.
D.2 Policy implementations and adaptations
- Policy implementations retain released architectures and objectives while using a shared LeRobot-v3 interface, compact state/action convention, and source-balanced sampler.Adapters accommodate policy-specific inputs and outputs; KineDex adds five force targets to its 30 motion dimensions.Source: §D.2 · pp. 26–27
D.3 Optimization and transfer protocols
- Standard baselines draw half of each minibatch from Gen and divide the other half equally among its four Core tasks.This prevents the larger 50-trajectory Core sets from overwhelming the eight-shot target.Source: §D.3 · p. 28
- The adaptation study compares Gen-only, full or early Core-to-Gen fine-tuning, and joint Core+Gen training under a fixed target-stage budget.Sequential runs reinitialize target normalization and optimizer state; joint training uses the 1:1 mixture for the full schedule.Source: §D.3 · p. 28
- The Core-source study varies only the Core source paired with a fixed Gen target while preserving the 1:1 mixture ratio.Selected Core tasks are balanced within their half of each minibatch.Source: §D.3 · p. 28
D.4 Evaluation, metrics, and uncertainty
Evaluation uses frozen policies on shared episode schedules and reports task success and normalized stage progress. The supplied result tables cover baseline performance, Core-source conditions, and Core-set breadth and composition.
D.4 Evaluation, metrics, and uncertainty
- Each task score aggregates 100 frozen-policy episodes, with 20 nominal and 20 for each of four variation conditions; no test-time adaptation occurs.Compared models receive identical registered episodes.Source: §D.4 · p. 28
- Success rate measures terminal task success, while StageScore measures the mean normalized index of the highest completed stage; aggregate scores macro-average tasks.The 95% Wilson interval quantifies rollout uncertainty for fixed policies and tasks, not training-seed variability; condition-specific success rates resolve to five percentage points.Source: §D.4 · p. 28
E.1 Complete baseline results
- Table 15 gives complete baseline results for 15 tasks evaluated by all five policies, with overall SR and StageScore plus condition-specific SR.The condition columns are nominal, camera, lighting, physical, and spatial.Source: Table 15 · p. 29, §E.1 · pp. 29–30
E.2 Complete Core-source relatedness results
- Table 16 lists retained π0.5 Core-source conditions, with each mixed condition allocating half its samples to the target and half to the listed Core source.Its Gen-only values use the same retested reference runs as the transfer heatmap.Source: §E.2 · p. 30
E.3 Core-set breadth and composition
- Table 17 compares the complete four-task Core family with two-task subsets while holding Gen data and epoch budget fixed.The comparison changes both source-set breadth and composition.Source: §E.3 · p. 30
E.4 Complete tactile ablation results
The complete tactile pathway improves mean SR and StageScore in both reported functional categories, but task-level benefits are heterogeneous rather than uniform.
E.4 Complete tactile ablation results
- Touch helps strongly on StaplerPaperMark, CoffeeKnob, KettleLidTab, and BoxLidOpening, but does not improve every task.The results support a task-dependent tactile benefit rather than uniform dominance.Source: §E.4 · p. 30
- Full KineDex improves mean SR by 3.8 points on Rotational manipulation and 2.2 points on Articulated configuration control, with StageScore gains of 3.0 and 3.5 points, respectively.Table 18 reports the task-wise tactile ablation in SR (%) and StageScore.Source: §E.4 · p. 30, Table 18 · p. 31