PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Dexterous Manipulation via Action Representation Pretraining

Yanjinghao Xu, Jiexin Zhang, Huan Zhao, Jianjia Mo, Zhenghao Yin, Dingkang Liang, Xiang Bai, Han Ding

arXiv:2610.34453v1cs.RO

TL;DR

Robot demonstrations must support both action-representation learning and task-conditioned action modeling, limiting complex dexterous action generation. ARPG pretrains action representations on human manipulation data, then generates in the frozen representation space; it outperforms π0.5 overall on DexJoCo, with average bimanual success increasing from 10.1% to 44.7%.

  • Problem

    Robot demonstrations must support learning both action representations and task-conditioned action distributions, constraining complex action generation by demonstration scale and downstream-model capacity.

    Source: §1 · p. 1

  • Method

    ARPG pretrains an action encoder on human manipulation data, freezes it, and trains a generative backbone to model task-conditioned distributions in its representation space.

    Source: §1 · p. 2

  • Results

    ARPG outperforms π0.5 overall on DexJoCo; average bimanual dexterous-manipulation success rises from 10.1% to 44.7%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Broader coverage of manipulation behaviors in pretraining data effectively improves downstream performance.

    Source: §6 · p. 9

    ARPG has so far been validated only on dexterous-hand manipulation; transfer to other robot embodiments remains to be studied.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large-scale vision, language, and video pretraining has substantially improved the perception and generalization capabilities of robotic manipulation models, but action generation for complex dexterous manipulation remains constrained by the scarcity of robot demonstrations. Limited robot demonstrations must be used to learn both action representations and task-conditioned action distributions, making it difficult to adequately learn action representations and, in turn, limiting the ability of downstream generative models to generate complex dexterous actions. To address this limitation, we propose ARPG (Action Representation Pretraining and Generation), which explicitly separates action representation learning from action generation: we first pretrain an action representation encoder on large-scale human manipulation data. We then freeze the encoder and train a compact generative backbone (130M parameters) to learn task-conditioned action distributions in the encoder's representation space. Experimental results show that ARPG achieves state-of-the-art performance on the DexJoCo benchmark and improves the average success rate on bimanual dexterous manipulation from 10.1% for $\pi_{0.5}$ to 44.7%. Scaling experiments further show that broader coverage of human manipulation behaviors effectively improves downstream performance. These results indicate that action representation pretraining provides an independent scaling dimension for improving complex dexterous manipulation capabilities.

1 INTRODUCTION

Robot demonstrations must support both action-representation learning and task-conditioned action modeling, while visual representations are not directly decodable action spaces. ARPG separates these stages and reports improved DexJoCo bimanual performance with a compact generative backbone.

1 INTRODUCTION

  • Robot policies must learn action representations and task-conditioned action distributions from demonstrations, constraining complex action generation by demonstration scale and downstream-model capacity.The paper asks whether scalable action-representation pretraining can improve dexterous manipulation without scaling either robot demonstrations or the generative backbone.Source: §1 · p. 1
  • Visual representations encode motion and interaction semantics but are not designed for action recovery, while action-only learning must build representation structure from scratch.This motivates using pretrained visual representations as a reference for an independently learned action space.Source: §1 · pp. 1–2
  • ARPG pretrains action representations, freezes the encoder, and learns task-conditioned action generation in that representation space.The approach aligns action representations with visual representations, then decodes generated representations into executable actions.Source: §1 · p. 2
  • 44.7% versus 10.1% for π0.5 on average bimanual dexterous-manipulation success; ARPG also outperforms π0.5 overall on DexJoCo with an approximately 130M-parameter backbone.The paper also reports substantially reduced inference costs.Source: §1 · p. 2

2 RELATED WORK

Prior work improves robot generalization using pretrained vision-language or video models, scales learning with human manipulation data, and compresses or reparameterizes action spaces. ARPG instead independently pretrains action representations and uses them as a fixed generative space.

Generalist Robot Policies

  • VLA models transfer pretrained visual and semantic knowledge to robot control, while world action models jointly model future states and actions to add physical-dynamics and interaction priors.ARPG instead emphasizes complex dexterous action generation and action representations.Source: §2 · p. 3

Learning from Human Manipulation Data

  • Human-data approaches retarget trajectories, learn latent actions from video, or pretrain complete models; ARPG uses human manipulation data to pretrain reusable bimanual dexterous-action representations.These approaches address the cost and difficulty of scaling high-quality robot demonstrations.Source: §2 · p. 3

Learning Action Spaces for Generation

  • Prior generative policies compress or reparameterize action spaces, whereas ARPG independently pretrains action representations and models task-conditioned distributions in that fixed space.The related work contrasts this with visual-generation approaches that use pretrained representations to shape the generative space.Source: §2 · p. 3

3 ACTION REPRESENTATION PRETRAINING

ARPG first learns a decodable action representation space and then fixes it for conditional generation. Human manipulation data provide scalable pretraining, while target-robot demonstrations train decoding and help the representation produce executable actions.

3 ACTION REPRESENTATION PRETRAINING

  • ARPG separates action-representation learning from downstream conditional generation, fixing the learned representation space before generation.The framework uses human manipulation data for representation pretraining and target-robot demonstrations for decoding.Source: §3 · p. 3, Figure 2 · p. 4

3.1 ACTION REPRESENTATION LEARNING

  • The action encoder maps continuous sequences to temporal representations that must support both structured distribution modeling and recovery of executable actions.Video features serve as a fixed reference because visual representations carry motion and interaction structure but cannot directly serve as a decodable action space.Source: §3.1 · pp. 3–4
  • The action representation averages normalized features from the last K Transformer layers and serves as both the decoder input and downstream generative space.The K = 2 aggregation setting achieves the best result in the reported ablation.Source: §3.1 · p. 4
  • A frozen video encoder and trainable action Transformer are aligned on synchronized video-action samples, transferring visual structure into the action representation.Same-clip pairs are positive and cross-sample pairs negative; the visual encoder remains frozen and only the action encoder and alignment modules are retained for subsequent action modeling.Source: §3.1 · pp. 4–5

3.2 PRETRAINING ON HUMAN MANIPULATION DATA

  • Decoupling representation learning from robot-specific generation allows pretraining on broader human manipulation data, while target-robot demonstrations support later decoding and conditional generation.A fixed-dimensional bimanual interface represents left- and right-hand actions; validity masks handle missing dimensions, and the data use synchronized one-second video-action clips.Source: §3.2 · p. 5

3.3 ACTION DECODER

  • After pretraining, the encoder is frozen and a decoder trained on target-robot demonstrations maps representations to executable actions.Noise-augmented decoder training uses locally perturbed representations with the same action supervision to mitigate distribution shift from generated representations.Source: §3.3 · p. 5

4 GENERATION IN THE PRETRAINED ACTION SPACE

ARPG learns conditional distributions over frozen pretrained action representations rather than raw controls, then decodes generated representations into action chunks. Its training combines flow matching with representation- and action-level auxiliary constraints.

4 GENERATION IN THE PRETRAINED ACTION SPACE

  • The encoder and decoder remain frozen during conditional generation, constructing representation targets and recovering executable actions, respectively.This stage models action distributions in the fixed pretrained space rather than directly in raw action space.Source: §4 · p. 5
  • Future action chunks are encoded and normalized as generation targets, with the policy learning their distribution conditioned on visual observations and robot state.Generated representations are denormalized and passed through the frozen decoder to recover continuous actions.Source: §4 · p. 6
  • A DiT trained with Flow Matching generates representations conditioned on visual features and robot state, starting from Gaussian noise.The visual condition comes from frozen V-JEPA 2.1 features.Source: §4 · p. 6
  • The training objective adds representation alignment, decoded-action constraints, and adjacent-action smoothness to Flow Matching.The action-level terms constrain decoded values and changes between adjacent actions.Source: §4 · p. 6
  • At inference, the policy generates representations from Gaussian noise and decodes them into future action chunks; the action encoder is not used.Source: §4 · p. 6

5 EXPERIMENTS

Experiments evaluate ARPG on DexJoCo and a real-world robot, examining dexterous performance, deployment efficiency, scaling, and representation and generative design choices.

5.1 EXPERIMENTAL SETUPS

  • Evaluation spans 11 DexJoCo tasks, three long-horizon real-world tasks, and comparisons with π0.5 and GR00T N1.5.DexJoCo includes six single-arm and five bimanual tasks; the real-world tasks are hammer handover, pouring coffee beans, and cabinet retrieval.Source: §5.1 · p. 6

5.2 EFFECTIVE AND EFFICIENT DEXTEROUS MANIPULATION

  • 57.8% vs 35.0% overall success under joint multi-task training, with ARPG maintaining approximately 45% bimanual success versus π0.5's 10.1%.The comparison is reported alongside single-task and multi-task macro-average success rates and inference costs.Source: Table 1 · p. 7, §5.2 · p. 7
  • ARPG achieves higher full-task success than π0.5 on all three real-world tasks: hammer handover, pouring coffee beans, and cabinet retrieval.Figure 3 reports successful rollouts over 30 trials for each task.Source: Figure 3 · p. 7, §5.2 · p. 7
  • ARPG uses substantially fewer active parameters, lower peak VRAM, and lower inference latency while retaining high complex-action performance.The results indicate that improving complex action generation need not rely solely on increasing generative-model capacity.Source: §5.2 · p. 7

5.3 SCALING ACTION REPRESENTATION PRETRAINING FOR DOWNSTREAM GENERATION

  • 40.9% to 57.8% as human pretraining data grow from 10 to 500 hours; at 3,000 hours, performance does not improve further as behavioral coverage remains largely unchanged.The 10-to-500-hour expansion adds behavioral coverage, while the 500-to-3,000-hour expansion mainly adds samples of existing manipulation types.Source: Figure 4 · p. 8, §5.3 · p. 8
  • DexJoCo success does not consistently improve with larger action encoders or more pretraining compute.The proposed explanation is a mismatch between sequence-level vision–action alignment and the token-level temporal representations modeled downstream.Source: §5.3 · pp. 8–9

5.4 ABLATIONS AND ANALYSIS

  • Visual Alignment gives the best representation result; K=2 aggregation and the full DiT objective also achieve the best results in their respective ablations.Action-MAE and Action-iBOT improve over Raw Action, but gains from action-only representation learning remain limited.Source: Table 2 · p. 9, §5.4 · p. 9

6 LIMITATIONS AND CONCLUSION

ARPG separates action-representation pretraining from downstream conditional action generation, and the authors frame this as a path to scaling dexterous manipulation independently of robot-demonstration scale and downstream-policy capacity. They report limitations in embodiment coverage, open-environment evaluation, and the scale of current data and compute.

6 LIMITATIONS AND CONCLUSION

  • ARPG separates action-representation pretraining from downstream conditional action generation, using more readily scalable human manipulation data to shape representations for downstream policies.The conclusion presents this as a path to scaling complex dexterous manipulation independently of robot-demonstration scale and downstream-policy capacity.Source: §6 · p. 9
  • The work is validated only on dexterous hands; transfer to grippers, open-environment semantic understanding and task generalization, and larger-scale behavior remain unvalidated.The authors state that current pretraining data and compute scales are limited and that larger-scale behavior requires further validation.Source: §6 · p. 9

A.1 ACTION ENCODER

  • The action encoder uses a Transformer to represent 30-step bimanual action sequences as 30 tokens of 512 dimensions, aggregating features from multiple terminal layers.The main model aggregates the last two Transformer blocks; separate attention-pooling modules produce sequence-level visual and action representations in a shared 512-dimensional space.Source: §A.1 · p. 14

A.2 ACTION ENCODER TRAINING OBJECTIVES

The section describes visual–action alignment for pretraining, then the decoder and conditional policy that use the learned representation. It also specifies the unified action format, human-data sources, and filtering process.

A.2 ACTION ENCODER TRAINING OBJECTIVES

  • Visual Alignment trains action representations by pairing synchronized video and action clips, treating same-clip pairs as positives and cross-clip pairs as negatives under a pairwise sigmoid objective.The visual encoder remains frozen while the action encoder and pooling and projection modules are updated.Source: §A.2 · p. 14

A.3 ACTION DECODER

  • The action decoder maps 30 action latents of 512 dimensions to a 30-step, 198-dimensional executable action sequence.Its input is the encoded latent sequence; the 126 additional wrist-relative keypoint dimensions are not output as robot commands.Source: §A.3 · p. 15

A.4 GENERATIVE BACKBONE

  • The conditional DiT generates future action latents in the frozen encoder’s representation space, conditioned on multi-view visual tokens, robot proprioception, and flow time.The main sequence comprises 30 latent tokens corresponding to a one-second, 30-step future action chunk.Source: §A.4 · pp. 15–16

B UNIFIED ACTION FORMAT

  • The unified bimanual format assigns each hand a 162-dimensional slot containing joint and palm rotations, absolute wrist position, and wrist-relative keypoints.The encoder uses the full 324 dimensions, while the decoder outputs 198 executable dimensions; the remaining 126 keypoint dimensions are auxiliary geometric information.Source: B UNIFIED ACTION FORMAT · p. 16

C HUMAN MANIPULATION PRETRAINING DATA

  • Human-manipulation pretraining uses synchronized one-second clips, sampling video at 4 Hz and actions at 30 Hz, while target-robot demonstrations train the decoder and downstream policy.The pretraining corpus is drawn primarily from EgoDex and EgoVerse.Source: C HUMAN MANIPULATION PRETRAINING DATA · p. 17

C.1 MAIN DATASETS

  • EgoDex supplies first-person RGB video and corresponding 3D hand trajectories, while EgoVerse supplies first-person videos and hand-motion trajectories.For EgoVerse clips with only one fully tracked hand, the valid hand trajectory is retained and the missing hand is handled with a validity mask.Source: §C.1 · p. 17

C.2 DATA FILTERING

  • Pretraining clips are filtered using hand-tracking confidence, valid-joint counts, reconstruction error, and inter-frame rotation change, then smoothed within contiguous valid intervals.Frames are invalidated below the 0.05 confidence threshold or 12 valid joints, above 0.02 m reconstruction error, or above 90° rotation change; smoothing does not cross missing-hand intervals.Source: §C.2 · p. 17

C.3 TRAINING-DATA SCALE

This material describes cumulative human-data scale sets and the robot demonstrations and hardware used for simulation and real-robot experiments. The robot data provide the target-platform demonstrations for decoding and policy training.

C.3 TRAINING-DATA SCALE

  • The human-data scale sets are cumulative, and the full pretraining set contains 11,117,819 one-second clips, totaling approximately 3,088 hours, called 3,000 hours in the main paper.The sets share preprocessing and architecture, with optimizer steps fixed across scales; the reported hours sum clip durations, not unique source-recording time.Source: §C.3 · p. 17, Table 4 · p. 18

D ROBOT DEMONSTRATIONS AND DOWNSTREAM DATA

  • The decoder and downstream policy train exclusively on target-robot demonstrations; the simulation experiments use DexJoCo demonstrations, while real-robot experiments use collected task data.The same data interface supplies visual observations, robot states, action targets, and validity masks.Source: D ROBOT DEMONSTRATIONS AND DOWNSTREAM… · p. 18

D.1 DEXJOCO DATA

  • DexJoCo provides approximately 1.1K demonstrations across 11 dexterous-manipulation tasks, including six single-arm and five bimanual tasks.The simulation uses Franka Panda arms and Wonik Allegro hands in the official rand-obj setting.Source: §D.1 · p. 18

D.2 REAL-ROBOT DATA-COLLECTION HARDWARE

  • The real-robot platform uses two 7-DoF KUKA iiwa arms and two 12-DoF DexHand hands, with one scene camera and two wrist cameras.Robot states and camera images are synchronized with a maximum permitted offset of 50 ms.Source: §D.2 · p. 18

D.3 REAL-ROBOT DATA CONTENT

  • Real-robot demonstrations cover Hammer Handover, Pouring Coffee Beans, and Cabinet Retrieval, with synchronized observations, states, and action targets.The action decoder and downstream policy use the same target-robot demonstrations to learn latent-to-action mapping and task-conditioned action generation.Source: §D.3 · pp. 18–19

E TRAINING DETAILS

The training procedure freezes the visual encoder during action-representation pretraining, then freezes the action encoder and decoder during policy training. The policy uses Flow Matching alongside auxiliary representation, decoded-action, and smoothness objectives.

E TRAINING DETAILS

  • Action-encoder pretraining uses frozen V-JEPA 2.1 visual representations and updates the action encoder, both pooling modules, and projection heads with same-clip positives and cross-clip negatives.The pairs are optimized with a pairwise sigmoid objective.Source: E TRAINING DETAILS · p. 19
  • During downstream policy training, the action encoder and decoder remain frozen while Conditional DiT learns future action-latent distributions conditioned on visual observations and robot proprioception.The training objectives include Flow Matching, REPA alignment, decoded-action supervision, and trajectory-smoothness supervision.Source: E TRAINING DETAILS · p. 19
  • The decoder is trained on target-robot demonstrations with masked Smooth-L1 reconstruction loss, and noise-augmented latents are used to improve robustness to generated-latent deviations.Twenty-five percent of samples retain clean latents; the maximum noise scale increases from 0 to 0.05 over the first 5,000 steps.Source: E TRAINING DETAILS · pp. 19–20
  • REPA aligns intermediate DiT features with clean encoded actions, decoded-action loss supervises executable outputs, and smoothness loss constrains adjacent-time-step action changes.Action-space losses apply validity masks; smoothness excludes a pair when a dimension is invalid at either adjacent time step.Source: E TRAINING DETAILS · p. 20

F SIMULATION EXPERIMENT DETAILS

The simulation experiments compare ARPG with reported DexJoCo baselines and evaluate single-task and joint multi-task training under the rand-obj setting. Evaluation uses a fixed rollout protocol and a specified rolling-replanning configuration.

Training setup

  • ARPG is trained separately per task for 60,000 steps or jointly on all 11 tasks for 120,000 steps, using the same architecture and optimization configuration.Multi-task training uses a task-aware shard-local sampler that favors distinct-task coverage within mini-batches, without enforcing equal global task probabilities; the final checkpoint is used without a separate validation set.Source: F SIMULATION EXPERIMENT DETAILS · p. 20

Baseline implementations

  • Baseline results for π0.5 and GR00T N1.5 come from the official DexJoCo report under the same rand-obj setting, while ARPG is retrained with the authors’ action-format conversion and pipeline.The official report supplies both π0.5 settings and GR00T N1.5 single-task results.Source: F SIMULATION EXPERIMENT DETAILS · p. 20

Evaluation protocol

  • Each task is evaluated over 150 rollouts—50 episodes for each of three random seeds—with per-task success rates reported as the mean and standard deviation across seeds.Methods follow the official DexJoCo environment settings and task-success criteria.Source: F SIMULATION EXPERIMENT DETAILS · p. 20

Inference configuration

  • Inference runs policy and control at 30 Hz, generates 30-step action chunks, and uses an eight-step rolling-replanning window with strict RTC (PI-VJP) execution.The configuration also applies previous-chunk guidance, exponential prefix attention, a maximum guidance weight of 5.0, and a 40 ms latency safety margin.Source: F SIMULATION EXPERIMENT DETAILS · p. 20

G REAL-WORLD EXPERIMENT DETAILS

The real-world evaluation tests three dexterous tasks as ordered, three-stage sequences, measuring both stage-level performance and completion of the full sequence. ARPG and π0.5 use matched demonstrations, initial-state distributions, and success criteria.

Task setup

  • Hammer handover, coffee-bean pouring, and cabinet retrieval each require three stages in order, assessing local manipulation and long-horizon completion.The procedures include transferring a hammer from right to left, pouring beans from the left cup into the right, and retrieving a cabinet object with the left hand.Source: G REAL-WORLD EXPERIMENT DETAILS · p. 21, Figure 6 · p. 22, Figure 7 · p. 22

Success criteria

  • A later stage counts as successful only after the preceding stage succeeds, and final task success requires completing all three stages in order.Stage success is the proportion of rollouts satisfying that stage’s criterion; final success is the proportion completing the full sequence.Source: G REAL-WORLD EXPERIMENT DETAILS · p. 21

Baseline setup

  • ARPG and π0.5 are compared using the same robot demonstrations, initial-state distribution, and task-success criteria.The comparison also uses the same three-view visual inputs; π0.5 is initialized from base weights and trained with mixed LoRA fine-tuning.Source: G REAL-WORLD EXPERIMENT DETAILS · p. 21

Task instructions

  • The task instructions specify the hand roles and action sequence for each of the three real-world tasks.For example, cabinet retrieval requires opening with the right hand, retrieving with the left, then closing with the right.Source: G REAL-WORLD EXPERIMENT DETAILS · p. 22

H EVALUATION DETAILS ON ROBOTICS SIMULATION

The evaluation reports task-level success rates across all 11 DexJoCo tasks, alongside macro-averages for single-arm, bimanual, and combined task groups. Results include three-seed variability when multiple seeds are used.

Reported results

  • The reported results cover all 11 DexJoCo tasks and include macro-average success rates for six single-arm tasks, five bimanual tasks, and all tasks.The supplied table caption identifies Table 9 as the full task-level results table, but does not provide its cell values.Source: H EVALUATION DETAILS ON ROBOTICS… · p. 22, Table 9 · p. 23

Reporting convention

  • For experiments with multiple random seeds, per-task results report mean success rate and standard deviation across three seeds.The table note specifies that values are success rates in percent and ± denotes standard deviation across three seeds when available.Source: H EVALUATION DETAILS ON ROBOTICS… · pp. 22–23

Related papers

Loading 2610.34453v1…