Source-linked AI summary
Coordi-pi: Coordination-Aware Dual-Hand Action Representation for Bimanual Dexterous Manipulation
Yiming Meng, Yanming Shao, Weijie Shen, Kunyang Lin, Jiayu Sun, Ping Luo, Yao Mu
TL;DR
Bimanual dexterous manipulation creates a high-dimensional, structured action space, while common concatenated representations leave cross-hand interactions implicit. Coordi-π interleaves side-specific action tokens and encodes hand identity and temporal synchronization; it improves results over π0.5 across DexJoCo settings and four real-world tasks.
Problem
Bimanual dexterous manipulation requires coordinating two high-DoF arm–hand systems, while common concatenated action representations leave cross-hand structure implicit.
Source: §I · p. 1Method
Coordi-π interleaves right- and left-side arm–hand action tokens and uses hand-identity embeddings and factorized RoPE to encode side identity and temporal synchronization.
Source: §I · p. 2Results
Coordi-π achieves state-of-the-art performance across five DexJoCo tasks and outperforms π0.5 across four real-world tasks.
Source: §I · p. 2Takeaways & Limitations
Across the reported simulated and real-world tasks, Coordi-π consistently improves over π0.5 for bimanual dexterous manipulation.
Source: §V · pp. 7–8
Abstract
from arXiv · showhide
Vision-language-action (VLA) models have shown strong potential for general-purpose robotic manipulation, but extending them to bimanual dexterous manipulation introduces a high-dimensional and structured action space. Existing bimanual VLA policies commonly concatenate right- and left-side arm--hand actions at each control timestep, leaving cross-hand interactions implicit within a single action vector. We introduce Coordi-$\pi$, a structured action representation for bimanual dexterous manipulation. Coordi-$\pi$ represents the right- and left-side arm--hand actions as individual action tokens and organizes them into an interleaved sequence, explicitly exposing the two sides within the action sequence. To explicitly encode hand identity and temporal synchronization in this interleaved representation, a learnable hand-identity embedding distinguishes the two sides, while sequence--control-timestep factorized rotary positional encoding (RoPE) separates sequence position from control timestep, allowing actions at the same control timestep to share a temporal coordinate while retaining distinct sequence positions. Across all five bimanual dexterous manipulation tasks in DexJoCo, Coordi-$\pi$ achieves state-of-the-art performance under both \textit{rand-obj} and \textit{rand-full}, improving the average success rate over $\pi_{0.5}$ from 31.9\% to 51.3\% and from 18.3\% to 32.3\%, respectively. Coordi-$\pi$ also consistently outperforms $\pi_{0.5}$ across four real-world bimanual dexterous manipulation tasks.
I. INTRODUCTION
Bimanual dexterous manipulation requires coordinating two high-DoF arm–hand systems, yet common action representations concatenate their actions and leave cross-hand structure implicit. Coordi-π interleaves side-specific action tokens and explicitly encodes hand identity and temporal synchronization, improving benchmark performance in simulation and real-world tasks.
I. INTRODUCTION
- Bimanual dexterous manipulation requires each side to control a high-DoF arm–hand system while coordinating across sides, increasing joint-action dimensionality and structural complexity.Source: §I · p. 1
- Conventional approaches concatenate right- and left-side actions at each timestep, leaving side identities and cross-hand structure implicit within one action vector.Source: §I · p. 1
- Interleaving makes bimanual structure explicit but breaks the one-to-one correspondence between sequence position and control timestep.Source: §I · p. 1
- Coordi-π achieves state-of-the-art results across five DexJoCo tasks, raising average success over π0.5 from 31.9% to 51.3% under rand-obj and from 18.3% to 32.3% under rand-full.It also outperforms π0.5 across all four real-world tasks.Source: §I · p. 2
- Coordi-π interleaves right- and left-side arm–hand actions as individual tokens, explicitly exposing both sides in the action sequence.Source: §I · pp. 1–2
- A learnable hand-identity embedding distinguishes the two sides, while factorized RoPE separates sequence position from control timestep to represent temporal synchronization.Actions at the same control timestep share a temporal coordinate while retaining distinct sequence positions.Source: §I · p. 2
II. RELATED WORK
Related work spans bimanual and dexterous manipulation systems, VLA policies, and action representations and sequence modeling. Coordi-π builds on π0.5 while focusing on structured action representation for bimanual dexterous control.
II. RELATED WORK
- Prior bimanual and dexterous manipulation work advances teleoperation, demonstration collection, policy learning, and evaluation, while Coordi-π focuses on action representation for bimanual dexterous VLA control.Examples include BiDex, DexMimicGen, Mobile ALOHA, ALOHA Unleashed, DexCap, InterACT, KStar Diffuser, RDT-1B, Dexora, and the DexJoCo benchmark.Source: §II · p. 2
- VLA research ranges from multimodal policies and pretrained generalist models to continuous-action methods, and Coordi-π builds on this line while targeting bimanual dexterous control.The passage discusses PaLM-E, VIMA, RT-2, OpenVLA, π0, π0.5, and DexVLA.Source: §II · p. 2
A. Overview
- Action-representation work covers action chunks, diffusion trajectories, continuous-action tokenization, spatial structure, and interleaved formulations; Coordi-π combines interleaved arm–hand tokens with explicit hand identity and control-timestep structure.Source: §II · p. 2, A. Overview · p. 2
B. Problem Formulation
Coordi-π restructures bimanual actions as separate, interleaved right- and left-side tokens rather than keeping both sides in one action vector. The representation preserves joint commands at each control timestep while making side identity and synchronization explicit.
B. Problem Formulation
- The policy receives visual observations, language, and robot state, then predicts an H-step bimanual action chunk.Each bimanual action contains right- and left-side arm–hand commands.Source: B. Problem Formulation · pp. 2–3
- The figure shows actions split into interleaved side tokens, identity embeddings, and factorized RoPE before paired outputs become joint commands.The action horizon remains unchanged.Source: Figure 2 · p. 3
- Conventional bimanual actions place the two sides in different feature dimensions of the same action vector.This representation preserves the joint action at each timestep, but leaves side identity and cross-hand structure implicit.Source: B. Problem Formulation · p. 3, C. Interleaved Bimanual Action… · p. 3
C. Interleaved Bimanual Action Representation
- The representation moves bimanual structure from feature dimensions to the token dimension by assigning each side a separate action token.Right- and left-side actions at the same timestep are neighboring tokens in the interleaved sequence.Source: C. Interleaved Bimanual Action… · p. 3
- Each H-step bimanual chunk becomes 2H side-specific tokens, with right- and left-side actions paired at each control timestep.The paired tokens are decoded as the joint action for that timestep.Source: C. Interleaved Bimanual Action… · p. 3
- Interleaving makes the two arm–hand systems explicit, but sequence position no longer coincides with control timestep.A learnable hand-identity embedding and factorized RoPE address side identity and temporal synchronization, respectively.Source: C. Interleaved Bimanual Action… · p. 3
D. Hand-Identity Embedding
Coordi-π adds learnable hand-identity embeddings and factorized RoPE to the interleaved action tokens. These components distinguish the two sides and separately encode sequence order and control-time synchronization.
D. Hand-Identity Embedding
- A learnable hand-identity embedding identifies each action token’s side while right- and left-side tokens share the same input and output projections.Each side’s embedding is shared across its tokens in the action chunk, and both embeddings are initialized to zero.Source: D. Hand-Identity Embedding · p. 3
E. Sequence–Control-Timestep Factorized RoPE
- Factorized RoPE distinguishes sequence position from control timestep, assigning synchronized right- and left-side tokens different sequence positions but the same timestep coordinate.For an interleaved chunk, sequence positions run from 0 to 2H−1 while timestep coordinates repeat for each paired action.Source: E. Sequence–Control-Timestep… · pp. 3–4
- Adjacent token pairs can represent either synchronized actions or actions at consecutive timesteps, so sequence offset alone does not encode their temporal relation.For paired actions, Δs = 1 and Δτ = 0; for consecutive-timestep actions, Δs = 1 and Δτ = 1.Source: E. Sequence–Control-Timestep… · p. 4
- Factorized RoPE applies separate positional encodings to action-token attention while context-token attention retains the original positional encoding.The sequence-position coordinate preserves interleaved ordering, while the shared control-timestep coordinate represents synchronization; paired tokens are decoded into a joint command.Source: E. Sequence–Control-Timestep… · p. 4
F. Training and Action Decoding
Coordi-π retains π0.5’s flow-matching objective while applying the interleaved representation, hand-identity embeddings, and factorized RoPE to action chunks. At inference, it pairs the generated side-specific tokens by control timestep to reconstruct joint bimanual actions.
F. Training and Action Decoding
- Coordi-π retains the flow-matching objective of π0.5 while training on noisy bimanual action chunks in interleaved token order.Training adds the corresponding hand-identity embeddings and applies factorized RoPE using sequence-position and control-timestep coordinates.Source: F. Training and Action Decoding · p. 4
- The control-timestep coordinate indexes temporal structure within the action chunk and is independent of continuous flow-matching time.Source: F. Training and Action Decoding · p. 4
- At inference, generated side-specific tokens are de-interleaved and paired by shared control timestep to reconstruct each joint bimanual action.Source: F. Training and Action Decoding · p. 4
IV. EXPERIMENTS
The experiments assess task performance, component contributions, coordination, and robustness across randomized environments and real-world evaluation.
IV. EXPERIMENTS
- The experiments assess whether Coordi-π improves task success, what the embedding and factorized RoPE contribute, whether coordination improves, and whether gains persist under stronger randomization and in real-world tasks.Source: §IV · p. 4
A. Experimental Setup
The evaluation covers five DexJoCo tasks under two randomization settings and four real-world tasks. Coordi-π leads the DexJoCo benchmark, with the largest reported gains on Hanoi and Unlock iPad, while Assembly remains difficult.
A. Experimental Setup
- The benchmark includes Hanoi, Photograph, Unlock iPad, Assembly, and Microwave, evaluated under rand-obj and rand-full.rand-obj varies object placement and table height; rand-full additionally varies camera poses, illumination, and tabletop textures.Source: A. Experimental Setup · p. 4
- The four real-world evaluation tasks are Photograph, Hanoi, Assembly, and Microwave.The real-world task set covers object support and operation, sequential cross-hand interaction, contact-rich assembly, and coordinated long-horizon execution.Source: A. Experimental Setup · p. 4
- The policy receives three 224 × 224 RGB views, a language instruction, and a 46-D proprioceptive state, with an action horizon of H = 30.Each side contributes a 22-D arm–hand action, forming a 44-D bimanual command per timestep; training uses 40k iterations and evaluation uses 150 episodes per method-task-setting combination.Source: A. Experimental Setup · pp. 4–5
B. Main Results on DexJoCo
- 51.3% average success under rand-obj exceeds DP-T at 33.5% and π0.5 at 31.9%; Coordi-π is best or tied-best on all five tasks.This is a 19.4 percentage-point improvement over π0.5.Source: B. Main Results on DexJoCo · p. 5, Table I · p. 5
- Under rand-obj, Hanoi improves from 15.3% with π0.5 to 48.7% with Coordi-π, and Unlock iPad improves from 12.0% to 52.0%.Both tasks require complementary hand roles and ordered cross-hand interactions.Source: B. Main Results on DexJoCo · p. 5
- Assembly remains difficult, with Coordi-π reaching only 7.3%; precise contact and insertion remain bottlenecks beyond action representation.Coordi-π also improves over π0.5 on Microwave, but by a smaller margin than on Hanoi and Unlock iPad.Source: B. Main Results on DexJoCo · p. 5
C. Ablation Study
Ablations show that hand-identity embedding and factorized RoPE complement the interleaved action representation, while their individual effects vary by task. Coordination analysis finds higher average coordination scores with Coordi-π, but shows that coordination criteria alone do not guarantee task completion.
C. Ablation Study
- Combining hand-identity embedding and factorized RoPE raises average success to 51.3%, an 11.6 percentage-point gain over the interleaved baseline at 39.7%.Either component individually improves average performance, but neither alone accounts for the full-model improvement.Source: C. Ablation Study · p. 5, Table II · p. 5
- The largest gains over the interleaved baseline occur on Unlock iPad, Photograph, and Hanoi, which involve sustained role differentiation or ordered cross-hand interactions.All variants use the interleaved action representation and share training data, optimization budget, and evaluation protocol.Source: C. Ablation Study · p. 5
- On Microwave, neither component alone improves over interleaving, whereas their combination performs best, showing task-dependent component effects.The contribution of each component cannot be understood independently of the full representation.Source: C. Ablation Study · p. 5
D. Bimanual Coordination Analysis
- Coordi-π has the highest average BCS across five tasks and the best BCS on four; task-specific role and temporal criteria define the score.BCS is the equal-weighted average of Role Coordination Score and Temporal Coordination Score; support-role criteria include stable contact by the non-operating hand.Source: D. Bimanual Coordination Analysis · p. 6
- In Hanoi, the interleaved baseline has higher BCS than Coordi-π (89.75 vs. 86.32) but lower success (37.3% vs. 48.7%).A rollout can satisfy the prescribed coordination pattern yet fail from execution errors or task conditions not captured by BCS.Source: D. Bimanual Coordination Analysis · p. 6
E. Performance under Full Randomization
Under rand-full, Coordi-π outperforms π0.5, though absolute success remains lower than under rand-obj and Assembly remains difficult.
E. Performance under Full Randomization
- 32.3% versus 18.3% average success rate gives Coordi-π a 14.0 percentage-point advantage over π0.5 under rand-full.The advantage persists under stronger visual and environment variation.Source: E. Performance under Full Randomization · p. 6
- Gains remain particularly clear on Unlock iPad and Hanoi when perception becomes more variable.The passage presents this as evidence that the structured action representation retains benefits under rand-full.Source: E. Performance under Full Randomization · p. 7
- Absolute success rates are substantially lower than under rand-obj, and Assembly remains particularly challenging under rand-full.Coordi-π achieves 2.0% success on Assembly, while π0.5 records no successful trials; remaining failures may also depend on perception and precise interaction.Source: E. Performance under Full Randomization · p. 7
F. Real-World Experiments
On four real-world bimanual tasks, Coordi-π achieves a higher average success rate than π0.5, with gains across all tasks. The comparison uses matched demonstrations, training budgets, initialization ranges, and success criteria.
F. Real-World Experiments
- 75.0% average success rate (60/80 trials) for Coordi-π exceeds π0.5’s 61.3% (49/80) across four real-world tasks.Both methods used the same demonstrations and training budget; each was evaluated for 20 trials per task.Source: F. Real-World Experiments · p. 7, Figure 6 · p. 7
- Improvement is consistent across all four tasks, with the largest gains on Photograph and Assembly.The four tasks are Photograph, Hanoi, Assembly, and Microwave on a real-world dual-arm dexterous platform.Source: F. Real-World Experiments · p. 7
- On real-world Assembly, Coordi-π improves success from 30% to 50%.The paper cautions that simulation and real-world absolute success rates are not directly comparable because data and execution conditions differ.Source: F. Real-World Experiments · p. 7
V. CONCLUSION
Coordi-π structures bimanual actions using interleaved hand tokens and factorized positional encoding. It improves performance across simulated and real-world tasks, while contact-rich manipulation remains difficult.
V. CONCLUSION
- Coordi-π interleaves right- and left-side arm–hand tokens, represents hand identity, and separates sequence position from control timestep through factorized RoPE.The representation is designed for bimanual dexterous manipulation.Source: §V · p. 7
- Gains are most pronounced on tasks with complementary hand roles and ordered cross-hand interactions, and generally accompany better bimanual coordination.The conclusion reports this pattern across the ablation and coordination analyses.Source: §V · p. 8
- Precise contact-rich manipulation remains difficult, particularly under stronger randomization.The authors identify perception and precise interaction as complementary challenges for future work.Source: §V · p. 8