PaperlayerView exact revision on arXiv

Source-linked AI summary

JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation

Chuyang Xiao, Peilin Meng, David Held

arXiv:2609.25322v2cs.RO

TL;DR

Bimanual policies often lack an explicit model of how actions change shared scene geometry, despite the importance of those changes to coordination. JAMB jointly denoises bimanual actions and future 3D point tracks in a shared spatiotemporal representation. It outperforms the strongest baseline by 23.9 percentage points across 16 simulated tasks and improves generalization over evaluated baselines.

  • Problem

    Most diffusion policies reconstruct actions from current observations without explicitly connecting them to resulting object motion or scene changes, limiting explicit reasoning about shared scene effects.

    Source: §I · p. 1

  • Method

    JAMB jointly denoises bimanual actions and future 3D point tracks, grounding multimodal tokens in shared world-space and trajectory time.

    Source: §I · p. 2

  • Results

    JAMB outperforms the strongest baseline by 23.9 percentage points across 16 simulated tasks and shows stronger generalization to cluttered scenes and out-of-distribution backgrounds.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results support future 3D tracks as a practical intermediate representation for coordinated robot action generation.

    Source: §VII · p. 7

    The method uses a single camera view and fixed-grid track queries, while real-world track supervision depends on point-tracking and depth-estimation accuracy.

    Source: §VI · p. 7

Abstract

from arXiv · show

Coordinated bimanual manipulation is challenging because the motion of either arm can alter the shared 3D scene and thereby affect the other arm. Yet most diffusion policies generate actions without explicitly modeling these future geometric consequences, while predictive variants typically use future state only as auxiliary supervision or fixed conditioning. We address this limitation by proposing JAMB, a diffusion policy that jointly denoises bimanual actions and future 3D point tracks. By allowing action and track hypotheses to evolve together within a shared Transformer, each can inform and refine the other throughout denoising. We further ground multimodal representations in a shared spatiotemporal coordinate system to facilitate geometry-aware interaction during joint denoising. We evaluate JAMB on diverse bimanual manipulation tasks in RoboTwin 2.0 and on a real-world robot, comparing it with action-only policies and alternative future-prediction approaches spanning different state representations and learning objectives. Across 16 simulation tasks, JAMB achieves an average success rate of 83.4%, outperforming the strongest baseline by 23.9 percentage points. On three real-world tasks, it outperforms the action-only and auxiliary geometry prediction methods by 50.0 and 21.2 percentage points, respectively. Beyond these performance gains, JAMB shows stronger generalization to cluttered scenes and out-of-distribution backgrounds than the evaluated baselines. Together, these results demonstrate the effectiveness of our joint action-motion modeling framework for coordinated bimanual manipulation. Our project website is available at https://jam-bimanual.github.io/

I. INTRODUCTION

Bimanual policies must account for how coordinated arm motions change shared objects and the scene over time. JAMB jointly denoises actions and future 3D point tracks, grounding multimodal tokens in shared space and time; evaluations report gains over baselines.

I. INTRODUCTION

  • Because either arm can alter object pose and the other arm’s feasible motion, bimanual policies must reason about coordinated effects on the scene over time.Most diffusion policies reconstruct demonstrated actions from current observations without explicitly connecting actions to resulting object motion or scene changes.Source: §I · p. 1
  • JAMB jointly denoises bimanual actions and future 3D point tracks so evolving action and motion predictions can mutually refine each other.Its multimodal tokens share a coordinate system based on world-space positions and trajectory time.Source: §I · p. 2
  • JAMB outperforms the strongest baseline by 23.9 percentage points across 16 simulated tasks and shows stronger generalization to cluttered scenes and out-of-distribution backgrounds.On three real-world tasks, it also achieves consistent improvements.Source: §I · p. 2

A. Bimanual Visuomotor Policy Learning

  • Prior bimanual visuomotor approaches primarily improve coordination at the action level, while JAMB models induced scene outcomes through jointly predicted future 3D point tracks.Point tracks preserve motion over time, unlike endpoint geometry prediction.Source: §I · p. 1, A. Bimanual Visuomotor Policy Learning · p. 2

B. Future Prediction for Robot Policy

Future-prediction policies use visual, geometric, or motion representations, each with limitations for control. JAMB treats future 3D point tracks and bimanual actions as jointly refined diffusion variables, with track queries aligned to visual patches.

B. Future Prediction for Robot Policy

  • Visual futures can model irrelevant appearance, dense geometry can spend capacity on static regions, and future scene states do not explicitly capture motion over time.These limitations motivate motion-focused representations for policy learning.Source: B. Future Prediction for Robot Policy · p. 2
  • Prior track-and-action methods use 2D image-space tracks, while JAMB jointly updates explicit future 3D motion and actions throughout generation.Each track query is paired with its aligned visual patch, and spatiotemporal encoding grounds tokens in a shared 3D coordinate frame.Source: B. Future Prediction for Robot Policy · p. 2

III. PROBLEM FORMULATION

  • JAMB associates the visual observation with a grid of patch-aligned 3D query points whose future positions represent scene motion.Each query corresponds to a visual patch, providing aligned visual and motion representations.Source: §III · pp. 2–3
  • The policy learns a conditional joint distribution so action and future-motion predictions can be generated and refined together.The overview places noisy action and track tokens with arm-state and visual tokens in one sequence for joint denoising.Source: §III · p. 3, Figure 2 · p. 3

IV. METHOD

JAMB represents future scene motion with patch-aligned 3D point tracks and jointly denoises those tracks with bimanual actions. Shared self-attention and spatiotemporal grounding let the evolving predictions interact during denoising.

A. 3D Track Construction

  • Each visual patch initializes a query lifted into 3D, producing scene-point tracks over the future horizon that preserve point correspondence with a lower-dimensional representation.Tracks are represented as displacements from each query's current 3D position.Source: A. 3D Track Construction · p. 3

B. Joint Action–Motion Diffusion

  • A shared DiT processes fused visual–track tokens, separate arm-state tokens, noisy action tokens, and a diffusion-timestep token.Each track embedding is concatenated with its aligned visual feature before projection into a fused token.Source: B. Joint Action–Motion Diffusion · p. 3
  • Full self-attention lets action and future-motion hypotheses mutually refine one another throughout denoising.Source: B. Joint Action–Motion Diffusion · p. 3
  • 4D RoPE encodes relative world-space positions and trajectory time, anchoring visual–track and arm-state tokens at current positions and action tokens at future steps.Action-token anchors interpolate from current end-effector positions toward action-derived positions as denoising progresses.Source: B. Joint Action–Motion Diffusion · p. 3

C. Training and Inference

Training jointly adds noise to normalized action and track targets and uses a shared DiT to predict their targets. At inference, both are denoised from noise, but only actions are executed.

C. Training and Inference

  • Actions are min–max normalized to [−1, 1] and track displacements are normalized before noise is added to both targets at a shared diffusion timestep.Source: C. Training and Inference · p. 4
  • The shared DiT predicts action and track targets from conditioning tokens and noisy action and track tokens in normalized spaces.Source: C. Training and Inference · p. 4
  • The training objective sums action mean squared error with a track loss weighted by λ_track.The track loss averages over spatial coordinates, sums across the prediction horizon, and takes a motion-weighted average over track queries.Source: C. Training and Inference · p. 4
  • At inference, normalized actions and tracks start independently from Gaussian noise and are updated jointly; only predicted actions are executed.Generated tracks are retained for analysis and visualization.Source: C. Training and Inference · p. 4

V. EXPERIMENTS

The experiments assess future 3D track prediction, joint action–motion diffusion, spatiotemporal grounding, and generalization across simulation and real-world bimanual tasks. The setup compares action-only and future-prediction methods under matched training and evaluation conditions where applicable.

V. EXPERIMENTS

  • The evaluation examines whether future 3D point tracks improve policy performance, how joint denoising and spatiotemporal grounding affect performance, and whether JAMB generalizes to unseen backgrounds and object appearances.Source: §V · p. 4

A. Experimental Setup

  • The simulation benchmark uses 16 RoboTwin 2.0 tasks: eight Sync-bimanual tasks requiring simultaneous coordination and eight Seq-coordinate tasks requiring sequential interaction.Tables I and II report per-task results for the two categories.Source: A. Experimental Setup · p. 4, Table I · p. 5, Table II · p. 5
  • Real-world evaluation covers Store Block, Stack Basin, and Place Duck Box using two xArm manipulators and a fixed overhead ZED Mini stereo camera.Teleoperation demonstrations use a Meta VR headset; the tasks cover complementary forms of coordination.Source: A. Experimental Setup · p. 4
  • Compared methods span action-only policies, future-state prediction, and trajectory-guided policies, with matched demonstrations, observation modalities, action representations, and initial-state distributions whenever applicable.Real-world comparisons use DP3 and GAP; simulation policies train on 100 expert demonstrations per task and use three evaluation seeds with 50 rollouts each.Source: A. Experimental Setup · p. 5

B. Simulation Policy Results

JAMB performs strongly across simulated and real-world bimanual tasks, with temporally aligned 3D tracks providing an intermediate representation of motion. Its results exceed evaluated baselines, especially where coordination depends on motion through execution.

B. Simulation Policy Results

  • 83.4% overall success across 16 simulated tasks, 23.9 percentage points above the strongest baseline.JAMB averages 85.2% on Sync-bimanual tasks and 81.6% on Seq-coordinate tasks.Source: B. Simulation Policy Results · p. 5
  • Compared with action-only policies DP and DP3 [17], JAMB additionally represents how the observed scene is expected to evolve.Fig. 4 compares predicted simulation tracks with ground-truth tracks; Fig. 5 shows predicted tracks over time for three real-world tasks.Source: B. Simulation Policy Results · p. 5, Figure 4 · p. 6, Figure 5 · p. 6
  • Unlike future visual observations or endpoint geometry, JAMB represents motion between current and future states with temporally aligned 3D point tracks.Its stronger performance on coordination-intensive and multi-stage tasks suggests this intermediate motion structure provides useful guidance.Source: B. Simulation Policy Results · pp. 5–6

C. Real-World Bimanual Evaluation

  • JAMB averages 85.6% across three real-world tasks, outperforming GAP by 21.2 percentage points and achieving the best result on each task.On Place Duck Box, JAMB scores 80.0% versus 33.3% for GAP; the gap is small on Stack Basin.Source: Table III · p. 6, C. Real-World Bimanual Evaluation · p. 6

D. Ablation Study

Ablations favor joint action–track denoising and spatiotemporal grounding, while harder visual conditions reveal a generalization advantage. The method’s current scope is bounded by its single-view setup and dependence on track and depth accuracy.

D. Ablation Study

  • 81.7% average success for joint action–track denoising, versus 76.3% for track-conditioned diffusion; removing 4D RoPE lowers success to 74.1%.Track regression raises success from 61.4% to 73.9%; joint denoising benefits synchronized and sequential tasks.Source: D. Ablation Study · p. 6, Table IV · p. 6
  • 7.6/10.0 mm ADE/FDE and 81.7% average success are achieved by the full model; joint denoising improves success by 5.4 percentage points over track-conditioned diffusion.The corresponding track-conditioned ADE/FDE is 9.9/12.5 mm.Source: D. Ablation Study · p. 6, Table V · p. 7
  • 17.9% average success in Easy-to-Hard evaluation without fine-tuning exceeds GAP’s 4.3%; full-horizon 3D tracks also outperform full-horizon 2D tracks and endpoint 3D geometry.The alternative representations score 9.3% and 10.3%, respectively.Source: D. Ablation Study · p. 7, Table VI · p. 7, Table VII · p. 7

VI. LIMITATIONS AND FUTURE WORK

  • The method currently uses one camera view and a fixed grid of track queries, while real-world track supervision depends on accurate point tracking and depth estimation.Future work includes multi-view fusion, adaptive query selection, robustness to noisy supervision, and force or contact information.Source: §VI · p. 7

VII. CONCLUSION

  • The conclusion presents future 3D tracks as a practical intermediate representation for coordinated robot action generation, with gains especially on coordination-intensive and multi-stage tasks.The ablations highlight joint action–motion generation and spatiotemporal positional encoding.Source: §VII · p. 7

APPENDIX

The appendix reports additional Dominant-select evaluation, defines moving-patch track-error metrics, and compares ablation variants. The full model combines the strongest reported track accuracy and policy success, while ablations show that track accuracy alone does not explain policy performance.

A. Additional Experimental Results

  • The additional Dominant-select evaluation covers tasks requiring selection of the appropriate arm, with results reported in Table VIII.The supplied passage does not state the table’s outcome values.Source: A. Additional Experimental Results · p. 9

B. Future Track Prediction Metrics

  • ADE averages Euclidean displacement error across the prediction horizon, whereas FDE measures error at the final step, using de-normalized displacements reported in millimeters.Both metrics are computed over moving patches whose ground-truth endpoint displacement exceeds 0.01 m.Source: B. Future Track Prediction Metrics · p. 9

C. Ablation Settings and Per-Task Results

  • The ablations compare action-only, track regression, track-conditioned action diffusion, and variants removing visual–track concatenation or 4D RoPE.The track-conditioned variant keeps predicted tracks fixed during action denoising; track regression does not feed predictions back as an evolving diffusion state.Source: C. Ablation Settings and Per-Task… · p. 9
  • 7.6/10.0 mm ADE/FDE and 81.7% average policy success are the full model’s best reported results across the ablations.Compared with track-conditioned action diffusion, it reduces ADE/FDE from 9.9/12.5 mm and improves success by 5.4 percentage points.Source: C. Ablation Settings and Per-Task… · p. 9
  • Track error does not consistently predict policy success: track regression has lower ADE/FDE than the variant without 4D RoPE, yet similar success rates.The comparisons do not isolate a causal effect of track accuracy on manipulation success.Source: C. Ablation Settings and Per-Task… · p. 9

D. Per-Task Easy-to-Hard Generalization Results

The appendix evaluates generalization from Easy to Hard scenes and compares additional future-prediction targets and representations. Full-horizon 3D tracks perform best in the reported visual-shift comparison, while success remains low overall.

D. Per-Task Easy-to-Hard Generalization Results

E. Additional Future Prediction Targets

  • 79.4% and 74.2% average success with future-image and geometry-feature prediction, respectively, versus 81.7% for the full model.Under the evaluated configuration, these additional targets provide no further performance benefit over joint action–track prediction alone.Source: E. Additional Future Prediction Targets · p. 10, Table XII · p. 11

F. Alternative Future Representations

  • The full-horizon 2D variant predicts pixel displacements, while the endpoint geometry variant predicts frozen Pi3 features with supervision in feature space.All compared future representations are jointly denoised with actions.Source: F. Alternative Future Representations · pp. 10–11
  • 81.2% success for endpoint 3D displacements, 81.7% for full-horizon 3D tracks, and 82.8% for endpoint 3D geometry.The three 3D prediction targets fall within 1.6 percentage points; the track-based formulation uses depth-derived spatial anchors without Pi3 feature extraction.Source: F. Alternative Future Representations · p. 11, Table XIII · p. 11
  • 79.5% average success with 4D RoPE versus 76.1% without it for full-horizon 2D tracks.This comparison supports the value of spatiotemporal grounding even when motion is predicted in image space.Source: F. Alternative Future Representations · pp. 10–11, Table XIII · p. 11
  • 17.9% Easy-to-Hard success for full-horizon 3D tracks, versus 12.2% for endpoint displacements and 10.3% for endpoint geometry.The proposed explanations involving intermediate-motion supervision and sensitivity to background or clutter are hypotheses, not isolated mechanisms.Source: F. Alternative Future Representations · p. 11

G. Additional Implementation Details

The real-world setup uses a single cropped head-camera view and patch-level track queries. The policy uses a four-block DiT and AdamW optimization.

G. Additional Implementation Details

  • The policy uses a single head-camera RGB view cropped to 168 × 322 pixels, with a 12 × 23 patch grid and 276 track queries.The action and track prediction horizons are both 16 steps.Source: 1) Architecture and Optimization · p. 11
  • JAMB uses a DiT with four Transformer blocks, hidden dimension 1024, and eight attention heads.The model is optimized with AdamW at learning rate 10^-4, weight decay 10^-6, and batch size 64.Source: 1) Architecture and Optimization · p. 11
  • Optimization uses 500 warm-up steps followed by cosine learning-rate decay.The AdamW schedule uses β1 = 0.9 and β2 = 0.999.Source: 1) Architecture and Optimization · p. 11

2) Diffusion and Loss Hyperparameters:

The implementation specifies diffusion schedules and track-loss weighting, as well as how track targets are validated and normalized. It also uses final-epoch checkpoints for evaluation.

2) Diffusion and Loss Hyperparameters:

  • Both action and track branches use the same squared-cosine noise schedule, with 100 training diffusion timesteps and 10 DDIM inference steps.The track-loss hyperparameters are λtrack = 10^-4, wmin = 0, and τ = 0.01 m; the loss is not averaged over the prediction horizon.Source: 2) Diffusion and Loss Hyperparameters · p. 12

3) Track Validity and Invalid-Query Supervision:

  • Track-query validity is determined at the anchor frame and propagated across the horizon, without reevaluating future visibility or occlusion.Queries with invalid depth or untracked segmentation IDs receive zero displacement targets; invalid queries remain in the movement-weighted loss with weak zero-displacement supervision.Source: 3) Track Validity and Invalid-Query… · p. 12

4) Training Duration and Checkpoint Selection:

5) 4D-RoPE Implementation:

4D RoPE encodes relative spatial and temporal relationships between multimodal tokens. Token coordinates, axis-specific rotations, and noise-dependent action anchors ground attention in the shared world frame and trajectory time.

5) 4D-RoPE Implementation:

  • Each attention head’s 128 dimensions are split into four 32-dimensional groups for x, y, z, and time; rotations use frequency base 100, spatial scale 0.01 m, and temporal scale 16.The rotations encode relative coordinate offsets in query–key dot products.Source: 5) 4D-RoPE Implementation · p. 12
  • Visual patches, track queries, and arm-state tokens use current 3D positions and time index 0; action tokens use future-step indices and noisy-action-derived positions.Special tokens, including CLS and diffusion-timestep tokens, are excluded from rotary encoding.Source: 5) 4D-RoPE Implementation · p. 12
  • Each track token represents one query point’s full future trajectory, with its temporal coordinate marking the current reference frame.Ordered future displacements are encoded in the track embedding, while temporally indexed action tokens interact through attention.Source: 5) 4D-RoPE Implementation · p. 12
  • Action-token spatial anchors combine the arm’s current world-frame end-effector position with the absolute position derived from its normalized noisy action.Both arms’ positions are expressed in the same world frame.Source: 5) 4D-RoPE Implementation · pp. 12–13
  • The scheduler timestep k is indexed on the training diffusion grid, not by the DDIM iteration; inference uses scheduler-selected timesteps to compute ρk..Source: 5) 4D-RoPE Implementation · p. 13
  • At high noise, anchors stay close to current end-effector positions; as denoising progresses, they increasingly follow positions from the evolving action sequence.Anchors are recomputed at every denoising step and use the same construction in training and inference, without clean future actions.Source: 5) 4D-RoPE Implementation · p. 13
Loading 2609.25322v2…