PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributed Dexterous Manipulation with Spatially Conditioned Multi-Agent Transformers

Sarvesh Patil

arXiv:2609.06930v1cs.ROcs.LG

TL;DR

Dexterous manipulation of varied objects remains challenging, motivating distributed systems that coordinate actuators. This paper learns planar DDM policies with spatially conditioned multi-agent transformers and action selection; its discrete-penalty policy uses up to 65% fewer active robots while maintaining ~1.5 cm average trajectory tracking error.

  • Problem

    Dexterous manipulation of a wide variety of objects remains a fundamental challenge, and distributed manipulation offers a promising approach.

    Source: §1 · p. 1

  • Method

    The framework combines spatially conditioned Multi-Agent Transformers with behavior-cloning pretraining, reinforcement-learning fine-tuning, and action selection for robot-use trade-offs.

    Source: §1 · p. 2

  • Results

    Up to 65% fewer active robots while maintaining ∼1.5 cm average trajectory tracking error is reported for MATBC-FT-DEC.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Discrete-penalty action selection provides a policy that balances manipulation performance with robot-use efficiency compared with continuous penalties.

    Source: §6 · p. 8, §7 · p. 9

    Evidence is limited to planar tasks on delta arrays; 3D policies remain unexplored, and fine-tuning still requires expert demonstrations.

    Source: §8 · p. 10

Abstract

from arXiv · show

Distributed Dexterous Manipulation (DDM) is a novel paradigm that presents significant control challenges due to high action-space redundancy, inter-robot cooperation, and dynamic object-robot interactions. This paper introduces a framework based on spatially conditioned Multi-Agent Transformers (MATs) to efficiently learn robust control policies for a DDM system grounded in an array of 64 soft delta robots arranged in an 8x8 grid. Our three core contributions are: (i) an MAT with adaptive layer norm for compute efficiency, (ii) spatial contrastive embeddings to ground transformer embeddings in the spatial configuration of the robots, and (iii) an MAT-based behavior cloning method fine-tuned using Soft Actor Critic. We also propose an action selection formulation to analyze the trade-off between task performance and the number of robots utilized. Our experiments show that MATs iteratively refine their actions through the stacked attention blocks. This further informs the benefit of spatial conditioning in transformers to learn DDM policies. We demonstrate long-horizon planar manipulation tasks with objects of various geometries in simulation and real-world. Finally, we show how action selection mitigates robot maintenance by reducing wear and tear due to inter-robot collisions while maintaining the ability to manipulate objects along various trajectories in the real-world, achieving an average error of ~1.5 cm, while using ~65% fewer robots.

1 Introduction

Distributed dexterous manipulation uses cooperating robots to move objects, but delta arrays pose challenges from a 192-DoF action space, changing neighborhoods, and sample-efficient learning. The paper addresses these challenges with spatially conditioned Multi-Agent Transformers, spatial contrastive embeddings, and behavior-cloning pretraining followed by reinforcement-learning fine-tuning.

1 Introduction

  • Delta-array control combines a 192-DoF action space, multimodal dynamics, changing robot neighborhoods, and a need for sample-efficient learning.Earlier distributed systems using rotary or linear actuators do not cover the full 3-DoF capabilities needed for dexterous tasks.Source: §1 · p. 1
  • The proposed framework uses spatially conditioned MATs, with spatial contrastive embeddings grounding robot relationships in the array layout.MAT policies are pretrained by behavior cloning from a visual-servoing expert and fine-tuned with off-policy Soft Actor-Critic.Source: §1 · p. 2
  • The paper analyzes reward shaping for trading off task success against the number of active robots and examines iterative transformer message-passing.The contributions also include empirical analysis of attention-layer action refinement for cooperative manipulation.Source: §1 · p. 2

2 Related Work

Prior work studies distributed manipulation with varied actuator systems, imitation and reinforcement learning for dexterous control, and transformer-based multi-agent policies. Applying these approaches to fine-grained DDM remains an open area in the cited literature.

2 Related Work

  • Distributed manipulation spans air-jet arrays, actuated surfaces, workbenches, and delta arrays, which offer dense 3-DoF compliant robots for complex interactions.Other 3D cooperative-pushing systems are described as limited by end-effector dexterity and workspace constraints.Source: §2 · p. 2
  • BC can provide expert imitation and RL can learn through trial and error, but RL's interaction demands are problematic for real robots.BC pretraining followed by RL fine-tuning has been studied, while applying these methods to fine-grained DDM remains unexplored.Source: §2 · pp. 2–3
  • MARL commonly uses centralized training with decentralized execution, whereas transformer approaches can use centralized training and execution with variable-length inputs.The paper positions its methods as inducing implicit spatial reasoning for delta arrays and reports zero-shot simulation-to-real-world transfer.Source: §2 · p. 3

3 Problem Formulation and Preliminaries

The formulation uses a 64-robot delta array and models planar object manipulation as an MDP with changing neighborhoods of eligible robots. It specifies robot states and planar actions, with a separate action-selection mechanism for choosing active robots.

3.1 Delta Arrays for Distributed Dexterous Manipulation

  • The real-world delta array has 64 compliant robots with three linear actuators each, producing a 192-DoF action space across the array.Each robot has an approximately 2.5 cm XY workspace radius and 10 cm of Z-axis range; neighboring workspaces overlap.Source: §3.1 · p. 3
  • A bottom-up camera observes the scene through a glass plane, and MuJoCo models each robot as a floating capsule fingertip with real-world workspace constraints.The robots are densely arranged in a hexagonal grid with centers approximately 4.35 cm apart.Source: §3.1 · p. 3

3.2 Problem Definition

  • The task is formulated as an MDP with joint state and action spaces, transition and reward functions, and an objective of maximizing expected cumulative reward.Each robot has its own state space and action space.Source: §3.2 · p. 3
  • At each timestep, image processing pairs robots with object-boundary points and selects a neighborhood of robots whose workspaces overlap the boundary.Robots directly on top of the object or without workspace overlap are excluded.Source: §3.2 · p. 3
  • Each robot's state includes initial and goal object-boundary points and the robot's position.These components form a per-robot state space in R^7.Source: §3.2 · p. 3
  • Actions are 3D displacement vectors, but planar tasks use the XY components while the Z component selects which neighborhood robots are active.The selected set is a subset of the current robot neighborhood.Source: §3.2 · p. 3
  • Figure 2 depicts decoder-only MAT policy and Q-value networks, neighborhood-based positive and negative embeddings, AdaLN, and spatial contrastive embeddings.Its embedding diagram uses triplet margin loss to create pairwise conformity and contrast.Source: Figure 2 · p. 4

4 Methods

The method combines a spatially conditioned Multi-Agent Transformer with behavior-cloning pretraining and Soft Actor-Critic fine-tuning. Spatial Contrastive Embeddings encode robot layout, while action-selection rewards trade task performance against robot use.

4.1 Multi-Agent Transformers

  • The decoder-only Multi-Agent Transformer iteratively conditions policy actions on robot states and position embeddings, while its Q function processes concatenated states and actions to produce agent-specific values.AdaLN-Zero uses conditional scaling and shifting, with zero-initialized scaling and a learned residual scale for training efficiency and stability.Source: §4.1 · p. 4

4.2 Spatial Contrastive Embeddings

  • Spatial Contrastive Embeddings map robot indices to learned vectors that make physically adjacent robots similar and distant robots dissimilar, preserving spatial topology for dynamically changing active subsets.These embeddings condition AdaLN-Zero layers in the transformer; Figure 3 compares their robot-embedding similarities with RoPE.Source: §4.2 · pp. 4–5, Figure 3 · p. 5

4.3 Learning Phase 1: Pre-training via Behavior Cloning (MATBC)

  • Behavior-cloning pretraining uses visual-servoing demonstrations to initialize the MAT policy and a critic trained to predict immediate rewards from expert and random transitions.The critic omits Bellman updates during behavior cloning because the visual-servoing data are quasi-static.Source: §4.3 · p. 5

4.4 Learning Phase 2: Fine-tuning via Reinforcement Learning (MATBC-FT)

  • Soft Actor-Critic fine-tunes the pretrained actor and critic, using twin critics and an actor objective that combines expected returns with entropy.The minimum of the twin critics is used for target calculations.Source: §4.4 · p. 5
  • Action-selection rewards add discrete and continuous costs to encourage sparse robot selection, with weighting hyperparameters controlling the task-performance and resource-efficiency trade-off.Robots with high z actions do not interact with objects and can mitigate unnecessary collisions due to crowding.Source: §4.4 · p. 6

5 Experiments

The experiments evaluate learned policies on long-horizon closed-loop planar manipulation and examine position embeddings, attention mechanisms, and robot-selection trade-offs. Evaluation spans simulation and real hardware, measuring robot use, tracking error, and subgoal success.

5 Experiments

  • Evaluation uses quasi-static closed-loop rollouts across 20 subgoals for 10 objects, measuring success within three attempts and mean trajectory-tracking error in simulation and on real hardware.Figure 4 reports active-robot percentage, error per execution step, and subgoal success rate for hexagon, star, and trapezium objects.Source: §5 · p. 6, Figure 4 · p. 7
  • Position-embedding ablations compare SCE with LCRoPE, learned relative embeddings, and unpretrained integer embeddings through the MATBC-FT pipeline.LCRoPE masks attention activations beyond a distance threshold; learned relative embeddings assign angle vectors to neighbors.Source: §5 · p. 6
  • The attention-mechanism experiment compares AdaLN-Zero with standard self-attention and cross-attention for MATSAC training stability and convergence speed.Source: §5 · p. 6
  • The robot-selection experiment compares four reward-shaped MATBC-FT policies with visual servoing, MATBC, scratch-trained MATSAC, and MATBC-FT selecting all robots.The stated trade-off measures average active robots, trajectory tracking error, and subgoal success.Source: §5 · pp. 6–7

6 Results and Discussion

Results show that spatial embeddings and AdaLN-Zero affect MAT training, while behavior-cloning pretraining with RL fine-tuning improves manipulation performance. Discrete-penalty action selection reduces robot use while retaining trajectory tracking, though performance differs between simulation and real-world deployment.

6.1 Significance of Spatial Embeddings

  • SPE and RoPE fail to converge in scratch SAC training; after BC initialization, RoPE improves over scratch RL, while SCE supplies more contrastive conditioning than RoPE.The authors identify structured relative spatial bias across attention as crucial for centralized DDM policy learning.Source: §6 · p. 7

6.2 Efficiency of Attention Mechanisms

  • AdaLN-Zero trains in approximately half the wall-clock time of standard self- or cross-attention (∼23 versus ∼44 hours) and achieves higher final average reward.The reported training is also described as more stable.Source: §6 · p. 7

6.3 Trajectory Tracking Performance and Action Selection Tradeoff

  • MATBC-FT-DEC faithfully tracks long-horizon trajectories using ∼55% fewer robots than MATBC-FT-OG and ∼65% fewer than MATBC.Figure 5 covers the three hardest real-world trajectories with hexagon, trapezium, and star objects.Source: §6 · p. 8, Figure 5 · p. 8
  • RL fine-tuning improves on MATBC and MATSAC from scratch, while MATBC-FT has the lowest tracking error and highest success rate among full-robot methods.The reported comparison concerns methods using all available robots, Nt.Source: §6 · p. 8
  • MATBC-FT-OG improves average tracking error by ∼1.8% in simulation and ∼40.7% in the real world; discrete penalties better leverage the execution-cost and manipulation-performance tradeoff than continuous penalties.The authors attribute the smaller simulation improvement to deterministic physics causing performance gains to saturate.Source: §6 · p. 8

6.4 Qualitative Evaluation

  • Real-world soft-contact changes create out-of-distribution tracking errors, with success rates dropping more for MATBC-FT and MATBC-FT-OG than for the lower-robot MATBC-FT-DEC.The authors contrast this with higher performance in deterministic simulation.Source: §6 · p. 8

7 Conclusion

The paper presents an initial exploration of robust, sample-efficient DDM policies on delta arrays using spatially conditioned MATs. Its discrete-penalty MATBC-FT-DEC reduces robot use while maintaining reported tracking performance, and action selection mitigates wear and tear in real-world deployment.

7 Conclusion

  • MATBC-FT-DEC reduces active robot use by up to 65% while maintaining ∼1.5 cm average trajectory tracking error on a soft robot array.The conclusion also reports that action selection significantly mitigates robotic wear and tear in real-world deployment.Source: §7 · p. 9

8 Limitations

The paper identifies limits in task dimensionality, training data, hardware scope, and policy-learning approach. These boundaries leave broader DDM generalization and alternative training paradigms for future study.

8 Limitations

  • The studied task is limited to 2D planar manipulation; learning DDM policies in 3D remains unexplored.The authors note that expert demonstrations may be harder to obtain in 3D.Source: §8 · p. 10
  • Fine-tuning improves sample efficiency but still requires expert demonstrations for behavior-cloning pretraining, and scratch training does not match fine-tuned performance..Source: §8 · p. 10
  • Experiments use delta arrays only, limiting the demonstrated generalizability of the method to other DDM settings..Source: §8 · p. 10
  • The study uses centralized MAT policy learning; understanding differences from decentralized DDM methods remains future work..Source: §8 · p. 10
  • RL fine-tuning of multi-agent diffusion policies for highly redundant action spaces remains unexplored..Source: §8 · p. 10

9 Appendix

The appendix describes a visual-servoing pipeline for generating expert demonstrations and specifies the losses used for behavior-cloning pretraining and multi-agent Soft Actor Critic training.

9.1 Visual Servoing

  • Visual servoing identifies robots near an object's boundary, transforms boundary points to a target pose, and uses the resulting displacement as robot actions.Object segmentation supplies 2D boundary points; nearest-neighbor search selects robots whose workspaces overlap the boundary while their bases lie outside the initial polygon.Source: §9.1 · p. 15

9.2.1 Behavior Cloning (BC) Loss Functions

  • The behavior-cloning policy is trained by minimizing mean squared error between its predicted actions and expert actions for agents in the neighborhood.The initial critic is trained on the mixed dataset to predict the immediate global reward from the joint state and action.Source: §9.2.1 · pp. 15–16

9.2.2 Multi-Agent Soft Actor Critic Loss Functions

  • The Soft Actor Critic losses train two per-agent critics from replay data and an actor that optimizes expected returns and entropy.Target Q-values use target-network parameters, the discount factor, and entropy temperature; actions are sampled from a squashed Gaussian policy.Source: §9.2.2 · p. 16

9.3 Training Protocol and Hardware Specifications

This appendix section reports the training setup, ablations, trajectory evaluations, and object-level comparisons in simulation and the real world. The reported simulation trend is that convex objects are generally easier to manipulate than non-convex objects, particularly at larger rotations.

9.3 Training Protocol and Hardware Specifications

  • Training uses 10-layer MATs with 128-dimensional hidden layers, 500 visual-servoing demonstration episodes per object, and 3,000,000 environment steps for each MATSAC policy.The protocol also specifies 400 pretraining epochs, batch size 256, and learning rates of 3e-4.Source: §9.3 · p. 16

9.4 Ablation Results

  • AdaLN performs better than cross-attention and self-attention, while its SAC-fine-tuned transformer outperforms tabula-rasa training; SCEs learn spatial correlations that SPEs and RoPE fail to learn.The figure reports inference reward plots for attention-mechanism and position-embedding comparisons.Source: Figure 7 · p. 17

9.5 Position Embedding Ablations

  • Figure 8 presents performance plots for MATBC-FT-OG with the position-embedding ablations.The accompanying description identifies SPE, RoPE, LC-RoPE, LRE, and LE among the ablations discussed.Source: §9.4 · p. 17, Figure 8 · p. 17

9.6 Experiment Trajectories

  • The experiment trajectories are the ground-truth trajectories used as inference tasks for closed-loop DDM.Source: Figure 9 · p. 18

9.7 Object Level Performance - Real

  • Figure 10 compares methods on a real-world T-shaped object unseen during training, tracking active-robot percentage, execution-step error, and attempts to complete a 20-step trajectory.Source: Figure 10 · p. 18

9.8 Object Level Performance - Sim

  • Convex objects are generally easier for delta arrays to manipulate than non-convex objects, which are especially challenging at larger rotations.The comparison averages algorithm performance over all objects and trajectories in simulation.Source: §9.8 · p. 19, Figure 11 · p. 19

9.9 Results Per Trajectory

  • Figure 12 compares methods tracking manipulated objects across target trajectories using 10 objects and five simulation trials.Source: Figure 12 · p. 20

10 Spatial Contrastive Embeddings Ablations:

The appendix specifies spatially constrained and learned-relative-position variants for conditioning transformer attention on robot geometry. The learned variant maps robot relationships to learned angles that modify attention scores.

10.1 Locally Constrained RoPE (LC-RoPE)

  • LC-RoPE masks attention to robots within the 6 cm spatial locality radius after query-key multiplication.Source: §10.1 · p. 21

10.2 Learned Relative Embeddings (LRE)

  • LRE replaces RoPE's fixed-angle calculation with learned embeddings for 13 relative-neighborhood classes in the hexagonal array.The classes include a robot's own position, six immediate neighbors, and six intermediate-nearest neighbors.Source: §10.2 · p. 21

10.2.1 Learnable Angle Lookup Table

  • The LRE lookup table maps each spatial class to a vector of d/2 learned angles.Source: §10.2.1 · p. 21

10.2.2 Mapping and Angle Retrieval

  • A deterministic mapping converts the robots' 2D grid displacement into a relative-position class index used to retrieve learned angles.Source: §10.2.2 · p. 21

10.2.3 Relative Rotation Matrix Construction

  • Retrieved angles form a block-diagonal rotation matrix representing the relative spatial transformation between two robots.Source: §10.2.3 · p. 21

10.2.4 Application in Self-Attention

  • LRE applies the relative rotation to the key vector before its dot product with the query, so attention scores incorporate learned grid geometry.The rotation angles are learned through gradient descent rather than fixed by a mathematical formula.Source: §10.2.4 · p. 22
Loading 2609.06930v1…