CoRE: Learning Collaboration-Role Experts for Decentralized Collaborative Manipulation with One Policy

I. INTRODUCTION

• Single-policy decentralized collaboration asks each robot to predict its own actions from visual observations and proprioception, without task prompts, robot identities, role labels, or inter-robot messages.The shared policy must produce complementary behaviors as interactions unfold, although demonstrations specify actions rather than how to organize and select behavioral capabilities.Source: §I · p. 1

II. RELATED WORK

• Earlier collaborative-manipulation approaches include centralized control and decentralized methods with separate policies, while shared-policy approaches may use explicit coordination inputs or robot-identifying prompts [1],, [16].CoRE instead studies one shared policy using each robot’s local observations.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

• CoRE uses adaptable cross-attention experts for complementary behaviors and a local router that selects their contributions from appearance–geometry evidence and action queries.Action–expert alignment compares forced-route predictions against demonstrations under fixed inputs, using relative errors to guide expert selection without role labels.Source: §I · p. 2
• CoRE achieves the highest average performance among evaluated decentralized methods across three simulation benchmarks, including 84.2% success on RoboFactory.Physical experiments achieve 91.3% success across five tasks and examine robustness to partner delays and slowdowns.Source: §I · p. 2

II. RELATED WORK

• Related work covers visuomotor action prediction and spatial perception, collaborative control and learned coordination, and input-dependent routing in mixture-of-experts models –.CoRE’s routing guidance uses forced-route decoder predictions against demonstrations rather than a separate teacher or role labels.Source: §II · p. 2

III. CORE: COLLABORATION-ROLE EXPERTS

• Each robot’s local observation includes its available RGB and registered-depth camera views and proprioceptive state.The demonstrated action chunk contains H consecutive actions.Source: A. Problem Formulation · p. 2
• Training pools local observation–action pairs from each task-and-robot dataset to learn the shared policy.The policy independently predicts each robot’s action chunk.Source: A. Problem Formulation · p. 2
• All tasks and robots share policy parameters, but task and robot indices identify data sources rather than policy inputs.Each policy copy receives only its robot’s local observation.Source: A. Problem Formulation · p. 2

B. Overview

• Fused observations pass through an action decoder and action head to predict action chunks.Source: B. Overview · p. 3
• Separately trainable collaboration-role experts provide prediction paths whose contributions a local router allocates at each action position.Source: B. Overview · p. 3
• Action–expert alignment compares alternative expert-route predictions with demonstrated actions to supervise expert allocation alongside imitation learning.Source: B. Overview · p. 3

C. Local Spatial Evidence for Collaboration

• Fused RGB appearance and registered-depth features share an image-patch grid, preserving spatial correspondence to distinguish similar appearances that may require different motions.The approach assumes each robot’s local observations contain sufficient evidence for its demonstrated collaborative actions.Source: C. Local Spatial Evidence for… · p. 3

D. Learning Collaboration-Role Experts

• The policy learns collaboration-role experts with separately adjustable prediction paths, selects their contributions from local evidence, and uses demonstrated actions to supervise selection.Source: D. Learning Collaboration-Role Experts · p. 3

1) Constructing collaboration-role experts:

• Cross-attention experts learn different visual-to-action mappings for complementary roles, such as maintaining a grasp and reaching to receive an object.The shared action decoder predicts a chunk of future actions, and a shared action head converts decoder features into actions.Source: 1) Constructing collaboration-role… · p. 3

2) Choosing experts as collaboration unfolds:

• Each robot shares the expert pool, with routing conditioned on observations and action queries rather than predefined robot–expert or role–expert assignments.The position-specific routing weights are shared across layers and recomputed for each new observation.Source: 1) Constructing collaboration-role… · p. 3, Figure 3 · p. 4
• The local router scores experts using the observation, robot state, and action query, so expert combinations can vary across positions within a chunk.It retains the top-K experts and renormalizes their weights.Source: 2) Choosing experts as collaboration… · p. 3
• The locally routed expert structure, called action-role cross-attention (ARCA), adds its contribution to shared cross-attention before the feedforward block.Source: 2) Choosing experts as collaboration… · p. 3

• With decoder inputs fixed, CoRE activates each expert in turn across layers and prediction positions, retaining shared computations and the action head to produce full-chunk predictions.These forced-route comparisons provide allocation targets even for experts outside the current top-K.Source: 3) Learning action–expert alignment · p. 4
• The per-position error compares each forced-route prediction with the demonstrated action in normalized action space.The action dimension is d_a.Source: 3) Learning action–expert alignment · p. 4
• Relative expert errors form a softmax target scaled by the across-expert error standard deviation, lower-bounded by 10^-3, and detached during differentiation.Source: 3) Learning action–expert alignment · p. 4
• KL alignment supervises all pre-selection routing probabilities across the chunk, while imitation trains the selected action composition.All H positions are supervised; padding repeats the final demonstrated action.Source: 3) Learning action–expert alignment · p. 4

E. Training and Deployment

• Training combines action, CVAE, and action–expert alignment losses, using the same demonstrated-chunk latent across route comparisons and excluding it from router inputs.The reported configuration uses four rank-32 experts, top-2 routing, and seven decoder layers.Source: E. Training and Deployment · p. 4
• At deployment, each robot independently predicts 100 actions from its local observation, executes one, then observes again to update its prediction.Exponential temporal ensembling combines overlapping predictions with coefficient 0.01; deployment sets the latent to zero and omits expert comparison.Source: E. Training and Deployment · p. 4

IV. EXPERIMENTS

• The experiments address five questions spanning decentralized performance, component contributions, expert allocation, partner perturbations, and physical-robot collaboration.Source: §IV · p. 4

A. Benchmarks and Evaluation Protocol

• The simulation benchmarks cover 13 tasks across RoboFactory, DuoBench, and BiCoord, including sequential transfer, coupled motion, asymmetric support, and joint placement.DuoBench and BiCoord arms execute independently from local observations.Source: A. Benchmarks and Evaluation Protocol · p. 4
• Table I reports SSR/Avg. as percentages and SR as successes per 100 without retries; its bold and underlining indicate best and second-best results, including ties.Source: Table I · p. 5
• Methods are evaluated on 100 seeds per task with one rollout each and equal task weighting; physical experiments use two independent Piper arms running copies of one shared policy.The physical evaluation covers five tasks, with 30 demonstrations per task.Source: A. Benchmarks and Evaluation Protocol · p. 5

B. RQ1: Single-Policy Decentralized Collaboration

• CoRE reaches 84.2% success on RoboFactory and the highest mean performance among evaluated decentralized methods on all three simulation benchmarks.One shared policy learns the selected tasks and robot behaviors within each benchmark; centralized regression reaches 90.4% on RoboFactory.Source: B. RQ1: Single-Policy Decentralized… · p. 5
• Pooling demonstrations within each task improves all five tasks, while extending the pool across all tasks largely removes gains on LPD and ThreeStack.Lift and Camera remain strong under all-task pooling; independent regression has limited success across the five tasks.Source: B. RQ1: Single-Policy Decentralized… · p. 5
• Figure 4 illustrates sequential transfer, coupled manipulation, and asymmetric cooperation as distinct collaboration requirements.The task sequences progress from left to right.Source: B. RQ1: Single-Policy Decentralized… · p. 5, Figure 4 · p. 5

C. RQ2: Component Contributions and Routing Sparsity

• With spatial inputs and cross-attention experts fixed, action–expert alignment raises LPD success from 15% to 94%.The comparison supports a role for demonstrated-action supervision in selecting experts from local observations.Source: C. RQ2: Component Contributions and… · p. 5
• Figure 5 depicts shared expert use in sequential LPD transfers and asymmetric Take Photo cooperation, with colors summarizing action-history-weighted expert use.Gray bars mark grasp intervals, and snapshots align with dashed time guides.Source: Figure 5 · p. 6

E K Lift Camera 3Stack LPD Photo Avg.

• At E = 4, top-2 routing has higher mean success than top-1 or activating all experts; the selected default is (E, K) = (4, 2).At K = 2, increasing E from two to four improves LPD but reduces Take Photo performance, while eight experts offer no further mean improvement.Source: E K Lift Camera 3Stack LPD Photo Avg · p. 6
• CoRE (RGB) performs strongly on both sequential tasks, and adding spatial perception in full CoRE further improves ThreeStack; results support complementary contributions from spatial representations and action-guided expert selection to collaborative learning.The approach uses demonstrated-action supervision to learn expert selection from local observations alongside separately adaptable prediction paths.Source: E K Lift Camera 3Stack LPD Photo Avg · p. 6

D. RQ3: Shared Expert Use during Collaboration

• In LPD, transient increases in E4 and reductions in E1 recur near successive object-transfer transitions across robots.In Take Photo, the robots use similar E1/E3 mixtures near completion while the robot holding the subject retains a larger E2 contribution.Source: D. RQ3: Shared Expert Use during… · p. 6

E. RQ4: Robustness of Decentralized Collaboration

• CoRE achieves the highest success at every nonzero perturbation level in simulation, retaining more success as partner timing is disrupted.The evaluations introduce random partner delays in Lift and slow one ThreeStack robot to 0.25× speed; CoRE retains a larger fraction of nominal success as ThreeStack slowdown increases.Source: E. RQ4: Robustness of Decentralized… · p. 6, Figure 6 · p. 6, Figure 7 · p. 7

F. RQ5: Collaboration and Robustness on Physical Robots

• The five tasks span coupled, asymmetric, and sequential collaboration, including box assembly, pan transport, bridge assembly, block stacking, and block exchange.Two independently acting Piper robots perform the tasks.Source: F. RQ5: Collaboration and Robustness… · p. 7
• Across five physical tasks, CoRE achieves 91.3% success, versus 56.0% for CHORUS and 35.3% for CLS-DP, leading on all five tasks.Bridge assembly reaches 25/30 trials for CoRE versus 7/30 for CHORUS.Source: F. RQ5: Collaboration and Robustness… · p. 7
• Box assembly shifts from coupled part reorientation to asymmetric joining, whereas stacking transfers placement responsibility between arms after each block.The joining trajectories maintain mating alignment; each completed stack placement establishes the next support surface.Source: F. RQ5: Collaboration and Robustness… · p. 7, Figure 8 · p. 7
• Partner pauses leave box halves misaligned, while slowing the right arm to 0.25× tilts and rotates the pan, leaving it offset from its supports.Both failures disrupt the required object alignment.Source: F. RQ5: Collaboration and Robustness… · p. 7, Figure 9 · p. 7
• With 1.0 s pauses, CoRE completes 22/30 Box trials versus 4/30 for CHORUS; at 0.25× Pan speed, it completes 17/30 versus 1/30.CoRE outperforms CHORUS at every tested perturbation level, though success decreases as pauses lengthen or partner motion slows.Source: Figure 10 · p. 7, F. RQ5: Collaboration and Robustness… · p. 8

V. CONCLUSION

• CoRE learns single-policy decentralized collaboration from pooled multi-task, multi-robot demonstrations.Collaboration-role experts provide adaptable paths, and forced-route prediction errors against demonstrations supervise local expert selection without role labels.Source: §V · p. 8
• Across simulation benchmarks and physical manipulation tasks, identical policy copies support coupled, asymmetric, and sequential collaboration with improved tolerance to partner timing disruptions.Source: §V · p. 8
• The conclusion states that demonstrated actions can guide behavior learning and expert selection as collaborative responsibilities evolve.The shared policy conditions coordination on each robot’s own observations.Source: §V · p. 8