PaperlayerView exact revision on arXiv

Source-linked AI summary

One for All, All for One: Coordinated Multi-Agent Diffusion Steering via Stochastic Optimal Control

Riccardo Barbano, Vincent Pauline, Runchang Li, George Webber, Alexander Denker, Željko Kereta, Stefan Bauer, Francisco Vargas, Esmeralda S. Whitammer

arXiv:2610.08595v1cs.ROcs.AI

TL;DR

Structured generation must model both component distributions and their interactions, while recurring coordination tasks can make instance-by-instance steering costly. CMDS uses stochastic optimal control to coordinate frozen component diffusion models through an amortised control and assembly-level reward. Experiments demonstrate target-law recovery, relational assembly, source recovery, and coordinated multi-agent generation.

  • Problem

    Structured outputs require modelling both component distributions and their interactions, while related coordination problems can make per-instance sampling-time coordination costly.

    Source: §1 · p. 1

  • Method

    CMDS learns an amortised stochastic-optimal-control policy that couples frozen component diffusion processes using an assembly-level reward while penalising deviation from pretrained dynamics.

    Source: §1 · p. 3, §2 · p. 4

  • Results

    CMDS demonstrates target-law recovery, amortised relational assembly, source recovery, and coordination in multi-arm planning and multi-person motion.

    Source: §4 · p. 9

  • Takeaways & Limitations

    Frozen component models can support structured tasks beyond their individual training settings when coordination is learned separately from component generation.

    Source: §4 · p. 9

    CMDS's central hypothesis assumes locally valid components can be learned separately, with global validity supplied through an assembly-level criterion.

    Source: §2 · p. 4

Abstract

from arXiv · show

Deep generative models often produce structured outputs composed of interacting components. Modelling these outputs with a single model requires learning both the component distributions and their interactions. We pursue a modular alternative: reuse independently trained component generators and learn only how to coordinate them to produce coherent structured outputs. Our framework, Coordinated Multi-Agent Diffusion Steering (CMDS), treats frozen pretrained diffusion models as reusable generative primitives and coordinates their reverse processes through a learned control. We formulate coordination as a stochastic optimal control problem, balancing an assembly-level reward that specifies the desired properties of the combined output against deviations from the pretrained dynamics. The learned control amortises this optimisation, allowing reuse across new task instances. Experiments show that CMDS can recover a known target distribution, satisfy different spatial constraints with the same trained control, and recover individual sources from degraded mixtures. Across multi-agent maze navigation, articulated robot planning, and text-conditioned human motion, CMDS turns frozen models into coordinated multi-agent generators.

1 INTRODUCTION

CMDS addresses how to coordinate independently trained component generators into coherent structured outputs, amortising coordination across recurring tasks. It steers frozen diffusion processes with a learned control and assembly-level reward; experiments span target-law recovery, source separation, and coordinated generation.

1 INTRODUCTION

  • CMDS asks how to learn an amortised control that coordinates frozen component diffusion models using an assembled-output reward while remaining close to pretrained dynamics.Related compositional diffusion methods resolve coordination separately at sampling time for each instance; amortisation targets settings where related coordination problems recur.Source: §1 · pp. 1–2
  • The framework combines independently trained component generators through an assembly map and uses an assembly-level reward to coordinate their outputs, without requiring valid structured-output training examples.The stated inputs are component-level data or pretrained generators, an assembly map, and an assembly-level reward.Source: §1 · p. 2
  • CMDS formulates coordination as stochastic optimal control over frozen component reverse processes, coupling them through a learned control and an assembly-level reward.Independent reverse processes provide reusable component priors; the reward specifies desired properties of the assembled output.Source: §1 · pp. 1, 3
  • In four-person motion generation, CMDS achieves 90% joint success versus 75% for PCD++, with over 250× faster sampling.The introduction also reports coordination of up to four PointMaze agents, generalisation to unseen start–goal regions, and transfer of the learned KUKA control to new geometries and additional arms without retraining.Source: §1 · p. 3

2 COORDINATED MULTI-AGENT DIFFUSION STEERING (CMDS)

CMDS controls the joint reverse process of frozen diffusion components so their assembled terminal output meets a global criterion while limiting deviation from pretrained dynamics. It learns an amortised control and interprets optimal coordination as reward-based reweighting of joint trajectories.

2 COORDINATED MULTI-AGENT DIFFUSION STEERING (CMDS)

  • CMDS steers the joint reverse process of N pretrained diffusion agents so the assembled terminal output meets a global criterion while remaining close to independent pretrained dynamics.The assembly operator maps the joint state to an assembled state, and the reward evaluates global consistency while quadratic control cost penalises deviation from pretrained distributions.Source: §2 · pp. 3–4

2.1 LEARNING COORDINATED REVERSE DYNAMICS

  • Adjoint matching trains a parametrised coordination control to match the optimal correction estimated along sampled controlled trajectories.The authors choose it because it propagates gradient information without backpropagating through the complete sampling-rollout computational graph; direct discretised-objective optimisation and path-space objectives are alternatives.Source: §2.1 · p. 4

2.2 PATH-SPACE INTERPRETATION OF COORDINATION

  • Under the stated change-of-measure conditions and a shared initial law, the quadratic control cost equals path-space relative entropy, and optimal coordination exponentially tilts the reference path law by assembly-level reward.The reference law is induced by independent pretrained reverse diffusions; the controlled law is the corresponding joint process under control.Source: §2.2 · p. 5

3 EXPERIMENTAL EVALUATION

The evaluation tests recovery of a known target law, source separation from blurred mixtures, and coordination of pretrained models across navigation, robot planning, and human motion.

3.1 TWO-AGENT OU COORDINATION

  • CMDS recovers the oracle joint terminal law with discrepancies comparable to oracle self-comparisons and lower than importance sampling.An unconstrained control can satisfy the scalar distance constraint yet fail to recover the target's angular structure.Source: §3.1 · p. 6, Figure 3 · p. 6

3.2 SOURCE SEPARATION

  • Under τ = 1 blur, CMDS achieves the lowest clean-mixture reconstruction error and highest source IoU while retaining low shape distortion.DPS improves source localisation over independent sampling but distorts components more and has larger reconstruction error than CMDS.Source: §3.2 · p. 6, Figure 4 · p. 7

3.3 FROM SINGLE-AGENT PRIORS TO COORDINATED MULTI-AGENT MOTION

  • On unseen PointMaze Pool 2 instances, CMDS+TRG achieves 89.8%–99.2% success across reported settings and five-seed mean success of 90.6%–98.3%.Training-seed standard deviations are no larger than 1.5 percentage points.Source: Table 1 · p. 7, §3.3 · p. 8
  • On 413 held-out dual-arm tasks with one candidate, CMDS+TRG reaches 77.3% strict success versus 11.4% for independent sampling and 51.0% with collision guidance.The test layouts were unseen by both the priors and controller.Source: Table 2 · p. 8, §3.3 · p. 8
  • In zero-shot KUKA generalisation, for the task with ncand = 4, CMDS + TRG achieves 68.3% strict success, compared with 48.3% for guided single-arm priors.Source: Table 3 · p. 9, §3.3 · p. 9
  • CMDS achieves 90% four-person joint success versus 75% for PCD++ on the motion-coordination task.PCD++ has a higher collision-free rate, but lower joint success and longer sampling time.Source: Table 4 · p. 9, §3.3 · p. 9

4 CONCLUSION

The paper positions CMDS as a modular stochastic-control framework that coordinates frozen component generators through learned interactions. Its conclusion highlights target-law and source recovery, motion-planning applications, transfer, and open directions for improving reward design and scaling to more heterogeneous models.

4 CONCLUSION

  • CMDS coordinates frozen pretrained diffusion models through stochastic optimal control, with an assembly-level reward and an amortised joint control under a quadratic path-action penalty.The conclusion presents target-law recovery, source recovery, multi-arm planning, multi-person motion, and transfer to additional robot arms as evidence for reuse beyond component training settings; future work includes reward design, target-law approximation, and larger heterogeneous models.Source: §4 · p. 9

A RELATED WORK

  • CMDS follows modular generation while distinguishing component states from assembled outputs and using an assembly-level reward to specify their interactions.Earlier modular and compositional approaches include pattern-theory and structured-scene models, and recent work advocates composing specialised generators.Source: A RELATED WORK · p. 15
  • Unlike paired-data supervision or inference-time cost guidance, CMDS learns reusable coordination from an assembly-level reward; finite reward penalties do not guarantee hard constraints.The adjoint-based implementation assumes differentiable rewards, while inference-time guidance or projection could in principle be applied on top of CMDS when instance-specific constraints are needed.Source: A RELATED WORK · pp. 15–16
  • Across planning and source inference, CMDS keeps component priors frozen and amortises joint coordination rather than relying solely on offline multi-agent training, search, or inference-time guidance.Stochastic-control and amortised-steering work provides the foundations; adjoint matching is one learning strategy within CMDS, not part of the framework definition.Source: A RELATED WORK · p. 16

B DIFFUSION AND PATH-SPACE FORMULATION OF CMDS

This part formalizes CMDS as coordination of independently pretrained reverse processes through control of component paths, balancing assembly-level reward against path-space deviation while tracking the assembled process. It derives the induced assembled path measures and explains adjoint-matching training, including the choice to retain full cross-agent dependencies.

B.1 FORWARD AND REVERSE DIFFUSION PROCESSES

B.2 CONTROLLED REVERSE DIFFUSIONS AND THE ASSEMBLED PROCESS

  • Controlling the full component state yields an assembled state whose fixed-time law is a pushforward and whose marginals obey an exact projected Fokker–Planck equation, but the assembled process need not be Markov.Non-Markovianity can arise when the assembly map discards information about dynamics; an explicit density may also fail to exist without further assumptions.Source: B DIFFUSION AND PATH-SPACE FORMULATION… · p. 17, §B.2 · p. 20

B.3 PATH-SPACE CHANGE OF MEASURE AND KL COST

  • The quadratic control action equals the relative entropy from the independent pretrained product process, making coordination a minimum-KL path-space change of controls.The relative-entropy cost of controlling assembled paths is bounded above by the corresponding full component path-space cost.Source: B DIFFUSION AND PATH-SPACE FORMULATION… · p. 17, §B.3 · p. 21

B.5 PUSHFORWARD TO THE ASSEMBLED PROCESS

  • Pushing the path-space change of measure through the assembly map gives a direct density ratio when the map is injective, or a conditional average over component trajectories with the same assembly when it is non-injective.The non-injective case matters because distinct component trajectories can map to the same assembled trajectory.Source: B DIFFUSION AND PATH-SPACE FORMULATION… · p. 17, §B.5 · p. 23

B DIFFUSION AND PATH-SPACE FORMULATION OF CMDS

B.4 OPTIMAL COORDINATED PATH AND TERMINAL LAWS

  • The optimal path law is a reward tilt of the reference law, with an initial-state-dependent normalizer; under complete noising, the terminal law is reward-tilted and non-separable rewards induce cross-component dependencies.The tilt can change both dependence structure and component marginals; learned control dynamically realizes this change of measure.Source: §B.4 · pp. 21, 23

B.5 PUSHFORWARD TO THE ASSEMBLED PROCESS

  • Component-path controlled and uncontrolled samples map through φ to assembled samples and induce push-forward path measures; when φ is injective, the assembled path determines its component path.Source: §B.5 · p. 24

C ADJOINT MATCHING OPTIMISATION

  • The adjoint turns the global assembly objective into local learning signals by propagating terminal-cost feedback backward while accounting for remaining control cost and cross-agent dependencies.The terminal cost is g = −λr◦φ; the formulation also allows an additional running cost C.Source: C ADJOINT MATCHING OPTIMISATION · p. 25

C.1 FULL PATHWISE ADJOINT

C.2 ADJOINT MATCHING INSTANTIATION OF CMDS

  • Under the assumptions of Domingo-Enrich et al. (2025), adjoint matching and the original SOC objective share θ-gradients, and the matching loss’s unique control-space critical point is the optimal control.Source: §C.2 · p. 26

C.2.1 RELATION TO LEAN ADJOINT MATCHING AND AGENT-WISE STRUCTURE

  • The lean adjoint generally changes the objective gradient away from the optimum, although its unique optimum matches the SOC-optimal control under the stated control-affine assumptions.CMDS is formulated as a control-affine SOC problem, so the lean construction can also be applied to its continuous-time formulation.Source: §C.2.1 · pp. 26–27
  • With factorised reference dynamics and C = 0, lean-adjoint backward propagation separates by agent after terminal joint credit assignment, then carries each agent’s assigned information through its own frozen dynamics.Source: §C.2.1 · p. 27
  • Lean-adjoint propagation scales linearly with agent count for fixed component-model cost, but total CMDS computation need not, because the joint control and assembly reward may involve higher-order interactions.Source: §C.2.1 · p. 27
  • CMDS retains the full adjoint in its discrete DDIM recursion, differentiating controlled transitions and quadratic control cost to preserve cross-agent dependencies not removed by task-specific detaches.A discrete lean variant could propagate adjoints only through frozen base transitions.Source: §C.2.1 · p. 27

C.2.2 DISCRETE-TIME ADJOINT MATCHING FOR DDIM SAMPLING

The discrete-time implementation learns CMDS controls through DDIM rollouts, backward adjoint replay, and control regression. It uses full-adjoint replay, while matching loss can be evaluated on a subset of stochastic transitions.

C.2.2 DISCRETE-TIME ADJOINT MATCHING FOR DDIM SAMPLING

  • Each agent’s control can depend on the full multi-agent state, and control parameters are shared across DDIM transitions with the timestep supplied as an input.Source: §C.2.2 · pp. 27, 28, 30
  • The epsilon residual induces a one-step displacement that is normalized into a discrete control, with the transition’s stochastic scale entering the one-step SOC cost.Source: §C.2.2 · p. 28
  • The base DDIM map is a one-step counterfactual, while only the controlled map advances the sampled trajectory.Base and controlled updates use the same current state and noise; the terminal transition retains a positive stochastic scale and control cost.Source: §C.2.2 · p. 28
  • The adjoint recursion spans every DDIM transition, even when the matching loss uses a strict subset, and those matching terms are not importance re-weighted.Source: §C.2.2 · p. 29
  • The discrete procedure combines a controlled DDIM rollout, backward adjoint replay, and control regression, differentiating through controlled transitions and the quadratic control cost.The matched-transition set can be resampled each optimization iteration, while adjoint recursion always covers all reverse transitions.Source: §C.2.2 · p. 29
  • The two-agent toy compares the uncontrolled OU reference, a value-function oracle, importance sampling, and learned control for a prescribed terminal separation.Source: Figure 8 · p. 31, Figure 9 · p. 31

D.1 TWO-AGENT OU COORDINATION: A STOCHASTIC-CONTROL SANITY CHECK

A two-agent Ornstein–Uhlenbeck sanity check tests whether learned control can meet a relational terminal-distance target while preserving rotational symmetry and limiting path-action cost. CMDS approaches the stochastic-control oracle more closely than importance sampling at the reported proposal budget, while an unconstrained MLP can satisfy distance yet distort angular diversity.

D.1 TWO-AGENT OU COORDINATION: A STOCHASTIC-CONTROL SANITY CHECK

  • The experiment uses two agents with known two-dimensional Ornstein–Uhlenbeck reference dynamics and a reduced value-function oracle to isolate the stochastic-control mechanism.Their terminal relational cost depends on relative displacement, while minimum-action control leaves the common mode uncontrolled.Source: §D.1 · pp. 31–32
  • The radial HJB construction yields an oracle path-measure change through a desirability function and permits direct terminal sampling by radial quadrature.The oracle sampling avoids time-discretisation error from an Euler rollout of the optimal control.Source: §D.1 · p. 33
  • The rotation-equivariant control achieves terminal separation close to target while maintaining an approximately uniform angular distribution, unlike the unconstrained MLP, which may collapse angles toward a preferred direction.The reference process concentrates both agents near the origin; the relational terminal cost encourages their separation to match prescribed distance dT.Source: §D.1 · pp. 32, 34
  • The experiment illustrates least-action coordination: control seeks the smallest path-space perturbation needed for independently evolving components to satisfy a system-level consistency criterion.The objective balances the terminal constraint against quadratic path action rather than moving samples arbitrarily toward reward.Source: §D.1 · p. 34
  • 0.497 ± 0.022 vs 0.917 ± 0.035 on W2, 0.0180 ± 0.0082 vs 0.0656 ± 0.0075 on MMD, and 0.888 ± 0.011 vs 1.103 ± 0.030 on Sinkhorn transport cost for CMDS versus importance sampling.At 220 proposals per evaluation seed, importance sampling has pre-resampling ESS 99.6 ± 15.3 and 343.4 ± 20.6 distinct proposals after resampling.Source: §D.1 · p. 34

D.2 RELATIONAL VISUAL ASSEMBLY

Relational visual assembly tests whether frozen one-square generators can be coordinated into valid two-square canvases under left/right and separation constraints. CMDS uses an assembly-level reward and learned control, with a single amortised controller handling several requested separations.

D.2 RELATIONAL VISUAL ASSEMBLY

  • The SOC objective favours valid assemblies requiring the smallest deviation from independent pretrained dynamics, while reward-based coordination remains vulnerable to poorly specified rewards.The appendix experiment serves as a controlled testbed for studying component factorisation, optimisation, and reward-design trade-offs.Source: §D.2 · p. 35
  • For left-right steering, the trained controls guide each component square into its assigned half, yielding valid stitched assemblies under a fixed masking operator.The classifier-only reward scores square presence in each component’s corresponding masked half.Source: §D.2 · p. 35, Figure 11 · p. 36
  • A one-square baseline must create an additional square through reward tilting because its pretrained support contains only single-square canvases.The baseline’s support is misaligned with the target object class.Source: §D.2 · p. 36
  • A two-square baseline has the right object count but lacks component assignments, leaving the global reward to solve an ambiguous assignment problem.CMDS builds assignments into the sampler, so the shared reward can induce more targeted, lower-action coordination.Source: §D.2 · p. 37
  • Single-process baselines struggle with coupled validity, whereas CMDS coordinates two pretrained one-square generators through the assembled canvas and achieves near-perfect validity with improved ELBO.Table 7 reports coupled-steering diagnostics; log ZIS estimates the reward-tilted target’s log normalising constant.Source: Table 7 · p. 37
  • A single pair of jointly trained agent-specific controls steers assemblies toward requested separations d∆x ∈{3, 4, 5, 6}, with achieved separations closely matching the targets.The controls receive the requested separation as a one-hot input and add residuals to their respective frozen denoisers’ noise predictions.Source: §D.2 · p. 38, Figure 14 · p. 40

D.3 SOURCE SEPARATION

The source-separation experiment coordinates independently pretrained circle and square priors to explain blurred mixtures without paired source–mixture training data. CMDS improves mixture reconstruction and labelled source placement while retaining prior-like shapes; stronger blur can leave source locations ambiguous.

D.3 SOURCE SEPARATION

  • The task seeks to recover latent sources from a combined observation using independently trained source samplers, without paired source–mixture training examples.The recovered assembly should match the observation while each component remains consistent with its pretrained distribution.Source: §D.3 · p. 40
  • The evaluation uses blurred additive mixtures of ground-truth circle and square images, with Gaussian convolution at τ ∈{0.5, 1.0} and no additional observation noise.The priors are trained independently and frozen before coordination.Source: §D.3 · pp. 40–41
  • A centralised joint controller gives two frozen component processes source-specific additive controls using shared representations, measurement residuals, and component-wise reward gradients.Product-DPS recomputes observation-dependent corrections for each mixture, whereas CMDS amortises that dependence in a learned joint feedback control.Source: §D.3 · pp. 41–42
  • Reconstruction RMSE, source IoU, and shape distortion assess assembly recovery, labelled source placement, and source-shape validity, respectively.A good mixture reconstruction alone does not establish that each recovered structure has the correct source label.Source: §D.3 · p. 42
  • At τ = 0.5 and τ = 1.0, CMDS achieves the lowest reconstruction RMSE and highest source IoU while retaining low shape distortion of the frozen component priors.Under τ = 1.0, the circle and square can exchange ground-truth locations while preserving plausible shapes and a similar blurred assembly.Source: §D.3 · p. 43, Figure 15 · p. 43
  • Under stronger blur, the quadratic cost can favour a nearby prior-consistent decomposition over the larger intervention needed to recover exact ground-truth placement.Thus plausible source recovery need not equal ground-truth recovery when the observation is ill-posed.Source: §D.3 · pp. 40, 43

D.4 POINTMAZE COLLISION COORDINATION

  • Multi-agent maze coordination can be amortised across up to four frozen trajectory models, with high collision-free joint success and generalisation to unseen start–goal regions.Temporal and cross-agent attention reduces complexity from O(N^2H^2) to O(NH(N + H)); the self-calibrated pseudo-Huber loss improves mean success in six of eight comparisons but lacks squared adjoint matching’s guarantees.Source: §D.4 · p. 43

D.4.1 TASK, STATE, AND REWARD

The PointMaze task stacks independently generated agent trajectories and uses a shared control to coordinate reconstructed joint trajectories. Its reward combines wall and inter-agent collision avoidance with speed, acceleration, and path-length penalties.

D.4.1 TASK, STATE, AND REWARD

  • Start and goal positions condition reconstructed trajectories as residual modifications to straight-line base paths, while a shared control acts on the reconstructed joint trajectories.Goal reach is enforced structurally and is not part of the reward.Source: §D.4.1 · p. 44

D.4.2 REWARD

  • The reward evaluates wall clearance and inter-agent separation on the assembled trajectories, with collision penalties measured on a densified, interpolated waypoint grid.Wall penalties vanish at least one agent radius from a wall; pair penalties vanish at separation of at least 2r_agent + m.Source: §D.4.2 · pp. 44–45
  • Speed, acceleration, and path length regularise trajectories alongside feasibility penalties; speed is penalised only above vmax, while acceleration and path length are penalised throughout.The reward is evaluated at the terminal assembled state.Source: §D.4.2 · pp. 44–46

D.4.3 DATA SPLITS.

The maze setup separates training and held-out start–goal regions, then applies a shared control to frozen single-agent trajectory priors. Its axial attention lets agents communicate at shared waypoints while retaining temporal context.

D.4.3 DATA SPLITS.

  • Training and test pools use disjoint, spatially interspersed maze cells, and each split evaluates the same fixed 128 instances per method and setting.Training samples are retained when independent paths collide and Conflict-Based Search finds a joint solution within a specified compute budget.Source: §D.4.3 · pp. 46–47

D.4.4 FROZEN PER-AGENT PRIOR

  • Each layout uses a single-agent trajectory diffusion model pretrained once and frozen during coordination training, reaching approximately 0.99 wall feasibility on both layouts.The prior is a 100-step discrete-time DDPM with a DiT-style denoiser and approximately 7.3M parameters.Source: §D.4.4 · p. 47

D.4.5 CONTROL ARCHITECTURE

  • The shared control alternates within-agent temporal attention with cross-agent attention at shared waypoints, using geometric bias and a reward-gradient extension.The base control is initialized to leave the pretrained sampler unchanged; the extension adds a learned time-dependent, agent-shared gate.Source: §D.4.5 · pp. 47–48

D.4.6 TRAINING OBJECTIVES

The training section develops robust adjoint matching for stochastic coordination and reports its seed-variance comparison. It also describes evaluation protocols and summarizes qualitative maze and KUKA results.

D.4.6 TRAINING OBJECTIVES

  • Adjoint matching trains the control by rolling out without gradient tracking, propagating a terminal-cost adjoint backward through DDIM steps, and matching the resulting correction.Headline results use robust regression; clean estimates are clipped at low signal-to-noise ratios before conversion to equivalent noise predictions.Source: §D.4.6 · p. 48
  • Collision-heavy early trajectories can make squared matching amplify high-variance minibatch adjoints and produce unstable, seed-dependent optimization.Pseudo-Huber loss is introduced to reduce the influence of rare high-norm residuals.Source: §D.4.7 · pp. 48–49
  • Pseudo-Huber matching generally improves mean held-out success and reduces variance, particularly in the Large maze with four agents.The squared-regression guarantee does not automatically extend to this modified objective, whose connection to the original stochastic-control objective remains open.Source: §D.4.7 · p. 49, Table 9 · p. 49
  • Per-transition pseudo-Huber thresholds are calibrated from residual quantiles, then frozen; calibration settings are shared across mazes and agent counts.This replaces the squared regression surrogate, not the underlying stochastic optimal control objective.Source: §D.4.7 · p. 49

D.4.8 TRAINING VARIANCE

  • CMDS+TRG training variance is reported as mean and standard deviation across five seeds for each setting and pool.The maze baseline suite uses the same frozen prior, trajectory representation, reward, and fixed evaluation sets; DPS and FK-Steering are tuned per task.Source: §D.4.8 · p. 50, §D.4.9 · p. 50

D.4.10 QUALITATIVE EXAMPLES

  • Qualitative maze examples compare reference and controlled trajectories on the same six held-out instances and initial noise for each maze and agent count.Source: §D.4.10 · p. 50

D.5 MULTI-ARM KUKA COORDINATION FROM FROZEN SINGLE-ARM PRIORS

  • Across KUKA transfer settings, a two-arm controller trained on frozen single-arm priors generalizes without retraining and has higher mean strict success than evaluated guidance baselines.The section also reports that adjoint training differentiates reverse dynamics through the learned energy using transition-wise Hessian–vector products.Source: §D.5 · p. 50

D.5.1 TASK, REPRESENTATION, AND DATA

The KUKA experiments coordinate independently pretrained arm priors over shared scenes and task conditions. The control uses discrete adjoint matching and a differentiable collision reward, with PBDM refinement and guidance methods as baselines.

D.5.1 TASK, REPRESENTATION, AND DATA

  • KUKA paths use fixed start and goal joint configurations, while evaluation checks joint limits and collisions but does not enforce waypoint timing or joint-velocity limits.The arm-count comparison uses the same 209 tasks whose endpoints pass feasibility checks for all counts; results are conditional on that endpoint feasibility.Source: §D.5.1 · pp. 50, 53

D.5.2 ENERGY-BASED PRIORS AND DISCRETE ADJOINT MATCHING

  • Discrete adjoints propagate through all ten denoising transitions, while control costs and matching terms cover only the nine stochastic transitions.Because the frozen noise predictor is the gradient of an energy, the required vector–Jacobian products are Hessian–vector products through that energy.Source: §D.5.2 · p. 54

D.5.3 COLLISION REWARD, CONTROLLER, AND TRAINING

  • The differentiable collision reward estimates link clearances with sampled configurations and sphere approximations, penalizing inter-arm, obstacle, table, and self-collisions.Penalties activate within 1 cm of contact; the reward and control cost do not guarantee collision-free, smooth, or dynamically feasible motion.Source: §D.5.3 · p. 54

D.5.4 BASELINES AND PBDM REFINEMENT

  • PBDM results enable optional refinement with candidate budgets matched to other methods, while direct collision guidance uses the same reward and validation-selected strength.Refinement attempts and checking are included in planning time, though repair attempts are excluded from the candidate budget.Source: §D.5.4 · pp. 54–55

D.5.5 STRICT EVALUATION, TIMING, AND UNCERTAINTY

CMDS coordinates four text-conditioned human motions with frozen single-person priors, reaching 90.0% joint success versus 75.0% for PCD++ at substantially lower assembly time. The paper also specifies strict KUKA evaluation and the motion controller’s architecture and reward.

D.5.5 STRICT EVALUATION, TIMING, AND UNCERTAINTY

  • Strict KUKA success requires no detected collisions and finite joint angles within limits; timing includes generation, checking, and any repair through the final decision.The mesh checker samples each trajectory segment at least eight times, with joint increments no larger than 0.04 radians.Source: §D.5.5 · p. 55

D.6 GOAL-DIRECTED MULTI-PERSON MOTION COORDINATION

  • 90.0% joint success versus 75.0% for PCD++, with 0.54 s per assembly versus 145.34 s for PCD++.The shared control coordinates four independently text-conditioned frozen MDM processes without paired interaction data.Source: §D.6 · p. 55

Dual KUKA

  • Only CMDS+TRG maintains positive inter-arm clearance over the full illustrated KUKA path; the figure marks collisions and shows clearance below each path.Source: Figure 21 · p. 57

D.6.1 TASK, REPRESENTATION, AND FROZEN MDM PRIOR

  • The human-motion task sends four people from a randomly rotated ring toward opposite destinations, allowing waiting and lateral detours during 120-frame motions.Each frozen MDM denoiser receives its own caption, not the other people’s captions or the scene’s start and goal conditions.Source: Dual KUKA · p. 58, §D.6.1 · p. 58

D.6.2 COOPERATIVE CONTROL ARCHITECTURE

  • The shared controller alternates temporal attention over each person’s clip with attention across people at the same physical frame, using shared weights for permutation equivariance.Its inputs combine noisy features, clean-motion estimates, and decoded world-space joints.Source: §D.6.2 · p. 59

D.6.3 ASSEMBLY REWARD

  • The assembly cost combines collision avoidance, goal attainment, directional progress, and physical consistency; people may arrive at different frames within the final window.The progress threshold shapes training but is not an additional success condition.Source: §D.6.3 · p. 60

D.6.4 TRAINING BY ADJOINT MATCHING

The appendix specifies controller training by adjoint matching and contrasts projection-based baselines, including their feasibility checks and limits. The learned controller is trained on generated multi-person trajectories without paired interaction data.

D.6.4 TRAINING BY ADJOINT MATCHING

  • Only controller parameters are updated, using online generated trajectories without paired interaction data.Training scenes draw captions from walking, jogging, high-knee marching, forward jumping, and crouched walking.Source: §D.6.4 · p. 61
  • The expected sum of Gaussian mean-shift control energies equals path-space KL to the frozen sampler at guidance scale 2.5, but excludes arbitrary nonzero zero-noise corrections.The shared initial Gaussian law and stochastic transitions support this interpretation; deterministic DDIM with η = 0 is outside its scope.Source: §D.6.4 · p. 61
  • Adjoint replay detaches controller observations and the next-state adjoint while keeping the noisy-state input and frozen-prior mean differentiable.The deterministic final transition remains in adjoint propagation but adds neither control energy nor a matching term.Source: §D.6.4 · p. 62
  • After 500 successful squared-matching updates, per-instance RMS residuals calibrate each δℓ to its 95th percentile, floored at 10^-3.Pseudo-Huber matching downweights large residuals and generally changes the matching gradient, while leaving terminal reward, adjoint recursion, and quadratic control energy unchanged.Source: §D.6.4 · p. 62

D.6.5 BASELINES

  • PCD uses a root-goal projector but does not enforce collision clearance or kinematic validity; PCD++ instead jointly optimises all active feature channels.PCD++ checks feasibility using a freshly recomputed clean estimate at the candidate’s next noise level.Source: §D.6.5 · p. 63
  • PCD++ accepts the first verified feasible iterate within its update budget; its solver neither certifies a nearest feasible point nor continuous-time clearance or full kinematic validity.The implementation approximates the constrained projection, checks the stated goal constraints and capsule proxy only at checked times, and has no infeasible fallback.Source: §D.6.5 · pp. 63–64

D.6.6 ADDITIONAL EXPERIMENTAL EVALUATIONS

The evaluation compares methods on matched four-person scenes, and the extended qualitative example distinguishes methods that satisfy both goals and collision constraints. PCD++ performs joint state projection, accepting a feasible candidate or terminating the assembly on failure.

D.6.6 ADDITIONAL EXPERIMENTAL EVALUATIONS

  • Evaluation uses 24 fixed scenes and five sampling seeds per method, yielding 120 attempts; the seeds measure sampling variability, not training variability or held-out layout generalisation.Matched views use the same seed and scene conditions; Table 15 aggregates the full evaluation.Source: §D.6.6 · p. 64, Table 15 · p. 65
  • PCD++ jointly optimises each proposed state against a fresh clean estimate and accepts it when the stated constraints are feasible, otherwise terminating without an infeasible fallback.Under the reported settings, each assembly uses at most 5050 joint backward evaluations; feasibility checks do not establish continuous-time clearance or full kinematic validity.Source: §D.6.6 · p. 65
  • In the illustrated four-person example, PCD++ and CMDS reach all goals without inter-person collisions, unlike uncontrolled diffusion, Best-of-256, DPS, and PCD.The prompts assign jogging to Persons 1 and 4 and walking to Persons 2 and 3.Source: Figure 22 · p. 66

Related papers

Loading 2610.08595v1…