PaperlayerView exact revision on arXiv

Source-linked AI summary

Immiscible Diffusion Policy: Preserving Multimodal Robot Actions through Label-Free Noise Assignment

Xiao Zhang, Yuxin Chen, Zhixuan Liang, Guojian Zhan, Chenran Li, Chenfeng Xu, Masayoshi Tomizuka, Yiheng Li

arXiv:2610.09369v1cs.ROcs.LG

TL;DR

Diffusion policies may lose demonstrated action modes even when the training data are balanced, limiting available alternatives for robot execution. Immiscible Diffusion Policy assigns action chunks to noise samples during training, and across simulated and real-world tasks it improves modality preservation while maintaining strong task performance.

  • Problem

    Diffusion policies can collapse to a single action modality despite balanced demonstrations, limiting alternative strategies when environmental changes make one mode infeasible.

    Source: §I · pp. 1–2

  • Method

    Immiscible Diffusion Policy uses label-free Hungarian matching to couple action chunks with noise during training without changing policy architecture or inference.

    Source: §I · p. 2

  • Results

    Across seven tasks, modality preservation improved; non-dominant-modality proportions rose 6.0×–14.6× across three simulated two-modality tasks, and absent modes returned in both four-modality tasks.

    Source: §I · p. 2, B. Experiment Results · p. 7

  • Takeaways & Limitations

    The training-time add-on preserves demonstrated action modalities across varied robotic tasks and observation representations while maintaining strong task performance.

    Source: §I · p. 2

    Recovered modalities may have low success rates during restoration; achieving equally high-quality planning across all preserved modalities remains future work.

    Source: §VI · p. 8

Abstract

from arXiv · show

When diffusion policies were first introduced, they were expected to recover multi-modal action distributions. However, we find this expectation does not always hold, as diffusion policies often collapse to a single modality even when we guarantee the balance of dataset modalities and exact within-batch symmetry. Our analysis indicates that independent action-noise pairing contributes to this failure by increasing mixing and crossing among diffusion paths, which can produce averaged denoising responses and suppress modality-specific behavior. This issue is especially severe in robot planning, where action spaces are dense and low-dimensional, significantly increasing such mixing and crossing. To alleviate this problem, we propose Immiscible Diffusion Policy, a label-free training-time add-on to diffusion policy that uses action-noise assignment to preserve relatively distinct noise-to-action routes without modifying the policy architecture or inference procedure. Across five simulated and two real-world humanoid manipulation tasks spanning state, RGB, and point-cloud observations, our method significantly improves the policy's preservation of action modalities while maintaining strong task performance. It increases the proportion of the non-dominant modality by 6.0x-14.6x across three two-modality tasks and recovers demonstrated modalities that are entirely absent from vanilla policy rollouts on both four-modality tasks. These results demonstrate that Immiscible Diffusion Policy provides a simple yet robust approach to preserving action multi-modality in general robot learning tasks.

I. INTRODUCTION

The introduction identifies action-modality collapse as a problem for diffusion policies, even with balanced data, and motivates a label-free training add-on that preserves distinct action routes without changing the policy or inference procedure.

I. INTRODUCTION

  • Diffusion policies can collapse to one action mode despite balanced demonstrations and exact within-batch symmetry, with the selected mode varying across training seeds.In the Push-T example, all ten rollouts from each vanilla policy follow one of two symmetric pushing directions; collapse can remove an alternative needed when the preferred mode is infeasible.Source: §I · pp. 1–2
  • Preserving alternative action modes matters because shifts in obstacles, workspaces, objects, or execution constraints can make one demonstrated mode infeasible while another remains viable.Existing approaches may require mode-specific supervision, predefined mode structures, or changes to policy architecture or inference, limiting their fit to continuous, numerous, or unknown modes.Source: §I · p. 2
  • Immiscible Diffusion Policy uses label-free Hungarian matching to couple action chunks with noise during training, reducing path mixing while leaving architecture and inference unchanged.The approach seeks to keep each action modality reachable from distinct regions of Gaussian noise space.Source: §I · p. 2
  • 6.0×–14.6×: the non-dominant-modality proportion increases across three simulated two-modality tasks, while absent demonstrated modes are recovered in both four-modality tasks.The method is evaluated across seven tasks and is reported to maintain strong task performance.Source: §I · p. 2

A. Action Multimodality in Robot Learning

This section reviews approaches to representing diverse robot behaviors and prior data-noise assignment methods, then frames action-modality preservation in dense, low-dimensional robot planning as a distinct objective.

A. Action Multimodality in Robot Learning

  • Multiple task-equivalent action sequences can exist for one observation, and preserving this diversity can provide alternatives under environmental changes or perturbations.Prior approaches include latent plans, inference-time optimization, and explicit action discretization, each with a distinct representation or computational trade-off.Source: A. Action Multimodality in Robot… · p. 2
  • Diffusion Policy can model multimodal action distributions, but demonstrated modes may still be suppressed despite balanced data; this work targets that collapse without predefined modes or policy changes.Related approaches include goal-conditioned and point-cloud policies, consistency policies, IMLE Policy, and Deep Diffusion Policy Gradient.Source: A. Action Multimodality in Robot… · p. 2

B. Data-Noise Assignment in Diffusion Models

  • Prior data-noise assignment methods use optimal-transport coupling in flow matching or structured noise assignment in image diffusion, targeting straighter paths, simpler denoising, or efficiency.Immiscible Diffusion and Improved Immiscible Diffusion concern image spaces, while flow-matching approaches include and.Source: B. Data-Noise Assignment in Diffusion… · p. 3
  • Robot action spaces are lower-dimensional and denser than image-generation spaces, so nearby action clusters can overlap across noise levels and intensify path mixing and modality collapse.The proposed batch-level assignment instead targets action-modality preservation while keeping the policy architecture, noise schedule, and test-time sampler unchanged.Source: B. Data-Noise Assignment in Diffusion… · p. 3

A. Modality Collapse in Diffusion Policy

The analysis finds that balanced demonstrations do not prevent modality collapse and links the learned denoiser’s weak sensitivity to modality-related noise differences with independent action-noise pairing and path mixing.

A. Modality Collapse in Diffusion Policy

  • Balanced demonstrations and exact within-batch symmetry still leave vanilla policies concentrated on one pushing direction across all ten training seeds.Different seeds favor different directions, so the observed imbalance does not reflect an inherent geometric preference for one mode.Source: A. Modality Collapse in Diffusion Policy · p. 3

B. Why Action-Modality Collapse Happens

  • 97.6%: vanilla policies assign this average share of Push-T rollouts to each seed’s preferred modality; eight of ten policies produce only one modality across 50 rollouts.An analytical denoiser using the same finite-step reverse process produces both clockwise and counterclockwise modalities at an approximately balanced ratio.Source: B. Why Action-Modality Collapse Happens · p. 3
  • 174 versus 2.6: analytical-denoiser outputs cover 174 distinct expert action chunks, while vanilla outputs cover 2.6 on average across ten seeds.Three vanilla seeds cover only one distinct expert action chunk, indicating contraction in action space alongside single-modality rollouts.Source: B. Why Action-Modality Collapse Happens · p. 3
  • Independent noise pairing can make nearby action modalities overlap in noisy inputs, where the MSE-optimal prediction approaches their conditional average rather than either mode.The learned denoiser must separate that shared response as noise decreases; weak separation can let small optimization asymmetries bias the reverse process toward one branch.Source: B. Why Action-Modality Collapse Happens · p. 3
  • 4.29 versus 0.24: at the modality-separation noise level, the analytical denoiser’s modality-response range is 4.29, while the vanilla denoiser’s is 0.24.The vanilla response is 5.6% of the analytical response, indicating weaker sensitivity to noisy-input differences needed to route samples to distinct modalities.Source: B. Why Action-Modality Collapse Happens · pp. 3–4

A. Immiscible Diffusion Policy

Immiscible Diffusion Policy changes the batch-level pairing between action chunks and Gaussian noise, using Hungarian assignment to organize noise by action while retaining the standard diffusion training and inference setup.

A. Immiscible Diffusion Policy

  • Explicit action-noise coupling is intended to reduce diffusion-path mixing and crossing, keeping action modalities reachable from distinct regions of noise space.Fig. 2 shows clearer modality-dependent noise structure with Hungarian assignment than with independent pairing.Source: A. Immiscible Diffusion Policy · p. 4, Figure 2 · p. 4
  • The method adapts Immiscible Diffusion to conditional action generation, treating a mini-batch of dataset-normalized action chunks and an equally sized sampled noise pool as assignment inputs.Each noise sample has the same shape as an action chunk.Source: A. Immiscible Diffusion Policy · p. 4
  • After a vanilla-training warm-up, the method standardizes vectorized action chunks with mini-batch statistics and uses Hungarian matching to assign the batch’s noise samples.The matched noise then enters the standard forward diffusion process and the policy’s noise-prediction training formulation.Source: A. Immiscible Diffusion Policy · p. 4
  • The assignment only changes how action chunks and noise are paired; it leaves the batch-level noise marginal, architecture, loss, noise schedule, and test-time sampling procedure unchanged.It requires neither additional demonstrations nor additional gradient updates.Source: A. Immiscible Diffusion Policy · p. 4

B. How Noise Assignment Preserves Action Multimodality

Action-noise assignment organizes noise samples into modality-correlated regions, helping preserve distinct noise-to-action routes. This is especially useful when overlapping noisy action distributions can induce averaged denoising responses.

B. How Noise Assignment Preserves Action Multimodality

  • Hungarian matching pairs nearby action chunks with coherent Gaussian-noise regions, allowing initial noise to carry information about action modality without modality labels.Action chunks from the same modality tend to be closer to one another than chunks from different modalities.Source: B. How Noise Assignment Preserves… · pp. 4–5
  • On Push-T, assigned noise samples organize by modality and keep both action modes reachable from modality-correlated noise regions, unlike the overlap seen under vanilla Diffusion Policy.The method reduces cross-modality mixing and maintains distinct noise-to-action routes without changing inference or requiring modality labels.Source: B. How Noise Assignment Preserves… · p. 5
  • In low-dimensional action spaces, overlapping noisy inputs can have conflicting targets, encouraging averaged denoising responses that suppress modality-specific behavior.Assignment makes the noise region informative of the corresponding modality, helping maintain distinct responses across the overlapping regime.Source: B. How Noise Assignment Preserves… · p. 5
  • 1.93 versus 0.24 modality-response range at the modality-separation noise level; Immiscible retains approximately 45% of the analytical denoiser’s response, versus 5.6% for vanilla.Its response near the midpoint between modalities is also approximately 8.3 times stronger than vanilla’s.Source: B. How Noise Assignment Preserves… · p. 5

V. EXPERIMENTS

The experiments examine modality collapse despite balanced demonstrations, preservation of demonstrated action modalities, and whether multimodality can be preserved alongside task success.

V. EXPERIMENTS

  • The experiments ask whether Diffusion Policy collapses action modalities despite balanced demonstrations.This is the first stated experimental question.Source: §V · p. 5
  • The experiments test whether Immiscible Diffusion Policy preserves demonstrated action multimodality across different tasks.This is the second stated experimental question.Source: §V · p. 5
  • The experiments also ask whether multimodality can be preserved while maintaining task success rates.This is the third stated experimental question.Source: §V · p. 5

A. Experimental Setup

The evaluation measures rollout-mode balance and task performance across seven simulated and real-world tasks, using balanced demonstrations and matched training conditions for the two policies.

A. Experimental Setup

  • Rollout evaluation reports empirical action-mode frequencies and total variation distance from a balanced distribution, alongside task performance averaged over training seeds.Push-T performance is measured by environment coverage; the other tasks use success rate.Source: A. Experimental Setup · p. 5
  • The study evaluates seven tasks spanning Push-T, four bimanual RoboTwin tasks, and two real-world humanoid tasks with state, RGB, and point-cloud observations.The tasks include two- and four-mode settings, with all demonstrated modes feasible and task-equivalent in controlled configurations.Source: A. Experimental Setup · pp. 5–6, Figure 4 · p. 5
  • Table I covers action-modality preservation and task performance, reporting mean ± standard error over seeds except for TV, which is reported as a mean.Modes are ordered by vanilla-policy frequency.Source: Table I · p. 6
  • The demonstrations are balanced across valid modalities, with exact in-batch symmetry for simulated tasks and equal division between the two modes for real-world tasks.Vanilla and Immiscible policies use identical demonstrations and mini-batches.Source: A. Experimental Setup · p. 6
  • For each task, the policies share demonstrations, architecture, observations, action representation, and training settings; action-noise coupling is the methodological difference.At inference, they use the same sampler, number of denoising steps, and action-execution horizon.Source: A. Experimental Setup · p. 6

B. Experiment Results

Vanilla Diffusion Policy concentrates rollouts on a few action modalities across simulated and real-world tasks, while Immiscible Diffusion Policy produces more balanced modality distributions and largely maintains task performance. It also preserves modalities throughout training and recovers demonstrated modes absent from vanilla rollouts.

B. Experiment Results

  • Vanilla Diffusion Policy shows modality collapse across seven tasks, including simulated and real-world settings and low-dimensional, RGB, and point-cloud observations.In the two real-world humanoid tasks, all evaluated vanilla rollouts use a single hand modality.Source: B. Experiment Results · p. 6
  • Across three simulated two-modality tasks, Immiscible Diffusion Policy increases the average non-dominant-modality proportion by 26.05 percentage points, an 8.68× increase over vanilla.The non-dominant proportion rises from 2.40% to 35.00% in Push-T, 3.33% to 20.00% in Stack Blocks, and 4.44% to 33.33% in Place Bread; real-world tasks shift from a single modality to a 1:1 hand-modality split.Source: B. Experiment Results · pp. 6–7
  • In both four-modality tasks, Immiscible Diffusion Policy retains all demonstrated modalities, including the mode entirely absent from vanilla rollouts.Place Shoe’s four frequencies change from 37.8%, 37.8%, 24.4%, and 0.0% to 24.9%, 29.7%, 29.4%, and 16.0%; Rotate QR Code’s change from 57.8%, 35.6%, 6.7%, and 0.0% to 44.4%, 17.8%, 26.7%, and 11.1%.Source: B. Experiment Results · p. 7
  • During training, vanilla rollouts increasingly concentrate on one or a subset of modalities, whereas Immiscible Diffusion Policy maintains a more balanced, stable distribution.Figure 5 tracks epochs, with dominant-modality proportion for three two-modality tasks above and modality composition for two four-modality tasks below.Source: B. Experiment Results · p. 7, Figure 5 · p. 7
  • Task performance remains approximately similar on simulated two-modality tasks, while real-world and four-modality tasks show some success-rate drops.Push-T coverage changes from 95.16% to 95.39%, Stack Blocks success remains 96.67%, and Place Bread success changes from 100.00% to 97.78%; real-world success changes from 75% to 60% on Fruit-to-Plate and from 100% to 90% on Can-to-Bin.Source: B. Experiment Results · p. 7

C. Analysis and Ablation Studies

The analysis examines modality-conditioned success and assignment activation timing. Recovered modalities can remain less reliable, while activation timing trades off coverage and modality balance.

C. Analysis and Ablation Studies

  • The recovered M(4) modality has a conditional success rate of 85.71% in Place Shoe and 40.00% in Rotate QR Code, while success on the other shared modalities is largely comparable between methods.The authors suggest lower overall success primarily reflects the challenge of retaining and continuing to learn a modality vanilla policy discarded, rather than general degradation in policy execution.Source: C. Analysis and Ablation Studies · p. 8
  • Assignment activation timing matters because modality concentration grows during vanilla training, while delaying activation too long can allow demonstrated modes to disappear from rollouts.The analysis considers how timing affects final task performance and modality distribution.Source: C. Analysis and Ablation Studies · p. 8
  • Compared with activation at epoch 50, activating assignment at epoch 0 reduces coverage by 5.13 percentage points, while activation at epoch 100 increases dominant-modality proportion by 12.31 percentage points.Epoch 0 achieves similar modality balance to epoch 50; epoch 100 maintains coverage.Source: C. Analysis and Ablation Studies · p. 8, Table III · p. 8
  • The shared activation criterion uses the first evaluated checkpoint with task metric at least 60% and nonzero rollout frequency for every demonstrated modality.This leaves training time for assignment, starts before modality support is lost, and avoids tuning activation epochs per task.Source: C. Analysis and Ablation Studies · p. 8

VI. CONCLUSION

The conclusion links modality collapse to independent action-noise pairing and diffusion-path mixing in low-dimensional, clustered action spaces. Immiscible Diffusion Policy couples actions and noise during training, improving modality preservation while maintaining strong task performance; recovered modes may still have low success during restoration.

VI. CONCLUSION

  • Diffusion policies can concentrate on a single or partial action modality despite balanced demonstrations and exact within-batch symmetry.The paper connects collapse to independent action-noise pairing, which increases diffusion-path mixing and crossing and weakens denoiser sensitivity to modality differences in low-dimensional, clustered action spaces.Source: §VI · p. 8
  • Immiscible Diffusion Policy is a label-free training-time add-on that couples action chunks with noise, improving modality preservation across simulation and real-world tasks while maintaining strong task performance.It leaves policy architecture and inference unchanged and recovers valid modalities absent from vanilla rollouts.Source: §VI · p. 8
  • Recovered modalities may have low success rates during restoration, and achieving equally high-quality planning across preserved modalities remains future work.Source: §VI · p. 8
Loading 2610.09369v1…