PaperlayerView exact revision on arXiv

Source-linked AI summary

ManiFusion: Unlocking the High-Throughput Generation via Superposition in Manifold Space

Minkyu Kim, Baekseung Kim, Junhoo Lee, Jangho Kim, Nojun Kwak

arXiv:2610.21433v1cs.CVcs.LG

TL;DR

Diffusion sampling is costly because it requires iterative denoising, while gains from reducing per-sample cost are becoming harder to extend. ManiFusion fuses multiple samples for shared denoising and separates them afterward, reaching higher-throughput regimes across diffusion settings and alongside existing acceleration methods.

  • Problem

    Diffusion generation requires iterative denoising, and further per-sample acceleration faces increasing approximation sensitivity and cache mismatch.

    Source: §1 · p. 1

  • Method

    ManiFusion fuses multiple sample representations before a shared denoising backbone, separates them afterward, and schedules fusion factors by denoising stage.

    Source: §1 · p. 2

  • Results

    Across diverse diffusion settings, ManiFusion composes with existing acceleration strategies and reaches higher-throughput regimes than those strategies alone.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    ManiFusion adds a complementary throughput axis by increasing the number of samples processed per denoising evaluation.

    Source: §6 · p. 10

    ManiFusion requires full pretraining, preventing lightweight finetuning-based adaptation.

    Source: §6 · p. 10

Abstract

from arXiv · show

Diffusion models achieve high-quality image generation but remain computationally expensive due to iterative denoising across many timesteps. Existing acceleration methods mainly reduce per-sample latency, but gains along this axis are becoming saturated. We instead explore an orthogonal throughput-oriented direction by processing multiple samples within a single denoising evaluation. To this end, we propose ManiFusion, a backbone-agnostic framework for multi-sample generation through fused-space computation. ManiFusion jointly processes multiple samples by fusing them before the denoising backbone and separating them afterward through unfusion. We provide a geometric perspective on how multiple samples can share a fused representation while retaining the information needed to guide their individual generation trajectories. To better align fusion allocation with denoising stages, ManiFusion uses Trajectory-Fusion Scheduling (TFS), which dynamically adjusts fusion factors throughout sampling. ManiFusion generalizes across diffusion families, including both U-Net and transformer backbones in pixel and latent spaces. We further show that ManiFusion composes with existing acceleration methods and reaches throughput regimes unattainable by prior approaches with minimal quality degradation.

1 Introduction

ManiFusion targets throughput rather than further per-sample acceleration, fusing multiple samples before a shared denoising backbone and separating them afterward. Its geometric perspective and trajectory-aware fusion schedule support shared computation, while experiments span multiple diffusion settings and acceleration methods.

1 Introduction

  • Further per-sample acceleration is increasingly difficult because fewer denoising evaluations make each update more sensitive to approximation error, while wider cache intervals increase trajectory mismatch..Source: §1 · p. 1
  • ManiFusion fuses sample representations before a shared denoising backbone, then uses index-conditioned unfusion; TFS uses larger fusion factors in stable regions and smaller ones in sensitive stages..Source: §1 · p. 2
  • Fusion must preserve distinct sample-wise prediction targets despite combining sample states into a representation the size of a single input.The paper analyzes this challenge geometrically under the manifold hypothesis, relating fusion-induced information loss to prediction error.Source: §1 · p. 2
  • Across pixel-space U-Nets, diffusion transformers, and latent diffusion, ManiFusion composes with step reduction and caching to reach higher-throughput regimes than either alone.The experiments cover unconditional and conditional generation from CIFAR-10 to ImageNet.Source: §1 · p. 2

2 Method

ManiFusion replaces independent denoising evaluations with shared fused-space computation followed by index-conditioned recovery. It combines this design with trajectory-dependent fusion scheduling and a geometric analysis of when sample-wise predictions remain recoverable.

2.1 ManiFusion

  • ManiFusion fuses multiple sample representations before shared denoising and uses index-conditioned unfusion afterward to recover individual predictions.The framework couples the original sample space with a shared fused space.Source: §2.1 · pp. 2–3

2.2 Instantiation Across Architectures and Generative Flows

  • ManiFusion is instantiated for U-Nets, diffusion transformers, and autoencoders, with fusion and unfusion adapted to their representations and denoising designs.The implementations cover DDPM, DDIM, and Flow Matching; latent-space fusion keeps K fixed during sampling.Source: §2.2 · pp. 3–4

2.1 ManiFusion

  • TFS varies fusion factors with trajectory dynamics: DDPM/DDIM use larger factors early, while Flow Matching uses larger factors at smaller scheduler intervals.The stated DDPM allocation is 50% of timesteps at K=4, 25% at K=2, and 25% at K=1; this schedule has idealized 2× throughput.Source: §2.2 · p. 4, §2.3 · pp. 4, 6

2.3 Geometric Analysis

  • Fusion-related excess prediction error separates into information lost through collisions and prediction error on retained information; joint training can address both through FK and Gt.Collisions cause irreducible error when they hide differences in required predictions.Source: §2.3 · p. 5
  • Clean-tuple recovery requires D ≥ Kd; index-conditioned slot maps distinguish sample variations, and local separation limits inverse amplification to 1/c.Here d is each clean sample’s intrinsic manifold dimension, while D is the shared representation dimension.Source: §2.3 · p. 5
  • With nonzero Gaussian noise, a tuple has full support in R^KD, so continuous fusion into R^D cannot distinguish every tuple when K > 1; shared prediction instead requires target agreement within fibers.For velocity targets, small variation of the conditional mean velocity within a fiber is the stated compatibility condition.Source: §2.3 · pp. 5–6
  • For noise prediction, the optimal target at the pure-noise limit equals the noisy input, unlike clean-state prediction, so the information fusion must retain depends on target parameterization.The passage states that noise level and target parameterization together determine the information required by fusion.Source: §2.3 · p. 6

3 Experiments

ManiFusion is evaluated across diffusion backbones, spaces, and acceleration settings. Results report near-2× throughput with quality close to baseline, and show benefits when fusion is combined with step reduction or caching.

3 Experiments

  • Experiments span pixel-space U-Nets, pixel-space transformers, and latent-space autoencoders and transformers, including conditional and unconditional generation.Evaluations cover CIFAR-10, CelebA, CelebA-HQ, and ImageNet.Source: §3 · p. 7
  • Throughput comparisons use identical inference environments and end-to-end wall-clock generation time, with parameter count and MACs measuring computational overhead.Unless otherwise specified, ManiFusion results use fusion factors up to K=4 and models trained from scratch.Source: §3 · p. 7

3.1 Application to Baseline Models

  • Baseline diffusion models reach nearly 2× throughput with sample quality close to baseline and no material increase in model complexity.The comparison is reported in Table 1; qualitative appendix results are said to confirm perceptual fidelity and diversity.Source: §3.1 · p. 7
  • On latent diffusion, MF-AE exhibits smaller quality degradation than MF-UNet at the same fusion level.The comparison highlights the role of where the shared bottleneck is introduced; Table 3 covers MF-AE on CelebA-HQ and MF-DiT on ImageNet.Source: §3.1 · p. 7, Table 3 · p. 8

3.2 Orthogonal Acceleration Beyond Existing Throughput Limits

  • At matched throughput, ManiFusion with DeepCache consistently outperforms configurations relying solely on step reduction and caching.Table 2 covers CIFAR-10 and CelebA; its caption identifies MAC reduction ratios and throughput speedups in parentheses.Source: Table 2 · p. 8, §3.2 · p. 8
  • ManiFusion maintains higher sample quality than step-reduction methods at matched throughput, while baselines degrade beyond their practical throughput ceiling.The authors attribute the high-throughput advantage to trajectory-dependent fusion allocation.Source: §3.2 · p. 8

4 Analysis

The analysis tests whether fusion retains prediction distinctions, compatible dynamics, and slot-specific separation during training. Results show broad prediction differences are preserved, while finer distinctions and late-stage dynamics can be more affected.

4.1 Empirical Validation of the Geometric Analysis

  • Random-pair prediction-distance ratios concentrate near one, while local ratios shift downward as K increases, indicating preserved broad differences but attenuated fine distinctions.Ratios near one indicate prediction separation comparable to the single-sample model.Source: §4.1 · p. 9
  • Near-fiber pairs have more similar vector-field directions, while alignment with single-sample dynamics declines near the end, particularly for K = 8.These observations motivate sharing more computation early and restoring sample-specific capacity during final refinement.Source: §4.1 · p. 9

4.2 Training Dynamics of the Fusion Module

  • Fusion Weight Orthogonality begins decreasing after approximately 50k training steps as streams develop distinct color patterns, indicating increasingly orthogonal fusion weights.The authors interpret this as the fusion module allocating independent slot-specific directions to samples.Source: §4.2 · pp. 9–10

5 Related Work

Prior diffusion acceleration reduces sampling steps or per-step cost, while prior superposition work processes shared representations in other settings. ManiFusion applies superposition to generative trajectories to generate multiple images in one inference process.

Diffusion Models

  • Pixel-space diffusion, latent diffusion, and transformer-based diffusion differ in denoising representation and backbone, while all retain representational slack for fused multi-sample computation.DiT-scale latent representations can improve performance while increasing inference cost.Source: §5 · p. 10

Diffusion Model Acceleration

  • Diffusion acceleration reduces denoising steps or per-step latency, but those methods still process each trajectory independently and do not raise multi-sample throughput within one evaluation.DDIM and flow-matching methods exemplify step reduction; DeepCache and CacheMe reuse activations across timesteps.Source: §5 · p. 10

Superposition in Deep Learning

  • Earlier superposition work covers token processing and classification, whereas ManiFusion fuses signals within a latent trajectory to generate multiple images in one inference process.The cited examples include small-scale image classification such as MNIST and CIFAR-10.Source: §5 · p. 10

6 Conclusion

The paper concludes that ManiFusion adds a throughput axis by processing multiple samples per denoising evaluation, with geometric analysis and trajectory-aware fusion allocation supporting the approach. It also identifies full pretraining, unevaluated large-scale text-to-image models, and degradation at extreme fusion factors as limitations.

6 Conclusion

  • ManiFusion increases throughput by processing multiple samples per denoising evaluation, and combines with existing acceleration strategies to reach difficult-to-achieve throughput regimes.The conclusion frames trajectory-aware fusion allocation as balancing throughput and generation quality.Source: §6 · p. 10
  • The method requires full pretraining, large-scale text-to-image foundation models are not evaluated, and performance can degrade under extreme fusion factors.The authors note substantial computational cost and no lightweight finetuning-based adaptation.Source: §6 · p. 10

A Appendix Overview

  • The appendix covers implementation configurations, geometric derivations and empirical analyses, extensions, ablations, throughput comparisons, and qualitative results.Implementation details specify fixed-K training, inference-time TFS using models trained at different fusion factors, and evaluation on 50k generated samples.Source: A Appendix Overview · p. 13, B Implementation Details · p. 13

C Derivations for the Geometric Analysis

The derivations frame fusion as retaining prediction-relevant target information, not necessarily uniquely identifying every input tuple. They connect clean-state recovery to representation capacity and noisy prediction to target compatibility within fusion fibers, motivating trajectory-dependent fusion allocation.

Conditional means and the risk identities

  • For fixed fusion, excess prediction risk is the conditional variance of optimal targets within each fusion fiber; it vanishes exactly when the target is determined by the fused representation.End-to-end training can change both fiber ambiguity and recovery of retained targets.Source: C Derivations for the Geometric Analysis · p. 15

Clean-state prediction as an interpretable special case

  • Smooth local recovery of a complete clean tuple requires D ≥ Kd; for MF-AE, the analogous ideal shared latent dimension must satisfy D_AE ≥ Kd.When D < Kd, at least one intrinsic direction is lost; capacity alone does not ensure recovery, which also depends on local separation.Source: C Derivations for the Geometric Analysis · pp. 15–16

Noise, target compatibility, and TFS

  • At noisy timesteps, fusion cannot be globally injective for K > 1, so accurate prediction requires compatible optimal targets among tuples sharing a fused representation.For velocity prediction, this means low variation in slot-wise conditional-mean vector fields across tuples in the same fiber.Source: C Derivations for the Geometric Analysis · p. 16
  • TFS allocates fusion according to prediction target and trajectory: larger fusion factors suit smaller Flow Matching intervals, while DDPM/DDIM retain more per-sample capacity near final refinement.Velocity error scales with the signed update interval, whereas target parameterization affects the information demanded at a given noise level.Source: C Derivations for the Geometric Analysis · pp. 16–17

Conditional grouping and slot separation

  • Conditional diversity provides additional cues for assigning predicted features to slots, complementing cross-tuple target compatibility; same-condition groups can still separate through distinct noisy states and targets.The former concerns within-tuple slot separation, while the latter concerns tuples sharing a fused representation.Source: C Derivations for the Geometric Analysis · p. 17

Scope of the analysis

  • The analysis covers fixed-timestep population risk under squared loss and assumes smooth maps for clean-manifold capacity; learned-model geometry and cumulative generation effects are assessed empirically.Empirical analyses use learned reference fields and finite perturbations, while generation quality and cumulative fusion effects are evaluated with sampling metrics and TFS ablations.Source: C Derivations for the Geometric Analysis · p. 17

D Empirical Geometric Analysis

Toy experiments test whether fusion preserves clean-manifold recovery, reference geometry, responsive denoising, and coherent interpolation. Capacity and local-response findings align with the geometric analysis, while visualizations show smoother interpolation and comparable feature-space coverage in the evaluated settings.

Toy experiment

  • Recovery fails when D cannot represent the fused product manifold, while stable separation is observed in evaluated configurations with D ≥ Kd.This empirical transition supports representation capacity as a condition for clean-sample recovery.Source: D Empirical Geometric Analysis · p. 17

Representation alignment

  • Aligned fused geometry closely matches the single-sample manifold structure in recoverable regimes.An affine alignment substantially reduces the discrepancy between the fused geometry and the single-sample reference; Figure 4 reports RMSEs of 4.83 × 10−4 for (D=16, K=2), 7.03 × 10−4 for (D=16, K=4), and 7.8 × 10−5 for (D=32, K=8).Source: D Empirical Geometric Analysis · p. 17, Figure 4 · p. 18

Local Smoothness and Empirical Lipschitz Analysis

  • For K=2 and K=4, empirical local response ratios remain clearly above zero and well controlled, while K=8 remains positive but is noticeably smaller.The authors interpret the K=8 result as attenuated local prediction differences and link it to reducing fusion during sensitive refinement stages.Source: D Empirical Geometric Analysis · p. 18, Figure 5 · p. 18

Interpolation Behavior under ManiFusion

  • On CelebA, ManiFusion (K=2) produces smoother identity transitions and fewer abrupt mode switches than the K=1 baseline.The examples interpolate noise and run the reverse process; the figure also depicts interpolation across pose and appearance factors.Source: D Empirical Geometric Analysis · p. 19, Figure 6 · p. 19

Manifold Coverage under ManiFusion

  • ManiFusion slightly reduces precision while recall remains comparable to baseline, and its generated samples occupy a similar displayed feature-space region.Recall and the t-SNE visualization support comparable sample diversity in the evaluated setting.Source: D Empirical Geometric Analysis · p. 19, Figure 7 · p. 19

E Extensions of ManiFusion

The extensions examine larger fusion factors with TFS, combinations with DeepCache, ImageNet scaling, and a latent-diffusion design. The reported results indicate that TFS can use larger fusion factors with less severe quality degradation, while MF-UNet trades throughput gains for fidelity loss as fusion increases.

Large-Fusion Extension and Trajectory-Fusion Scheduling

  • Uniform K=8 scheduling achieves higher throughput and better generation quality than schedules using only {4, 2, 1}, while large fixed fusion factors can significantly degrade quality despite throughput gains.For MF-DiT, ManiFusion (-TFS, K=4) achieves throughput gains similar to increasing patch size as in DiT-S/4 while preserving substantially better sample quality.Source: E Extensions of ManiFusion · p. 20

Combination with DeepCache and ImageNet scaling

  • The supplied extension passages identify DeepCache combination and ImageNet scaling comparisons, but do not report their outcome values.Table 6 covers ManiFusion with and without TFS combined with DeepCache; Table 7 covers fixed fusion factors and TFS on ImageNet using DiT-S/2.Source: E Extensions of ManiFusion · p. 21, Table 7 · p. 21

Extension to Latent Diffusion Models

  • MF-UNet in latent diffusion provides throughput gains up to 3.87×, but generation quality degrades noticeably as fusion factor increases.MF-UNet consistently yields worse fidelity than MF-AE across metrics such as FID and sFID.Source: E Extensions of ManiFusion · p. 21, Table 8 · p. 22

F Analysis of Fusion Behavior

The analysis distinguishes separating individual slots within a fused group from maintaining compatible predictions across different tuples that share a representation. In MF-DiT, mixing class conditions within groups outperforms grouping samples from one class.

Conditional Grouping and Intra-Tuple Separability

  • Randomly mixing class conditions consistently outperforms uni-class grouping, supporting the role of conditional diversity in providing slot-specific cues for index-conditioned unfusion.Same-condition grouping remains effective because trajectories retain distinct noisy states and targets, though fewer condition-specific cues can make separation harder.Source: F Analysis of Fusion Behavior · p. 22, Table 9 · p. 22

G Ablation Study

Larger fusion kernels increase spatial entanglement and degrade generation quality; compact kernels, such as size 3, perform best across the reported metrics.

G Ablation Study

  • Larger fusion kernels consistently degrade generation quality across FID, sFID, and IS.The authors attribute this to stronger spatial entanglement, which mixes inputs more aggressively and makes unfusion harder.Source: G Ablation Study · p. 22
  • Smaller kernels reduce entanglement and preserve sample quality under fusion.The paper identifies size 3 as the best-performing kernel size across FID, sFID, and IS.Source: Table 10 · p. 22, G Ablation Study · p. 22
  • Compact kernels perform best across FID, sFID, and IS.The table caption specifically names size 3 as an example of the smaller kernels with the best performance.Source: Table 10 · p. 22

H Comparison with Throughput-Oriented Baselines

ManiFusion outperforms naive patch compositing and capacity-reduced models in the reported comparisons. It preserves quality against patch-based throughput increases and achieves better quality with lower MAC counts and higher throughput than the tested smaller backbones.

H Comparison with Throughput-Oriented Baselines

  • The naive patch-based strategy downsamples each input to one-quarter resolution, arranges four inputs in a 2×2 grid, then splits and upsamples the output.The comparison reports artifacts, unnatural boundaries, and facial or background blur for patch compositing, while ManiFusion preserves texture and color consistency.Source: H Comparison with Throughput-Oriented… · p. 23, Table 11 · p. 23
  • On CIFAR-10, ManiFusion's FID is 18.27 versus 91.71 for patch compositing, and its IS is 7.41 versus 4.55.On CelebA, ManiFusion's FID is 9.50 versus 26.07 for the grid approach, and its IS is 2.58 versus 2.37.Source: H Comparison with Throughput-Oriented… · p. 23, Table 12 · p. 23
  • ManiFusion with the full backbone at K = 2 has FID 5.93, versus 7.59 for the channel-reduced model and 6.55 for the depth-reduced model.The reduced models also require more MACs and achieve lower throughput than the ManiFusion reference; the comparison covers CIFAR-10 DDPM.Source: H Comparison with Throughput-Oriented… · p. 23, Table 14 · p. 24
  • In the tested configurations, sharing the full backbone through fusion achieves better quality at lower MAC counts and higher throughput than reducing channels or depth.The comparison frames fusion as sharing full-backbone computation across samples rather than reducing backbone capacity.Source: H Comparison with Throughput-Oriented… · p. 24, Table 14 · p. 24

I Qualitative Results

This section presents qualitative comparisons of ManiFusion across samplers, fusion designs, and throughput levels, alongside checklist notes on reproduction resources, statistical testing, societal-impact discussion, and LLM involvement.

I Qualitative Results

  • Tables 15–16 show ManiFusion samples across DDPM, DDIM, and Flow Matching for K = 1, 2, 4, and TFS on CIFAR-10 and CelebA, with coherence, diversity, and fidelity noted.These visual comparisons complement quantitative sample-quality evaluation across samplers and fusion factors.Source: Table 15 · p. 25, I Qualitative Results · p. 25, Table 16 · p. 26
  • Table 17 compares vanilla LDM, MF-AE, and MF-UNet at K = 2 and K = 4; ImageNet samples are omitted because baseline DiT-S/2 FID exceeds 50.The authors describe visual comparison as uninformative for that ImageNet configuration.Source: I Qualitative Results · p. 26, Table 17 · p. 26
  • Tables 18–19 compare baseline, fixed fusion (K = 2), and TFS at matched throughput levels on CIFAR-10 and CelebA using DDIM.The tables show random samples at 10, 20, 50, and 100 timesteps; reported throughput spans 66.80–565.03 img/s on CIFAR-10 and 26.83–228.77 img/s on CelebA.Source: I Qualitative Results · p. 27, Table 18 · p. 27, Table 19 · p. 27

NeurIPS Paper Checklist

  • The checklist says the paper provides experimental settings and key model and performance information, supplementary code, and single-NVIDIA-A6000 measurements.It also says geometric-analysis assumptions and proofs appear in Sections 2.3 and Appendix C, respectively.Source: NeurIPS Paper Checklist · pp. 29–31
  • The authors report no repeated experimental runs or error bars, despite using 50k generated images per setting, and no separate discussion of societal impacts.They state that limitations are discussed at the end of the conclusion.Source: NeurIPS Paper Checklist · pp. 28, 30, 31, 32
  • The checklist marks LLM involvement as N/A because LLMs are not important, original, or non-standard components of the method development.Source: NeurIPS Paper Checklist · p. 34
Loading 2610.21433v1…