PaperlayerView exact revision on arXiv

Source-linked AI summary

Paris: A Decentralized Trained Open-Weight Diffusion Model

Zhiying Jiang, Raihan Seraj, Marcos Villagra, Bidhan Roy

arXiv:2510.03434v3cs.GRcs.DCcs.LG

TL;DR

Large-scale diffusion training relies on synchronized updates across thousands of GPUs, limiting access to dedicated infrastructure and specialized interconnects. Paris trains isolated experts on data partitions and routes among them at inference. The authors report comparable generation quality to the prior decentralized baseline with 1/14 of its training data and 1/16 of its GPU-days.

  • Problem

    Large-scale diffusion training requires synchronized updates across thousands of GPUs and high-bandwidth interconnects, limiting training to institutions with massive infrastructure and hindering use of distributed commodity hardware.

    Source: §1 · p. 2

  • Method

    Paris trains eight expert diffusion models independently on semantically coherent data partitions and uses a lightweight router to select experts at inference.

    Source: §1 · p. 2

  • Results

    Paris achieves comparable generation quality to DDM (McAllister et al., 2025) using 1/14 of the training data (11M vs. 154M images) and 1/16 of the GPU-days.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Fully decentralized training maintains generation quality while enabling development on fragmented compute resources and removing the need for specialized interconnects.

    Source: §1 · p. 2, §4 · p. 11

    DDM (McAllister et al., 2025) reports no single FID on LAION-Aesthetic; the cited 5.5–10.5 range is estimated from results at varying training FLOPs.

    Source: §3.1.2 · p. 10

Abstract

from arXiv · show

We present Paris, the first publicly released diffusion model pre-trained entirely through decentralized computation. Paris demonstrates that high-quality text-to-image generation can be achieved without centrally coordinated infrastructure. Paris is open for research and commercial use. Paris required implementing our Distributed Diffusion Training framework from scratch. The model consists of 8 expert diffusion models (129M-605M parameters each) trained in complete isolation with no gradient, parameter, or intermediate activation synchronization. Rather than requiring synchronized gradient updates across thousands of GPUs, we partition data into semantically coherent clusters where each expert independently optimizes its subset while collectively approximating the full distribution. A lightweight transformer router dynamically selects appropriate experts at inference, achieving generation quality comparable to centrally coordinated baselines. Eliminating synchronization enables training on heterogeneous hardware without specialized interconnects. Empirical validation confirms that Paris's decentralized training maintains generation quality while removing the dedicated GPU cluster requirement for large-scale diffusion models. Paris achieves this using 14$\times$ less training data and 16$\times$ less compute than the prior decentralized baseline.

1. Introduction

Paris addresses the infrastructure and synchronization barriers to large-scale diffusion training with independently trained experts and inference-time routing. The authors report comparable generation quality using substantially less training data and compute than the prior decentralized baseline.

1. Introduction

  • Large-scale diffusion training requires synchronized updates across thousands of GPUs and high-bandwidth interconnects, creating barriers to institutions without dedicated clusters and to distributed commodity hardware.The cited examples include Stable Diffusion’s 150,000 A100 GPU-hours and Imagen’s use of hundreds of TPU-v4 chips.Source: §1 · p. 2
  • Paris trains eight diffusion experts without exchanging gradients, parameters, or intermediate activations, then uses a lightweight router to coordinate them at inference.The framework partitions data into semantically coherent clusters so experts optimize locally while the ensemble approximates the full distribution.Source: §1 · p. 2
  • Paris maintains generation quality while enabling training on fragmented compute resources and removing the need for specialized interconnects and dedicated clusters.The authors describe this as empirical validation of fully decentralized training.Source: §1 · p. 2
  • Paris uses 1/14 of the training data (11M vs. 154M images) and 1/16 of the GPU-days while achieving comparable generation quality to DDM (McAllister et al., 2025).The work also reports a complete open-source implementation for heterogeneous, geographically distributed hardware.Source: §1 · p. 2

2. Method

Paris partitions training data into clusters, trains experts independently, and uses a lightweight router to combine them at sampling. Its implementation uses Diffusion Transformers at two model scales and routes from partially denoised latents.

2.1. Overview

  • Paris partitions the training set into clusters, trains one expert in isolation on each partition, and uses a lightweight router to fuse experts at inference.The overview lists latent pre-encoding, DINOv2 semantic features and clustering, expert denoiser training, independent router training, and expert selection during sampling.Source: §2.1 · p. 2

2.2. Implementation Scale and Architecture

  • Diffusion Transformers are used for their scaling properties, with initialization schemes intended to support convergence without coordination.Paris extends decentralized flow matching theory (Lipman et al., 2023; McAllister et al., 2025), DiT (Peebles and Xie, 2022), and PixArt-α (Chen et al., 2023).Source: §2.2 · p. 3
  • The framework is validated with DiT-B/2 experts of 129M parameters and DiT-XL/2 experts of 605M parameters, using 11M LAION-Aesthetic images clustered by DINOv2 embeddings.Across eight experts, the configurations total 1.03B and 4.84B parameters, respectively; clusters specialize in distinct visual domains.Source: §2.2 · p. 3
  • The router selects experts from partially denoised latents throughout reverse diffusion, so it must identify suitable experts from noisy intermediate states.The passage connects this noise-aware routing with velocity-prediction flow matching and initialization strategies for stable convergence without inter-expert coordination.Source: §2.2 · pp. 3–4

2.3. Decentralized Flow Matching Objective

The decentralized flow-matching objective decomposes the marginal flow across experts trained on disjoint data partitions. A router weights each expert’s predicted flow according to its posterior probability for the current noisy latent.

2.3. Decentralized Flow Matching Objective

  • The objective decomposes standard flow matching across K expert models, each trained on a disjoint data partition.The marginal flow transports samples from the noise distribution to the data distribution.Source: §2.3 · p. 4
  • The noisy latent is formed from the clean latent and noise at timestep t, with α_t = 1 − t and σ_t = t under linear scheduling.The timestep ranges over [0, 1].Source: §2.3 · p. 4
  • Each expert predicts a flow from its own cluster, and the router supplies the posterior probability for that expert given the noisy latent.This formulation allows experts to optimize independently.Source: §2.3 · p. 4
  • Each expert’s training target is the velocity direction from the noisy latent back to a clean sample, with clean samples drawn from that expert’s cluster and timesteps sampled uniformly.The predicted velocity field is evaluated at the noisy latent and timestep.Source: §2.3 · p. 4

2.4. Expert Models

Paris’s experts use a latent-space Diffusion Transformer backbone. Timestep-conditioned adaptive normalization modulates transformer blocks, and an optional AdaLN-Single variant reduces parameter count by sharing conditioning signals across blocks.

2.4. Expert Models

  • DiTExpert adapts the Diffusion Transformer for decentralized diffusion, providing the backbone for each expert with greater scalability than traditional U-Net architectures.Its design incorporates innovations from DiT and PixArt-α while aiming to maintain computational efficiency.Source: §2.4 · p. 4

2.4.1. Core Architecture

  • The expert operates on 32 × 32 latent tensors with four channels, encoded from 256 × 256 images by a pretrained VAE with 8× spatial downsampling.Fixed sinusoidal positional embeddings preserve spatial relationships between patches, and a timestep embedder injects temporal information.Source: §2.4.1 · pp. 4–5

2.4.2. Transformer Blocks with Adaptive Layer Normalization

  • Adaptive Layer Normalization modulates normalized activations using scale and shift parameters predicted from the timestep conditioning signal.This lets the model adapt across noise levels without increasing sequence length; text-conditioned blocks can also insert cross-attention between self-attention and the feed-forward network.Source: §2.4.2 · p. 5

2.4.3. Parameter-Efficient AdaLN-Single Variant

  • The optional AdaLN-Single variant computes conditioning signals once globally and uses learned per-block embeddings to reduce parameters.Each block has six modulation parameters in the learned embeddings.Source: §2.4.3 · p. 5

2.5. Router Networks

DiTRouter selects among independently trained experts from noisy latents during denoising, with routing conditioned on timestep and trained to predict each sample’s cluster assignment.

2.5. Router Networks

  • DiTRouter classifies noisy inputs to select experts dynamically, supporting hard top-1 routing or soft ensemble strategies during inference.It processes noisy latents at arbitrary timesteps rather than requiring clean inputs or a separate encoder.Source: §2.5 · p. 5, §2.5.1 · p. 6

2.5.1. Core Architectural

  • The router uses a reduced-capacity DiT-B/2 transformer, processing expert-compatible latent representations at around one-third the size of each expert.The architecture is designed to minimize routing overhead.Source: §2.5.1 · pp. 5–6

2.5.2. Timestep-Aware Processing

  • Timestep-aware conditioning lets the router adapt expert classification to the denoising stage, since expert specialization may vary across noise levels.The router uses the same sinusoidal timestep embedding and MLP projection as the experts.Source: §2.5.2 · p. 6

2.5.3. Training Objective and Loss Function

  • The router is trained with cross-entropy against clean-sample cluster assignments while learning to identify the correct expert from noisy inputs.Training samples timesteps uniformly and applies the corresponding noise schedule to expose the router to noise levels throughout the reverse diffusion process.Source: §2.5.3 · p. 7

2.6. DINOv2-Based Semantic Clustering

Paris uses DINOv2 features and hierarchical clustering to partition images into semantic clusters for expert specialization.

2.6. DINOv2-Based Semantic Clustering

  • DINOv2-ViT-L/14 extracts 1024-dimensional features from center-cropped 224 × 224 images to guide semantic data partitioning.DINOv2 is a self-supervised vision transformer trained on 142M images.Source: §2.6 · p. 7
  • A two-stage hierarchical clustering process first uses fine-grained k-means centroids, then consolidates them into K coarse clusters, typically K = 8.The approach follows McAllister et al. (2025).Source: §2.6 · p. 7
  • The resulting semantically guided partitions provide the basis for expert specialization.DINOv2 features are used to guide expert specialization through data partitioning.Source: §2.6 · p. 7

2.7. Inference Strategies for Decentralized Generation

Paris combines router-guided expert predictions during reverse diffusion through top-1, top-K, or full-ensemble strategies, trading off quality, computational cost, and diversity.

2.7.1. Top-1 Expert Selection

  • Top-1 selection sends each denoising step to the single expert with the router’s highest-confidence prediction, requiring one expert forward pass per step.This hard-routing strategy has constant computational cost regardless of ensemble size.Source: §2.7.1 · p. 7
  • In experiments on smaller networks, Top-1 routing achieves competitive quality, while Top-2 routing surpasses both monolithic and full-ensemble approaches.Hard routing maintains constant computational cost regardless of ensemble size, requiring only one expert forward pass per denoising step.Source: §2.7.1 · p. 7

2.7.2. Top-K Weighted Ensemble

  • Top-K routing combines predictions from the most relevant experts using renormalized router probabilities, balancing computational efficiency with ensemble benefits.Per-sample selection allows different samples in a batch to use different expert combinations.Source: §2.7.2 · pp. 7–8

2.7.3. Full Ensemble Integration

  • Full-ensemble integration weights all experts by router probabilities and requires K forward passes per step, but can yield diminishing returns or degraded quality.The passage suggests less-relevant experts may add noise to predictions.Source: §2.7.3 · p. 8

2.8. Comparison with Parallelization Strategies

Paris trains experts without inter-expert communication, unlike distributed strategies that retain synchronization or coordination requirements.

2.8. Comparison with Parallelization Strategies

  • Data, model, and pipeline parallelism impose distinct synchronization patterns, including all-reduce updates, layer dependencies, or pipeline fill-and-drain idle time.Local SGD and DiLoCo synchronize every N iterations rather than every iteration.Source: §2.8 · p. 8
  • Paris’s zero-communication recipe enables training on heterogeneous infrastructure using 120 A40 GPU-days, while centralized approaches require thousands of GPU-days and specialized interconnects.The passage states that modern optimizations reduce synchronization overhead but do not eliminate worker coordination requirements.Source: §2.8 · p. 8
  • Paris uniquely eliminates all communication overhead among the compared parallelization strategies by training experts independently without gradient synchronization, parameter sharing, or activation exchange.Experts can train asynchronously across heterogeneous infrastructure without communication.Source: §2.8 · p. 8, Table 1 · p. 9

3. Experiments and Results

Paris is evaluated at two expert scales and against monolithic and decentralized baselines. Its DiT-XL/2 comparison reports substantially lower training-resource use than DDM alongside a higher FID, while DiT-B/2 results favor selective Top-2 routing over monolithic training and full ensembling.

3.1. Training Details

  • The experiments use DiT-B/2 experts with 129M parameters and DiT-XL/2 experts with 605M parameters, across ensembles of eight experts.The corresponding total ensemble sizes are 1.0B and 4.84B parameters, respectively.Source: §3.1 · p. 9

3.1.1. DiT-B/2 Configuration

  • The DiT-B/2 configuration trains on 256 × 256 images represented as 32 × 32 VAE latents, while its separately trained DiT-S router uses the full dataset.Expert training uses an effective batch of 256; router training also uses an effective batch of 256 and runs for 25 epochs.Source: §3.1.1 · p. 9

3.1.2. DiT-XL/2 Configuration

  • The DiT-XL/2 configuration uses a DiT-B router, and PixArt-α optimizations reduce its expert parameter count from 0.89B to 0.6B.The DDM LAION-Aesthetic FID range of 5.5–10.5 is estimated from results at varying training FLOPs.Source: §3.1.2 · pp. 9–10

3.2. Results

  • FID-50K was 22.60 for Top-2 routing, 29.64 for monolithic training, and 47.89 for the Full Ensemble on Laion-art.All comparisons use DiT-B/2; lower FID indicates better generation quality.Source: Table 2 · p. 10, §3.2 · p. 10, §3.1.2 · p. 10
  • On LAION-Aesthetic, Paris reports FID 12.45 using 11M training images and 120 A40 GPU-days (~72 A100 equivalent), while DDM reports FID 5.5–10.5 using 154M images and ~1176 A100 GPU-days.Paris uses 14× fewer training images and 16× fewer GPU-days, with 1.19–2.26× higher FID.Source: §3.2 · p. 10

4. Conclusion

Paris demonstrates that competitive text-to-image synthesis can be trained without gradient synchronization. The work establishes the feasibility and practical implementation patterns for decentralized diffusion training, with extensions to video generation and deployment efficiency left for future work.

4. Conclusion

  • Paris demonstrates competitive text-to-image synthesis through fully decentralized computation without gradient synchronization.The authors identify model distillation for deployment efficiency, video-generation adaptation, and incorporation of architectural advances as future work.Source: §4 · p. 11

Related papers

Loading 2510.03434v3…