PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffCollage: Parallel Generation of Large Content with Diffusion Models

Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, Ming-Yu Liu

arXiv:2303.17076v1cs.CVcs.LG

TL;DR

Large-content generation can be difficult when target-scale datasets are unavailable or costly, and autoregressive generation has sequential limitations. DiffCollage combines diffusion models trained on smaller pieces through a factor graph and generates the pieces in parallel. Experiments across several tasks report that it outperforms existing approaches by a wide margin.

  • Problem

    Large-content tasks may lack affordable target-scale datasets, while autoregressive piecewise generation is sequential and can accumulate errors.

    Source: §1 · p. 1

  • Method

    DiffCollage uses a factor graph to combine diffusion-model outputs for overlapping content pieces, which are generated in parallel.

    Source: §1 · p. 1, §4 · p. 3

  • Results

    Across infinite-image, long-duration text-to-motion, looped-motion, and 360-degree-image tasks, experiments report that DiffCollage outperforms existing approaches by a wide margin.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Large-content generation can use diffusion models trained on marginal data, which are easier to obtain than samples from the joint distribution.

    Source: §6 · p. 8

    The Bethe approximation is exact for acyclic factor graphs; for graphs with cycles, it remains an approximation.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

We present DiffCollage, a compositional diffusion model that can generate large content by leveraging diffusion models trained on generating pieces of the large content. Our approach is based on a factor graph representation where each factor node represents a portion of the content and a variable node represents their overlap. This representation allows us to aggregate intermediate outputs from diffusion models defined on individual nodes to generate content of arbitrary size and shape in parallel without resorting to an autoregressive generation procedure. We apply DiffCollage to various tasks, including infinite image generation, panorama image generation, and long-duration text-guided motion generation. Extensive experimental results with a comparison to strong autoregressive baselines verify the effectiveness of our approach.

1. Introduction

DiffCollage targets large-content generation when target-scale datasets are unavailable or costly, using models trained on smaller pieces. It combines those pieces through a factor-graph approach and reports broad improvements over existing approaches.

1. Introduction

  • Target-scale datasets may be unavailable or prohibitively expensive to collect, even when smaller pieces such as ordinary perspective images are plentiful.The introduction gives 360-degree panoramas as an example of rare target content with abundant perspective-image pieces.Source: §1 · p. 1
  • Autoregressive piecewise generation is sequential, can accumulate errors, and makes loop start-to-end consistency difficult to enforce.Later-generated pieces cannot influence earlier ones, and generation time grows linearly with content size.Source: §1 · p. 1
  • DiffCollage represents content with a factor graph whose nodes cover contiguous portions with small overlaps, then merges outputs from piece-trained diffusion models.Each node's piece affects the generation of other pieces, which are generated in parallel.Source: §1 · pp. 1–2
  • Experiments across infinite-image, long-duration text-to-motion, looped-motion, and 360-degree-image tasks report that DiffCollage outperforms existing approaches by a wide margin.The introduction describes extensive benchmark results as showing effectiveness across these tasks.Source: §1 · pp. 1–2
  • DiffCollage can use pretrained diffusion models on different pieces out of the box, and its parallel generation is presented as efficient.The introduction also characterizes the approach as generic and applicable to multiple large-content tasks.Source: §1 · p. 2

2. Related Work

Related work covers diffusion models across generation tasks and several strategies for producing large content from models trained on smaller pieces. These include latent-variable approaches and autoregressive outpainting.

2. Related Work

  • Diffusion models have been applied to image generation, time series, point clouds, natural language processing, editing, inpainting, and adversarial defense.Prior work has also targeted improvements in generation quality, controllability, and efficiency.Source: §2 · p. 2
  • Latent-variable approaches use global and spatial codes to generate large images, including patch-based methods and one-shot architectures designed for translation equivariance.Some patch-based methods merge outputs from patches generated with the same global code and use a discriminator for coherence.Source: §2 · p. 2
  • Another approach gradually enlarges content through autoregressive outpainting implemented with diffusion models, autoregressive transformers, or masked transformers.The passage cites diffusion outpainting [9] [10] [22] [26], autoregressive transformers, and masked transformers [71].Source: §2 · pp. 2–3

3. Preliminaries

The preliminaries describe diffusion as forward Gaussian noising followed by reverse denoising. The reverse process uses a score function estimated by a neural network.

3. Preliminaries

  • The forward diffusion process progressively injects Gaussian noise into data samples, producing a family of noised distributions.The conditional noised-data distribution is Gaussian with mean equal to the clean data and variance σ_t^2 I.Source: §3 · p. 3
  • The noise standard deviation σ_t increases monotonically with forward diffusion time, while the reverse process iteratively removes noise to recover clean data.The reverse process is formulated as a stochastic differential equation.Source: §3 · p. 3
  • A neural network estimates the score of the noised-data distribution, and η controls the amount of random noise injected during denoising.η = 1 gives the reverse-time SDE, while η = 0 reduces the equation to a probability flow ODE.Source: §3 · p. 3
  • The score network is trained by minimizing a weighted sum of denoising autoencoder objectives, with ω(t) denoting a time-dependent weight.Source: §3 · p. 3

4. Diffusion Collage

DiffCollage represents large content with overlapping factor-graph pieces, approximates its joint score from smaller marginal diffusion models, and generates pieces in parallel. Figure 4 reports high-fidelity images from DiffCollage, while Replace and Recon introduce discontinuity artifacts.

4. Diffusion Collage

  • DiffCollage approximates a large-content distribution using factor-graph nodes for contiguous portions with small overlaps, each associated with a diffusion model.Its score approximation combines factor-node scores and subtracts non-leaf variable-node scores.Source: §4 · p. 3, §4.1 · p. 4

4.1. Representation

  • In the long-image example, outpainting assumes x(1) and x(3) are independent given x(2), while independently estimated scores enable parallel generation.Source: §4.1 · p. 3
  • The representation covers chains for sequences, cycles for loops, overlapping-patch grids for arbitrary images, and face-variable cycles for cubemap panoramas.Source: §4.1 · p. 4

4.2. Training and Sampling

  • Training matches learned marginal scores to real-data marginal scores with denoising score matching; independent marginal models can be learned separately and existing samplers used without modification.Source: §4.2 · pp. 4–5
  • DiffCollage generates high-fidelity long images in parallel, while the autoregressive Replace and Recon approaches introduce discontinuity artifacts.The approaches use diffusion models trained on smaller square images.Source: §4.2 · p. 5

5. Experiments

Experiments span infinite image generation, arbitrary-sized image translation, text-to-motion, and panorama generation. Across these settings, DiffCollage is evaluated for image quality, generation speed, and complex content synthesis.

5.1. Infinite image generation

  • H/2W times and H/W times faster than replacement and reconstruction, respectively, DiffCollage generates H × W images for H ≥ 2W; Tab. 1 reports the best FID+ quality.FID+ is computed on randomly cropped square views of generated panoramas.Source: Table 1 · p. 6, §5.1 · p. 6, Figure 5 · p. 6
  • DiffCollage achieves competitive standard FID and FID+ on LHQ and LSUN Tower despite never being trained for long-image generation.It also achieves higher quality than methods specifically designed for infinite image generation while being more general.Source: §5.1 · p. 6, Table 2 · p. 6
  • Given two 64 × 64 real images, DiffCollage generates a 1024 × 10752 image that transitions between them using interpolated CLIP embeddings for intermediate nodes.The endpoint images can have different styles.Source: §5.1 · p. 6, Figure 6 · p. 7

5.2. Arbitrary-sized image translation

  • Splitting non-square inpainting inputs into two or three square crops creates boundary artifacts, whereas DiffCollage captures global information and faithfully reconstructs images from sparse pixels.Source: Figure 7 · p. 7, §5.2 · p. 7

5.3. Text-to-motion generation

  • DiffCollage outperforms naive long-sequence denoising and autoregressive replacement and reconstruction on all evaluated HumanML3D metrics for 24-second motions.The 24-second sequences are around four times the average length of training data.Source: §5.3 · p. 7, Table 3 · p. 8
  • Different text prompts on factor nodes and null-token conditions on variables let DiffCollage compose motions with more actions than the pretrained model’s one-or-two-action scope.Source: §5.3 · pp. 7–8, Figure 8 · p. 8

5.4. Generation with complex graphs

  • Cycle graphs yield horizontal panoramas, while perspective-image diffusion conditioned on segmentation maps yields 360-degree panoramas from doodles.Source: §5.4 · p. 8, Figure 9 · p. 8

6. Conclusion

DiffCollage composes diffusion models trained on content pieces through factor graphs, whose marginal models can be trained and sampled in parallel. Its expressiveness depends on graph structure, and the authors leave more complicated graph representations for future work.

6. Conclusion

  • DiffCollage combines diffusion models trained on content pieces and supports parallel training and sampling of marginal models.The authors note that marginal samples are easier to obtain than samples from the joint distribution.Source: §6 · p. 8

A. Factor graph and Bethe approximation

  • Bethe-approximation likelihoods are exact for acyclic factor graphs, though the approximation is also used for graphs with loops.Source: A. Factor graph and Bethe approximation · p. 12
  • Graph structure sets a trade-off: sparse graphs simplify sampling but can miss correlations, while fully connected graphs are difficult to infer and may incur large Bethe-approximation bias.Source: A. Factor graph and Bethe approximation · p. 12
  • Hierarchical factor graphs can represent nodes and factors as joint distributions over multiple variables and model those distributions with further factor graphs.For 360-degree images, the described construction uses three factor nodes over sets of faces and three variable nodes representing opposing face pairs.Source: A. Factor graph and Bethe approximation · p. 12
  • Different conditional signals at different nodes can be viewed as a hierarchical factor graph with latent codes; the paper applies this approach to text-conditioned motion and segmentation-conditioned images.The authors relate independent image patches conditioned on a global code to this representation.Source: A. Factor graph and Bethe approximation · p. 12
  • For dependencies that demand difficult inference, junction trees may offer advantages over factor graphs, and generalizing DiffCollage to more complicated graphs remains future work.Source: A. Factor graph and Bethe approximation · p. 12

B. Training and Sampling of DiffCollage

DiffCollage trains diffusion models on factor-node and variable-node marginals independently, then composes their scores for sampling. Its sampling computations can run in parallel, and it supports conditioning through replacement or reconstruction procedures adapted from standard diffusion models.

B.1. Training

  • Diffusion models for each factor-node and variable-node marginal can be trained independently and in parallel, with symmetric nodes potentially sharing models.Source: §B.1 · p. 13

B.2. Sampling

  • After marginal models are trained, DiffCollage composes their scores to obtain the score of an implicit joint distribution, evaluating marginal scores in parallel to reduce latency.The resulting score can be used with existing diffusion samplers, and the method permits controllable generation without retraining [8] [26].Source: §B.2 · p. 13

C.1. Replacement and Reconstruction Methods for Conditioning

  • Replacement conditioning predicts a clean image and substitutes observed information, whereas reconstruction adds a gradient step minimizing the L2 distance between the observation and its predicted measurement.The paper notes that reconstruction has produced higher-quality super-resolution and inpainting images than replacement methods [9].Source: §C.1 · p. 14

C.2. Image experiments

The image experiments describe training and sampling choices for infinite-image and other image-generation settings, while the motion experiments specify inpainting, sampling, and prompt-conditioning procedures. The supplied passages report limited qualitative sampling observations and describe the evaluation reporting convention, but provide no numerical comparison values.

C.2. Image experiments

  • For image models, the paper fine-tunes GLIDE at 64 × 64 and its upsamplers, while LHQ and LSUN Tower models are trained from scratch with reduced channel size and iteration count due to limited resources.Source: §C.2 · p. 15
  • For infinite images, shift invariance lets the same diffusion model fit factor and variable nodes, with variable-node training data formed from random crops of factor-node images.The training procedure randomly crops by half with 50% probability; the variable-node image width is half the factor-node width.Source: §C.2 · p. 15
  • Image experiments use 80-step stochastic sampling, which the authors find slightly better than deterministic sampling; conditional signals can be interpolated to connect styles.The paper reports that DiffCollage with Algorithm 4 produces satisfying conditional samples efficiently.Source: §C.2 · p. 16

C.3. Motion experiments

  • Motion inpainting masks 50% of each sliding window and uses 50-step DDIM sampling; long prompts are split into short action prompts assigned to factor nodes, with null conditioning at variable nodes.For looped motion, an additional factor node connects the head and tail variable nodes; Table 4 reports each metric as the mean and standard deviation over 20 evaluations.Source: §C.3 · p. 16, Table 4 · p. 16

D. Limitations

DiffCollage’s piecewise modeling imposes conditional-independence assumptions that may omit important long-range dependencies. Parallel generation also trades lower latency for higher peak memory, and long factor graphs may require many sampler iterations.

D. Limitations

  • Piecewise conditional independence can omit needed long-range links; for example, a generated snake could have two heads or tails, though global conditioning can mitigate part of this issue.Source: D. Limitations · p. 16
  • Parallel diffusion computation reduces generation latency but uses more peak memory than autoregressive methods.Source: D. Limitations · p. 17
  • A linear-chain graph of length L may require O(L) iterations for optimal results, and very few steps can cause infinite-image artifacts; replacement or reconstruction may require O(L × K) steps.Source: D. Limitations · p. 17

E. Additional samples

The supplementary material provides additional examples and details of DiffCollage’s graph construction, training, sampling, and conditioning procedures. It also discusses the trade-offs and limits of the factor-graph formulation.

E. Additional samples

  • Additional samples show parallel patch-based inpainting on non-square images and DiffCollage-generated long landscapes with zoomed regions for high-resolution detail.The material also includes 360-degree cubemap examples using baseline and DiffCollage methods.Source: Figure 11 · p. 19, Figure 12 · p. 20, Figure 13 · p. 21

A. Factor graph and Bethe approximation

  • DiffCollage’s expressiveness depends on graph structure: sparse graphs ease sampling but miss correlations, while fully connected graphs are harder to infer and may increase Bethe-approximation bias.The authors report that DiffCollage empirically approximates complex distributions across real datasets and modalities.Source: A. Factor graph and Bethe approximation · p. 25
  • Hierarchical factor graphs model nodes or factors over multiple variables; for 360-degree images, smaller graphs represent groups of cubemap faces and their overlaps.The example uses three cycle factors over face pairs, with Bethe approximation applied at both the larger and face-level graphs.Source: A. Factor graph and Bethe approximation · pp. 25–26
  • For more complex dependencies and difficult inference, Junction trees may have advantages over factor graphs; generalizing DiffCollage to such graphs is left for future work.Source: A. Factor graph and Bethe approximation · p. 26

B.1. Training

  • Marginal diffusion models can be trained independently and in parallel, and symmetric factor or variable nodes may share models to further improve scalability.When pretrained models for each node are available, DiffCollage can work out of the box.Source: §B.1 · p. 26

B.2. Sampling

  • DiffCollage composes marginal scores into a score for the full content, allowing marginal-score computations to run in parallel during sampling.The method can use existing diffusion samplers and supports training-free conditioning such as inpainting and super-resolution.Source: §B.2 · pp. 26–27

C.1. Replacement and Reconstruction Methods for Conditioning

  • Replacement conditioning alters denoised predictions using observations, whereas reconstruction conditioning adds a gradient step minimizing the L2 discrepancy between observations and the predicted measurement.Both methods adapt standard diffusion-model conditioning procedures by using DiffCollage’s composed diffusion; the baselines use regular diffusion models.Source: §C.1 · p. 27

C.2. Image experiments

The image experiments describe training and sampling choices for infinite-image and conditional image generation. They report that stochastic sampling is slightly better than deterministic sampling for DiffCollage and that conditional samples are satisfying and efficient.

C.2. Image experiments

  • The image experiments fine-tune GLIDE upsamplers and train LHQ and LSUN Tower models from scratch, using reduced channel size and training iterations due to limited compute.Segmentation-conditioned models are further fine-tuned on segmentation maps; the 256→1024 upsampler is trained on 256×256 patches and used at 1024×1024 inference resolution.Source: §C.2 · pp. 28–29
  • For infinite images, the same shift-invariant diffusion model fits factor and variable nodes, with variable-node images half the factor-node width and random crops supplying variable-node training data.The training procedure randomly crops an image by half with 50% probability.Source: §C.2 · pp. 28–29
  • Stochastic sampling is reported as slightly better than deterministic sampling for DiffCollage, and Algorithm 4 produces satisfying conditional-generation samples efficiently.The experiments use 80 sampling steps and share the sampler across baseline and autoregressive replacement or reconstruction methods.Source: §C.2 · p. 29

C.3. Motion experiments

The motion experiments use a pretrained diffusion model and evaluate inpainting and long-prompt composition. For multiple actions, prompts are manually divided into short action descriptions assigned across factor nodes.

C.3. Motion experiments

  • Motion inpainting masks 50% of the content in the sliding window for replacement and reconstruction methods, and experiments use deterministic DDIM sampling with 50 steps.Source: §C.3 · p. 29
  • For multi-action motion composition, prompts are manually split into short sentences with one or two actions, assigned sequentially to factor nodes, while variable nodes receive a null token.Source: §C.3 · p. 29
  • For long-motion experiments, an additional factor node connects the head and tail variable nodes, analogous to the cycle construction used for circle-image generation.Source: §C.3 · p. 29

D. Limitations

DiffCollage’s limitations include conditional-independence assumptions that can omit long-range dependencies, greater peak memory use than autoregressive methods, and potentially many sampling iterations for large graphs. The appendix also points to additional samples and motion videos.

D. Limitations

  • Conditional independence between content pieces can omit necessary long-range dependencies; for example, a generated snake could have two heads or tails.Global conditioning information, such as landscape segmentation maps, can mitigate part of this issue.Source: D. Limitations · p. 30
  • Parallel diffusion computations reduce latency but require more peak memory than autoregressive methods.Source: D. Limitations · p. 30
  • Large graph diameter may require many iterations for information to flow; a length-L linear chain may need O(L) sampler iterations, and very few steps such as 35 can produce artifacts in infinite images.The authors compare this with O(L × K) steps for replacement or reconstruction methods, where K is the number of iterations for the small diffusion models.Source: D. Limitations · p. 30

E. Additional samples

Related papers

Loading 2303.17076v1…