Source-linked AI summary
COCO-GAN: Generation by Parts via Conditional Coordinating
Chieh Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, Hwann-Tzong Chen
TL;DR
COCO-GAN addresses the challenge of generating large-field-of-view images under computational constraints by generating image parts conditioned on spatial coordinates and coordinating their assembly. It produces high-quality full images despite never generating full images during training, while enabling beyond-boundary and panorama generation, parallel and memory-saving computation, and on-demand synthesis.
Problem
Full-image processing can become a computational bottleneck for large-field-of-view vision, motivating generative models that assemble coordinate-aware local views.
Method
COCO-GAN generates micro patches conditioned on coordinates and trains discrimination over assembled macro patches using realism and spatial-consistency objectives.
Results
COCO-GAN achieves state-of-the-art generation quality and produces full images during inference even though full images are never generated during training.
Takeaways & Limitations
Coordinate-aware generation enables beyond-boundary images, horizontally cyclic panoramas, parallel and memory-saving computation, and on-demand image-part synthesis.
Takeaways & Limitations
Generated samples can contain discontinued or mottled local structures, and sparse coordinate sampling can produce incorrect object relationships.
Abstract
from arXiv · showhide
Humans can only interact with part of the surrounding environment due to biological restrictions. Therefore, we learn to reason the spatial relationships across a series of observations to piece together the surrounding environment. Inspired by such behavior and the fact that machines also have computational constraints, we propose \underline{CO}nditional \underline{CO}ordinate GAN (COCO-GAN) of which the generator generates images by parts based on their spatial coordinates as the condition. On the other hand, the discriminator learns to justify realism across multiple assembled patches by global coherence, local appearance, and edge-crossing continuity. Despite the full images are never generated during training, we show that COCO-GAN can produce \textbf{state-of-the-art-quality} full images during inference. We further demonstrate a variety of novel applications enabled by teaching the network to be aware of coordinates. First, we perform extrapolation to the learned coordinate manifold and generate off-the-boundary patches. Combining with the originally generated full image, COCO-GAN can produce images that are larger than training samples, which we called "beyond-boundary generation". We then showcase panorama generation within a cylindrical coordinate system that inherently preserves horizontally cyclic topology. On the computation side, COCO-GAN has a built-in divide-and-conquer paradigm that reduces memory requisition during training and inference, provides high-parallelism, and can generate parts of images on-demand.
1. Introduction
COCO-GAN learns to generate and assess image parts conditioned on spatial coordinates, addressing computational limits while assembling coherent full images at inference. It achieves strong generation quality and supports beyond-boundary, panorama, parallel, memory-saving, and on-demand applications.
- 1. Introduction: COCO-GAN generates image patches from latent vectors conditioned on spatial coordinates and evaluates assembled macro patches for realism and spatial consistency.The discriminator assesses structural soundness, visual homogeneity, and continuity across patch edges.
- 1. Introduction: COCO-GAN achieves state-of-the-art FID generation quality across multiple experimental setups, despite generating full images only during inference.With 4 × 4 micro patches, 1024 separately generated patches still form complete and plausible human faces.
- 1. Introduction: Cylindrical coordinates support panorama generation with horizontally cyclic samples, preserving the native topology of panoramic images.The coordinate system is selected specifically to preserve horizontal cyclicity.
- 1. Introduction: Generation by parts provides parallelism, reduces memory requirements, and supports on-demand patch generation for computation-restricted environments.The disjoint patch procedures particularly fit mobile and virtual-reality applications.
- 1. Introduction: Patch-guided generation reconstructs an entire image using a real image patch as guidance through an added latent-vector prediction branch.This application extends coordinate-aware generation beyond unconditional image synthesis.
2. COCO-GAN
COCO-GAN generates images from independently produced coordinate-conditioned micro patches, assembling them into macro patches while training the discriminator on patch realism, spatial consistency, and boundary continuity.
- Spatial coordinate system: COCO-GAN uses a generator and discriminator with finer-grained micro and coarser-grained macro coordinate systems.The generator produces micro patches, while the discriminator evaluates assembled macro patches.
- Spatial coordinate system: The generator independently produces micro patches from a shared latent vector and spatial coordinate, then merges neighboring patches into macro patches.The merging function can be simple concatenation when the micro patches are arranged as neighbors.
- Discriminator design: The discriminator uses larger assembled macro patches so adversarial training penalizes seams and promotes continuity across neighboring generated patches.This exposes multiple consecutive or nearby micro patches to the discriminator simultaneously.
- Loss functions: COCO-GAN combines patch Wasserstein, gradient-penalty, and spatial-consistency losses instead of training both networks directly on full images.The spatial-consistency loss trains the discriminator to estimate macro coordinates, while the other losses operate on macro and micro patches.
- Inference: At test time, a coordinate matrix independently produces the micro patches needed to constitute a full image.The coordinate design can also encode topology, including cylindrical coordinates for panoramic images.
- Qualitative results: Figure 3 shows visually smooth, globally coherent full images alongside macro and micro patches without post-processing.For the first five columns, corresponding examples share latent vectors, although the rows are not spatially aligned because their sizes differ.
3. Experiments
Experiments show that COCO-GAN generates coherent images from independently generated patches, achieves competitive FID, and supports continuous coordinate-aware generation and extrapolation. The same design also enables panorama generation, patch-guided generation, and lower-memory computation, while overly small macro patches can produce incorrect spatial relationships.
- 3.1. Quality of Generation by Parts: Even 4 × 4 micro patches can be assembled into complete and plausible human faces using 1024 separately generated patches.The micro-patch sweep covers 32 × 32, 16 × 16, 8 × 8, and 4 × 4 resolutions.
- 3.1. Quality of Generation by Parts: COCO-GAN is competitive with other state-of-the-art generative models on FID without additional hyper-parameter tuning.FID is evaluated against 50,000 held-out real and generated samples.
- 3.2. Latent Space Continuity: Latent interpolation changes all micro patches synchronously, while coordinate interpolation produces overall spatial continuity with a fixed latent vector.The coordinate interpolation spans the micro-coordinate range [−1, 1].
- 3.2. Latent Space Continuity: Coordinate interpolation exposes a limitation: sparse coordinate sampling can make the model deform one eye into another instead of representing the glabella between them.This indicates that visually high-quality faces can still contain incorrect learned object relationships.
- 3.3. Beyond-Boundary Generation: Extrapolating the learned coordinate manifold generates 384 × 384 images from a model trained only on 256 × 256 images.Each image direction is extended by one 64-pixel micro patch beyond the regular generation region.
- 3.4. Panorama Generation & Partial Generation: COCO-GAN’s coordinate-aware, patch-based design supports cyclic panoramas, patch-guided generation, and parallel or memory-efficient computation.Its disjoint patch procedures also support generating image regions on demand, while an auxiliary network can infer a latent vector from a real macro patch.
- 3.6. Computation-Friendly Generation: A 32 × 32 macro patch with 16 × 16 micro patches produces duplicated faces, showing that insufficient macro context can mislead spatial-relation learning.The minimum suitable macro-patch size varies across datasets.
- 3.7. Ablation Study: The ablation study finds similar eventual FID for COCO-GAN, an optimal discriminator, and an optimal generator, whereas multiple generators fail to produce coherent full images.The multiple-generators setting can concatenate micro patches without obvious seams but often yields globally incoherent results.
4. Related Work
COCO-GAN differs from prior coordinate-aware and sequential generation methods by generating at the patch level and enforcing spatial coherence adversarially.
- COCO-GAN decomposes image generation into patch-level computation dependencies across spatial dimensions, unlike Pixel-RNN’s pixel-level sequence generation.
- Adversarial loss in COCO-GAN encourages smoothness between adjacent generated micro patches.
- Unlike CoordConv, which supplies positioning information to convolutional kernels, COCO-GAN uses spatial coordinates as an auxiliary adversarial-learning task.
5. Conclusion and Discussion
COCO-GAN generates images by parts while supporting extrapolation beyond training boundaries and achieving competitive generation quality, though some local discontinuities remain.
- COCO-GAN enables generation by parts and achieves generation quality competitive with state-of-the-art GANs.
- Beyond-Boundary Generation produces images larger than training samples by extrapolating the learned coordinate manifold.
- 128 × 128 images can be produced with micro patches as small as 4 × 4 pixels, although the overall FID score slightly degrades.
- Some generated samples contain discontinued or mottled local structures, motivating additional refinements or blending approaches.
C. Experiments Setup and Model Architecture Details
The implementation uses coordinate-conditioned generator and discriminator architectures with conditional normalization, spectral normalization, and configurable patch sizes.
- Architecture: The generator and discriminator use projection-discriminator backbones with class projection and spectral normalization throughout convolutional and feed-forward layers.
- Architecture: Conditional batch normalization modulates normalized generator features using parameters conditioned on spatial coordinates and latent vectors.
- Coordinate Setup: The coordinate setup normalizes spatial conditions to [−1, 1] and uses discrete coordinate sampling for improved visual quality.
- Architecture: Micro patches are generated at 32 × 32 pixels, while the discriminator evaluates 64 × 64 macro patches in the illustrated architectures.
- Architecture: The discriminator architecture includes global pooling, element-wise multiplication, reduction, and residual-block operations.
E. Beyond-Boundary Generation: More Examples and Details of Post-Training
Beyond-boundary generation extrapolates coordinates outside the training region, using post-training to reduce discontinuities while stabilizing only high-level generator layers.
- Direct training on coordinates outside [−1, 1] is infeasible because no real data exist there, allowing the generator to exploit the discriminator.
- Post-training expands coordinate conditions beyond [−1, 1] until patch discontinuities become invisible.
- Without extra training, slight coordinate extrapolation can extend content beyond the normal area but remains discontinuous at edges.
- Training only the first two generator layers while updating the discriminator stabilizes post-training and avoids weird, mottled artifacts.
- 384 × 384 samples are generated from 256 × 256 training images by extrapolating the learned coordinate manifold.
F. More Full Image Generation Examples
COCO-GAN generates full images across multiple CelebA configurations and LSUN at higher resolution, while also supporting latent-vector interpolation for micro patches and full images.
- Full CelebA images are shown at 128 × 128 across settings from (N2,M2,S32) to (N16,M16,S4).
- A 256 × 256 LSUN example demonstrates full-image generation with the (N4,M4,S64) setting.
- Interpolation: Interpolating between two latent vectors produces corresponding micro patches and full images.
H. More Panorama Generation Samples
The panorama samples demonstrate horizontally cyclic structure and spatial continuity under interpolated coordinate conditions.
- Panorama samples preserve cyclic structure along the horizontal direction at 768 × 256 resolution with 64 × 64 micro patches.
- Spatially interpolating micro coordinates between −1 and 1 with a fixed latent vector shows continuity between micro patches.
J. Ablation Study
The ablation figures compare COCO-GAN variants and report similar image quality, while patch-guided generation retains some structure or global characteristics from source images.
- Variant comparison: COCO-GAN variants show no significant differences in image quality, indicating no significant quality trade-off for conditional coordinates.
- Patch-guided generation: Patch-guided generation can loosely retain local structure or global characteristics from the original image.
L. Training Indicators
Training indicators on CelebA at 128 × 128 show stable behavior over time.
- Wasserstein distance and FID curves both indicate stable COCO-GAN training on CelebA at 128 × 128 resolution.