Source-linked AI summary
Learning What and Where to Draw
Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, Honglak Lee
TL;DR
Existing text-to-image models lack control over object locations and poses. GAWWN adds bounding-box and part-keypoint conditioning, including generation of unspecified keypoints, and produces high-quality 128 × 128 bird images plus preliminary pose-controlled human images.
Problem
Existing image-generation models use global labels or captions without controlling object location or pose.
Method
GAWWN synthesizes images from text plus bounding boxes or part keypoints, and models conditional keypoint distributions for arbitrary observed subsets.
Results
On CUB, location conditioning enables compelling 128 × 128 images, while generated keypoints show no average quality degradation versus ground-truth keypoints.
Takeaways & Limitations
Separating image content from location makes high-resolution synthesis more controllable and provides an interface for specifying only selected part locations.
Takeaways & Limitations
Human-image synthesis remains blurry and struggles with complex descriptions and unusual poses, and further architectural advances are required.
Abstract
from arXiv · showhide
Generative Adversarial Networks (GANs) have recently demonstrated the capability to synthesize compelling real-world images, such as room interiors, album covers, manga, faces, birds, and flowers. While existing models can synthesize images based on global constraints such as a class label or caption, they do not provide control over pose or object location. We propose a new model, the Generative Adversarial What-Where Network (GAWWN), that synthesizes images given instructions describing what content to draw in which location. We show high-quality 128 x 128 image synthesis on the Caltech-UCSD Birds dataset, conditioned on both informal text descriptions and also object location. Our system exposes control over both the bounding box around the bird and its constituent parts. By modeling the conditional distributions over part locations, our system also enables conditioning on arbitrary subsets of parts (e.g. only the beak and tail), yielding an efficient interface for picking part locations. We also show preliminary results on the more challenging domain of text- and location-controllable synthesis of images of human actions on the MPII Human Pose dataset.
1 Introduction
GAWWN addresses the gap between global text conditioning and precise spatial control by separating what an image contains from where it appears. It targets controllable synthesis using text, bounding boxes, and part keypoints.
- Existing image-generation models condition on class labels or captions but do not control where objects appear.
- Separating “what” and “where” can improve parameter efficiency and make generated content more interpretable.
- CUB supplies captions, bird bounding boxes, and part keypoints, while MPII Human Pose includes three collected captions per image.
- GAWWN uses spatial masking and cropping for coarse bird locations, and multiplicative gating for bird and human part coordinates.
- The model’s contributions include location-controllable synthesis, conditional part completion, and exploratory pose-conditional human-image synthesis.
2 Related Work
Related work spans deterministic generators, variational autoencoders, GANs, attention mechanisms, and autoregressive models. GAWWN applies the shared principle of separating content from spatial conditioning within a GAN framework.
- Earlier deterministic networks generated images or sequences conditioned on graphics codes, rotations, or actions.
- Variational autoencoders introduced probabilistic generation, including disentangled latent graphics codes and recurrent attention-based models.
- GAN samples were relatively sharp, and later architectures improved qualitative synthesis and training stability.
- Spatial Transformer Networks provided visual attention mechanisms in recurrent generative models for multi-object scenes and visual generalization.
- The what-where separation principle can also apply to autoregressive and spatial grid-structured image-generation models.
3 Preliminaries
The paper builds on GAN minimax training, conditional GANs, and structured text-image embeddings. These components make realism and correspondence with textual context joint training objectives.
- GANs train a generator and discriminator in a two-player minimax game over real and synthetic images.
- With sufficient capacity, the GAN minimax game has an optimum where the generator distribution matches the data distribution.
- Conditional GANs provide both generator and discriminator with context, requiring generated images to look realistic and match that context.
- The text encoder learns image-text correspondence through a structured loss over images, descriptions, and class labels.
- The implementation uses a char-CNN-GRU and averages four sampled captions per image when estimating training expectations.
4 Generative Adversarial What-Where Networks (GAWWN)
GAWWN combines text with coarse bounding-box control or detailed part-keypoint control, using spatial pathways and gating to localize synthesis. A conditional keypoint generator fills unspecified parts while preserving user-specified ones.
- 4.1 Bounding-box-conditional text-to-image model: The bounding-box model starts from noise and text, spatially warps the text embedding into the box, and encodes coarse spatial structure.
- 4.1 Bounding-box-conditional text-to-image model: Its generator separates local and global processing, masks regions outside the box, merges pathways, and outputs values constrained to [−1, 1].
- 4.1 Bounding-box-conditional text-to-image model: The discriminator also combines global image processing with a cropped local pathway conditioned on the bounding box and text.
- 4.2 Keypoint-conditional text-to-image model: The keypoint model encodes each part in a spatial feature-map channel and combines keypoints with text and noise for coarse conditioning.
- 4.2 Keypoint-conditional text-to-image model: Local activations are multiplicatively gated by the keypoint tensor before concatenation and final image generation.
- 4.3 Conditional keypoint generation model: Conditional keypoint generation uses observed-part switches to preserve specified locations while predicting unobserved parts from noise, text, and existing keypoints.
- 4.3 Conditional keypoint generation model: Training randomly samples switch units, with each bird part acting as a conditioning variable 10% of the time.
5 Experiments
Experiments evaluate GAWWN’s text- and location-controllable synthesis on birds and humans, testing bounding boxes, fixed or generated keypoints, partial keypoint specification, and comparisons with prior work.
- Datasets: CUB contains 11,788 bird images across 200 species, with captions, bounding boxes, and 15-part keypoint annotations.MHP contains 25K images spanning 410 activities, with three collected captions per image and joint keypoints; training retained 19K single-person crops.
- Bounding-box control: GAWWN varies bird bounding boxes while fixing text and noise, enabling control over object location, size, and aspect ratio.With fixed noise, backgrounds are usually similar but not perfectly invariant; bird orientation remains unchanged as the box moves.
- Keypoint control: With ground-truth keypoints and held-out text, generated birds accurately reflect text and pose constraints while pose remains invariant across noise samples.Noise changes backgrounds and small details such as branch thickness and color palette.
- Keypoint control: Specifying beak and tail keypoints lets GAWWN shrink, translate, and stretch birds, while also controlling their pointing direction.These two parts often define an approximate bounding box, and interpolating them allows the remaining parts to be sampled conditionally.
- Generated keypoints: Generated keypoints provide accurate bird samples without an average observed quality degradation relative to ground-truth keypoints.This reduces the effort required from a user who would otherwise need to enter 15 keypoint locations.
- Comparisons and human synthesis: On CUB, GAWWN’s 128 × 128 samples with real or synthetic keypoints are reported as the sharpest and most accurate, while human samples are blurrier but often respect text and pose.Simple skiing, golf, and yoga captions tend to work; complex descriptions and unusual poses remain challenging.
6 Discussion
GAWWN generates 128 × 128 images from informal text and object locations, while retaining test-time flexibility by modeling part locations. The authors identify decomposition into easier subproblems as a route to realistic high-resolution synthesis, but human-action synthesis remains unsaturated.
- GAWWN generates images conditioned on both informal text descriptions and object locations, controlling locations through bounding boxes or part keypoints.On CUB, location constraints supported compelling 128 × 128 images, compared with the 64 × 64 resolution of previous models.
- A text-conditional generative model of part locations allows locations to be generated at test time rather than fixed in advance.
- Decomposing synthesis into easier subproblems can help generate realistic high-resolution images and make GAN-based text-to-image pipelines easier to train.The decomposition also provides additional ways to control image synthesis.
- Text-to-human image synthesis is presented as a first result on that task, but performance is clearly far from saturated and further architectural advances are required.