PaperlayerView exact revision on arXiv

Source-linked AI summary

Boundless: Generative Adversarial Networks for Image Extension

Piotr Teterwak, Aaron Sarna, Dilip Krishnan, Aaron Maschinot, David Belanger, Ce Liu, William T. Freeman

arXiv:1908.07007v1cs.CV

TL;DR

Image extension is challenging because it must continue content beyond a boundary with context on only one side, while existing inpainting methods transfer poorly. The paper adapts GANs for extension and adds semantic conditioning to the discriminator; it reports reliable large extrapolations and strong image-extension results, while noting that objects remain more challenging than textures.

  • Problem

    Image extension must plausibly match the original image beyond a boundary where context is available on only one side, and unmodified inpainting algorithms produce poor results for this task.

    Source: §1 · p. 1

  • Method

    The authors adapt a GAN for image extension and use pretrained semantic information to modulate the discriminator, alongside changes to the generator architecture.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    The authors report reliable extensions up to 3 times the original image width and say semantic conditioning substantially improves results.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors describe their semantic-conditioning stabilization scheme as useful for adversarial models with a ground-truth sample for each generator input.

    Source: §1 · p. 2

    The model performs better on textures than objects; vehicles, people, and furniture remain challenging.

    Source: §9 · p. 12

Abstract

from arXiv · show

Image extension models have broad applications in image editing, computational photography and computer graphics. While image inpainting has been extensively studied in the literature, it is challenging to directly apply the state-of-the-art inpainting methods to image extension as they tend to generate blurry or repetitive pixels with inconsistent semantics. We introduce semantic conditioning to the discriminator of a generative adversarial network (GAN), and achieve strong results on image extension with coherent semantics and visually pleasing colors and textures. We also show promising results in extreme extensions, such as panorama generation.

1. Introduction

Image extension must plausibly continue structure, texture, and semantics beyond an image boundary, where context is available on only one side. The paper adapts GANs for this setting, adding semantic conditioning to the discriminator and reporting reliable large extrapolations.

1. Introduction

  • Image extension could support altered camera viewpoints in virtual reality, fill gaps in stitched panoramas, and extend videos for more immersive viewing.The cited applications include adapting video to screens with a different aspect ratio.Source: §1 · p. 1
  • Unlike inpainting, extension has boundary context on only one side, and unmodified inpainting algorithms produce poor results in this setting.The desired extension should blend seamlessly while matching the original in texture, structure, and semantics.Source: §1 · p. 1
  • The paper’s examples report better object shapes and good textures than DeepFill [48] and PConv [26].The comparison shows the input image extended into a masked area.Source: Figure 1 · p. 1
  • Prior extension methods either need a carefully chosen guide image or mainly extend texture without larger-scale structure or image semantics.The authors seek models that learn from diverse data and extend structure.Source: §1 · pp. 1–2
  • The paper starts from inpainting models but reports that their results degrade quickly as extension moves farther from the image border, motivating modifications and semantic conditioning.The authors say semantic conditioning substantially improves results.Source: §1 · p. 2
  • The method learns image extensions reliably up to 3 times the original image width and introduces semantic conditioning that modulates the GAN discriminator using pretrained-network information.The authors also report ablations on architectural components and say the stabilization scheme is useful for adversarial models with a ground-truth sample for each generator input.Source: §1 · p. 2

2. Related Work

Related work spans classical patch-based inpainting, learning-based inpainting, and image-extension methods that borrow external patches or use guide images. The authors position their work among GAN-based approaches and distinguish it from prior extension and inpainting methods.

2. Related Work

  • Classical inpainting uses patch similarity and diffusion to fill holes, works best on small holes in stationary textures, and generally lacks semantic understanding.Bidirectional Similarity and PatchMatch [7, 23] are identified as among the most successful methods.Source: §2 · p. 2
  • Some non-parametric extension methods rely on large photo databases, while another depends on a carefully selected guide image.Source: §2 · p. 2
  • Learning-based inpainting progressed from Context Encoder to methods using deep feature patches, local and global discriminator losses, and coarse-to-fine contextual attention.Other approaches include [47], while optimizes an unconditioned GAN’s latent code at inference time.Source: §2 · p. 2
  • The only prior fully parametric extension approach the authors knew of used an autoregressive model for 32x32-pixel images, which they describe as too small for practical applications.Concurrent work is similar but does not condition the discriminator with pretrained features.Source: §2 · p. 2

3. Model

The model uses a Wasserstein GAN with a masked-image generator and a discriminator conditioned on the extension inputs. Generator changes address patch repetition, while discriminator conditioning adds seam and semantic guidance; training combines reconstruction and adversarial losses.

3. Model

  • The Wasserstein GAN generator receives a masked image and binary mask, while the discriminator scores real or generated images and sees known pixels restored before evaluation.The unknown region is zeroed in the input; the mask marks unknown pixels, and the full generator output receives pixel loss.Source: §3 · pp. 2–3

3.1. Generator

  • Removing contextual attention avoids patch copying, while skip connections improve high-frequency synthesis and instance normalization reduces generated-image artifacts.The generator otherwise follows a fully convolutional encoder-decoder design with gated convolutions and dilated inner layers.Source: §3.1 · p. 3

3.2. Discriminator

  • The discriminator uses composited inputs and a mask for seam continuity, then semantic conditioning from pretrained InceptionV3 features of the target image to discourage drift in larger extensions.The semantic conditioning is a modified conditional projection discriminator.Source: §3.2 · p. 4

3.3. Training

  • Training combines full-output ℓ1 reconstruction loss for coarse agreement with a Wasserstein GAN hinge adversarial loss for refinement.The generator’s total loss weights the adversarial term by λ = 10^-2.Source: §3.3 · p. 4

4. Experimental Results

The full model maintains semantic consistency and mostly photorealistic, seamless synthesis on larger extensions, while ablations expose distinct quality losses. Recursive panorama generation is mostly plausible but shows degradation and semantic drift farther from the seed.

4. Experimental Results

  • Experiments evaluate image extension with rightmost 25%, 50%, and 75% masked and inpainting with a central square 25% mask; on 500 test images, they report FID on full outputs and PSNR on masked regions.Source: §4.1 · p. 5, Table 1 · p. 5

4.1. Image Extension

  • As extensions grow, DeepFill and CAF repeat textures, PConv blurs and artifacts, and No-Cond blurs and drifts; the full model remains semantically consistent, mostly photorealistic, and seamless.Figure 4 presents comparisons across 25%, 50%, and 75% extension masks.Source: §4.1 · p. 5, Figure 4 · p. 6
  • On 75% extensions, conditioning significantly outperforms feature matching and perceptual loss, while these alternatives perform similarly on smaller extensions.The reported comparison says feature matching and perceptual losses slightly outperform conditioning on short-range extensions, but conditioning has the best results on 75% extensions.Source: Table 2 · p. 5, §4.1 · p. 7, Figure 5 · p. 7

4.2. Ablations

  • Removing skip connections impairs high-frequency textures and can create seams; removing conditioning worsens quality farther from context, removing instance norm increases artifacts, and smaller batches blur results.Figure 6 shows the ablations, each removing one feature at a time.Source: §4.2 · p. 7, Figure 6 · p. 8

4.3. Panorama Generation

  • Five recursive extensions using the 25% model grew a 257x192 seed to 582 pixels (2.7x); results were mostly plausible and aesthetically pleasing, but degraded and drifted semantically.The panorama setup uses a sliding window, feeding synthesized content from each step into the next.Source: §4.3 · p. 7, Figure 7 · p. 8

4.4. Exploring the Space of Plausible Extensions

  • YouTube8m frames were extended independently on both sides with the 25% model to probe memorization and collapse under small natural input variations.The authors describe videos as a way to visually explore the range of generated results; they refer readers to a supplementary video.Source: §4.4 · p. 7

6. Network Training and Architecture Details

The appendix describes the generator and discriminator architectures and the training data and input preparation. The discriminator combines unconditional and semantic projection scores, while training uses resized Places365 images with randomized mask sizes.

6.1. Generator Network G

  • Table 3 documents generator layers by activation, kernel, stride, dilation, output channels or units, and skip connections; convolutions use same padding and resizing uses bilinear interpolation.Source: Table 3 · p. 11

6.2. Discriminator Network D

  • The discriminator spectrally normalizes every layer; its common tower feeds non-conditional and projection branches, whose scalar outputs are summed before the adversarial loss.The projection branch uses logits from an ImageNet-trained InceptionV3 network and its output is summed with the non-conditional branch before the hinge loss.Source: §6.2 · p. 11, Table 6 · p. 11, Table 4 · p. 11

6.3. Training details:

  • Training uses 1,953,624 images from the top 50 Places365-Challenge classes and 46,376 test images; inputs are resized to 257x257 with a mask channel and mask size jittered by ±4 pixels.Source: §6.3 · pp. 11–12
  • The generator receives an added channel of ones intended to indicate the padded image edge, though the authors do not verify this; generator and discriminator steps run jointly in a 1:1 ratio.Source: §6.3 · p. 12

7. Qualitative Results

The qualitative results include image extensions at three mask widths, inpainting and panorama samples, and an example of freeform-mask use.

7. Qualitative Results

  • Additional image-extension samples cover masks spanning 25%, 50%, and 75% of the image width.Source: §7 · p. 12
  • The qualitative examples also include center inpainting and panorama results.Source: §7 · p. 12
  • The method is also shown on freeform masks.Source: §7 · p. 12, Figure 8 · p. 12

8. Exploring the Space of Plausible Extensions

The section presents a video generated from a YouTube8m sample and invites frame-by-frame inspection of its completions. Tiny perturbations to the original frame yield different plausible completions.

8. Exploring the Space of Plausible Extensions

  • The accompanying video is derived from a YouTube8m dataset sample [2].Source: §8 · p. 12
  • Tiny perturbations of the original frame produce different plausible completions.Source: §8 · p. 12
  • The authors invite viewers to pause the video at arbitrary frames to inspect the model’s completions.Source: §8 · p. 12

9. Failure Cases

The model handles textures better than objects, with vehicles, people, and furniture identified as challenging cases. The supplied examples also include extensions at multiple mask widths and other image-completion settings.

9. Failure Cases

Loading 1908.07007v1…