PaperlayerView exact revision on arXiv

Source-linked AI summary

Manipulating Attributes of Natural Scenes via Hallucination

Levent Karacan, Zeynep Akata, Aykut Erdem, Erkut Erdem

arXiv:1808.07413v3cs.CV

TL;DR

Editing a natural scene to reflect transient attributes is challenging because its appearance must change while semantic details remain intact, and common style transfer relies on reference images. The paper hallucinates a layout-consistent scene with desired attributes and transfers its look to the input; the authors report plausible attribute edits, including multiple attributes in one model.

  • Problem

    Attribute manipulation must change a natural scene’s appearance while preserving semantic details, while common style-transfer approaches require a reference image.

    Source: §1 · p. 2

  • Method

    A conditional generative network hallucinates scenes with desired transient attributes, and photo style transfer transfers their look to the original input image.

    Source: §1 · p. 2

  • Results

    The authors report that the model hallucinates many attributes reasonably well and supports unified edits with multiple attributes.

    Source: §6 · p. 16

  • Takeaways & Limitations

    The framework synthesizes a layout-consistent reference style on the fly rather than requiring a separate reference style image.

    Source: §6 · p. 16

    The approach may fail when either component fails, including poor scene generation or unsuccessful style transfer.

    Source: §5.4 · p. 15

Abstract

from arXiv · show

In this study, we explore building a two-stage framework for enabling users to directly manipulate high-level attributes of a natural scene. The key to our approach is a deep generative network which can hallucinate images of a scene as if they were taken at a different season (e.g. during winter), weather condition (e.g. in a cloudy day) or time of the day (e.g. at sunset). Once the scene is hallucinated with the given attributes, the corresponding look is then transferred to the input image while preserving the semantic details intact, giving a photo-realistic manipulation result. As the proposed framework hallucinates what the scene will look like, it does not require any reference style image as commonly utilized in most of the appearance or style transfer approaches. Moreover, it allows to simultaneously manipulate a given scene according to a diverse set of transient attributes within a single model, eliminating the need of training multiple networks per each translation task. Our comprehensive set of qualitative and quantitative results demonstrate the effectiveness of our approach against the competing methods.

1 INTRODUCTION

Manipulating a natural scene’s transient attributes requires changing its appearance while preserving semantic details, a challenge complicated by the need for a suitable reference style image. The paper combines conditional scene hallucination with photo style transfer to edit outdoor images according to desired attributes.

1 INTRODUCTION

  • The model is designed to hallucinate multiple possible scene versions from data by learning attribute meanings and corresponding local and global transformations.Supported attributes include conditions such as cloudy, foggy, night, sunset, winter, and summer.Source: §1 · p. 2
  • Attribute editing must produce photorealistic changes while preserving the input’s object-specific semantic details, and common style-transfer methods require a reference image.The paper describes reference-image retrieval as another approach that raises additional challenges.Source: §1 · p. 2
  • The framework hallucinates a scene with the input’s semantic content and desired attributes, then transfers its visual look to the original image.This combines neural image synthesis and photo style transfer.Source: §1 · p. 2
  • The conditional GAN uses semantic layout to specify where to draw and transient attributes to edit visual properties in a semantically similar generated scene.The authors also use selectively sampled mismatched layouts and segmentation-invariant perceptual features to improve training and layout consistency.Source: §1 · p. 2
  • The contributions include a two-stage outdoor-image editing framework, a layout- and attribute-conditioned GAN, and a dataset combining images annotated with layouts and transient attributes.The code and models are publicly available.Source: §1 · p. 2

2 RELATED WORK

Prior image-synthesis and editing approaches offer varying degrees of realism, diversity, and control, but leave constraints on attribute control or reliance on reference images. The paper positions its approach as using semantic layouts and transient attributes for scene generation, alongside learned editing methods.

2.1 Image Synthesis

  • Existing scene translators can be deterministic or restrict control to domains, while the proposed model lets users vary multiple scene attributes at once.The related work contrasts domain translation with controls such as partial sunset and partly cloudy skies.Source: §2.1 · p. 3
  • The proposed scene generation network conditions on semantic layout, transient attributes, and latent noise, allowing the same content to be generated under different conditions.Prior multi-output methods may fix the number of outputs in advance.Source: §2.1 · p. 3
  • For training, the paper emphasizes visually similar negative examples and matches segmentation-invariant features to encourage generations that comply with layouts.The negative-pair strategy is inspired by relevant-negative sampling; the feature matching uses a pretrained segmentation network.Source: §2.1 · p. 3

2.2 Image Editing

  • Image-editing approaches include retrieval-based appearance transfer and neural photo style transfer, but many require a reference image with the desired effect.The reviewed photo-style methods use semantic labels to maintain consistency across regions.Source: §2.2 · p. 4

3 ALS18K DATASET

ALS18K combines outdoor images from ADE20K and Transient Attributes to provide semantic-layout and transient-attribute annotations. The datasets complement one another because each originally lacks one of those annotation types.

3 ALS18K DATASET

  • The dataset combines ADE20K’s semantic annotations with Transient Attributes’ weather, time-of-day, and season labels, since neither source originally provides both annotation types.ADE20K lacks transient attributes, while Transient Attributes lacks semantic layout labels.Source: §3 · p. 4
  • The authors select 9,201 outdoor ADE20K images and add manually verified attribute predictions, then assign layouts to Transient Attributes images using a selected segmentation prediction per webcam scene.The Transient Attributes layout assignment assumes images from the same webcam scene share a semantic layout.Source: §3 · pp. 4–5
  • The framework overview shows the semantic layout entering scene generation before the hallucinated target-attribute look is transferred to the original input image.The layout is resized and center-cropped to 512 × 512 pixels before generation.Source: Figure 2 · p. 5
  • ALS18K contains 17,772 outdoor images, with 16,434 training examples and 1,338 testing images, annotated with 150 semantic categories and 40 transient attributes.The images are resized and center-cropped to 512 × 512 pixels.Source: §3 · p. 5

4 ATTRIBUTE MANIPULATION FRAMEWORK

The framework generates outdoor scenes from semantic layouts and desired transient attributes, then transfers the generated look to the input image. Its scene generator is trained with multi-scale discriminators and additional strategies for improving training and generation quality.

4 ATTRIBUTE MANIPULATION FRAMEWORK

  • The two-stage framework generates a scene with desired transient attributes, then transfers its look to the input image for photorealistic manipulation.The generator is conditioned on the input’s semantic layout and a continuous-valued attribute vector.Source: §4 · p. 5

4.1.1 Background.

  • The scene generator takes semantic layout, transient attributes, and noise as inputs; the noise adds stochasticity while attributes control generation.Its coarse generator outputs 256 × 256 images, and its fine generator uses that output to produce 512 × 512 images.Source: §4.1.2 · p. 6, §4.1.1 · p. 6

4.1.2 Proposed Architecture.

  • The architecture conditions multi-scale generators on layout and attributes and uses three image-scale discriminators to assess realism and whether image-condition pairings are valid.The generator encodes layout categories as 8-bit binary codes and transient attributes as a 40-dimensional vector.Source: §4.1.2 · p. 6

4.1.3 Improved Training of SGNs.

  • Relevant negative mining pairs each layout with its nearest-neighbor layout as a mismatched example; the authors report improvement over random sampling.The method applies this mining strategy to layouts.Source: §4.1.3 · p. 7
  • Flipping a few attributes to create mismatched examples hurt performance, which the authors attribute to the attributes being too semantically close and confusing the discriminator.The authors use random sampling for mismatched attributes instead.Source: §4.1.3 · p. 7
  • The layout-invariant perceptual loss uses scene-parsing features to encourage layout fidelity and photorealism without suppressing generation diversity.The full objective combines this feature-matching loss with the multi-scale GAN loss.Source: §4.1.3 · p. 7

4.2 Style Transfer

  • The style-transfer stage uses the hallucinated scene as a reference, avoiding the need for a retrieved image with matching layout and desired attributes.The framework considers DPST and FPST; it post-processes their outputs to address smoothing, artifacts, or structural issues.Source: §4.2 · p. 8

5 RESULTS AND COMPARISON

The results section first evaluates scene generation, then presents attribute-manipulation results using hallucinated scenes as reference style images, and lastly discusses limitations.

5 RESULTS AND COMPARISON

  • The evaluation begins by assessing whether SGN can synthesize diverse, realistic-looking outdoor scenes.It also examines image quality, condition correctness, diversity, and adding or subtracting scene elements.Source: §5 · p. 8
  • The next part presents attribute-manipulation results from the two-stage framework, which uses hallucinated scenes as reference style images.Source: §5 · p. 8
  • The section concludes with a discussion of the approach’s limitations.Source: §5 · p. 8

5.1 Attribute and Layout Guided Scene Generation

This section evaluates layout- and attribute-conditioned scene generation, compares SGN with other methods and training variants, and explores diversity and scene editing. SGN’s results show stronger photorealism than Pix2pixHD at both scales, while offering control over transient attributes.

5.1.1 Training Details.

  • Training used batch size 40, 100 coarse-scale epochs, 10 fine-scale epochs, and then 70 epochs updating both scales.Fine-scale network training took about 10 days on a single GPU.Source: §5.1.1 · p. 8

5.1.3 Comparison with Image-to-Image Translation Models.

  • SGN generates realistic images on par with Pix2pixHD while enabling transient-attribute control that those comparison methods do not use.Attributes affect semantically related regions, such as clouds on the sky and winter on the ground.Source: §5.1.3 · p. 9, Figure 5 · p. 10

5.1.2 Ablation Study.

  • RNM and the proposed perceptual loss outperform other SGN variants qualitatively and quantitatively, with the perceptual loss improving photorealism and layout and attribute faithfulness.The ablation used unseen test layouts and fixed transient attributes to assess generation.Source: §5.1.2 · pp. 8–9, Table 2 · p. 9

5.1.3 Comparison with Image-to-Image Translation Models.

  • SGN produces higher-fidelity natural images than Pix2pixHD at both coarse and fine scales, as measured by IS and FID.Source: §5.1.3 · p. 9, Table 3 · p. 10
  • 66% of human-evaluation subjects selected SGN’s results as more realistic than Pix2pixHD’s.The study used 200 questions, with five subjects answering each.Source: §5.1.3 · p. 10

5.1.4 Diversity of the Generated Images.

  • With layout and attributes fixed, noise mainly changes local appearance, while transient attributes have a global effect without changing scene parts.Source: §5.1.4 · p. 10
  • The model can add or subtract scene elements through successive changes to a coarse semantic layout, with random attribute vectors enriching the possible edits.The examples add mountain, tree, and water elements to a layout initially containing sky and ground.Source: §5.1.5 · p. 10

5.2 Attribute Transfer

The framework manipulates transient attributes across varied conditions and combinations, with user evaluations favoring it over the retrieval-based comparison. Post-processing improves photorealism, while FPST is substantially faster than DPST.

5.2 Attribute Transfer

  • The framework produces photorealistic transfers across attributes such as Sunset, Spring, Fog, and Snow, combines attributes, and links attribute changes to scene elements.For example, Winter whitens grass, while Clouds mainly affect the sky and Fog blurs distant objects.Source: §5.2 · p. 11
  • 65% of the time, users preferred our approach over Laffont et al. [2014]; visual comparisons also favored its results, while FPST edged DPST by a very small margin.The user-study table reports preference rates for attribute manipulation.Source: §5.2 · pp. 11, 13, Table 4 · p. 13, Figure 10 · p. 13
  • Users can increase or decrease attribute strength by changing numerical condition values and move smoothly along the learned attribute manifold.The passage contrasts this control with retrieval-based editing, whose style images are limited by the database.Source: §5.2 · p. 13
  • The framework also produces season-transfer versions of oil paintings, although it is designed for natural images.The examples include three paintings depicting landscapes in different seasons.Source: §5.2 · pp. 13–14, Figure 12 · p. 14

5.2.1 Effect of Post-Processing and Running Times.

  • Post-processing addresses texture artifacts that make stylized outputs look more like paintings, making them photorealistic and more similar to the input.DPST-based outputs use a cross bilateral filter followed by screened Poisson enhancement; FPST-based outputs use screened Poisson enhancement.Source: §5.2.1 · p. 14, Figure 13 · p. 15
  • For 1024 × 512-pixel images, FPST-based manipulation takes 4 minutes, compared with 70 minutes for DPST.The reported times are for the framework's complete attribute-manipulation task.Source: §5.2.1 · p. 15, Table 5 · p. 15

5.3 Effect of Center-cropping

The generator uses a fixed 512 × 512 resolution and a center-cropped, resized input layout; cropping omits much of a panorama but does not pose a serious drawback in the reported examples.

5.3 Effect of Center-cropping

  • The scene generator operates at a fixed 512 × 512 resolution using the semantic layout of the center-cropped and resized input.For very wide images, this crop omits most of the source image.Source: §5.3 · p. 15
  • The authors report that center-cropping is not a serious drawback because style transfer constrains color transformations to image regions with the same semantic label.The section examines the effect on panoramic images from SUN360.Source: §5.3 · p. 15, Figure 14 · p. 15

5.4 Limitations

The framework can fail when either component does not function properly, with examples involving both scene generation and style transfer.

5.4 Limitations

  • Failures arise when a component underperforms: poor house synthesis weakens the manipulation, while style transfer can fail despite successful Fog and Night hallucinations.The examples distinguish failures in synthesized reference quality from failures transferring the look to the input.Source: §5.4 · p. 15

6 CONCLUSION

The paper concludes that its scene-generation network can synthesize attribute-conditioned reference images consistent with input layouts and supports multiple attributes in one model. It identifies local text-driven edits and single-shot manipulation as future directions.

6 CONCLUSION

  • The two-stage framework synthesizes an on-the-fly reference image that matches the input's semantic layout and exhibits desired attributes.The reference image is generated by the scene generation network for subsequent manipulation.Source: §6 · p. 16
  • Trained on ALS18K, the generative network hallucinates varied attributes reasonably well and supports edits combining multiple attributes in a unified model.Source: §6 · p. 16
  • Future directions include local edits from natural text queries and replacing the two-stage model with a single-shot architecture.Source: §6 · p. 16

Related papers

Loading 1808.07413v3…