PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Photorealism Enhancement

Stephan R. Richter, Hassan Abu AlHaija, Vladlen Koltun

arXiv:2105.04619v1cs.CVcs.AIcs.GRcs.LG

TL;DR

Deep-learning image translation can capture photographic appearance but may be disconnected from rendering pipelines, hard to control, and artifact-prone. This paper enhances rendered images using rendering buffers and layout-aware training, and reports the most realistic results among its baselines.

  • Problem

    Deep-learning methods can synthesize photographic appearance, but their disconnection from rendering pipelines, limited controllability, and artifacts hinder their use alongside conventional rendering.

    Source: §1 · p. 1

  • Method

    A convolutional network uses intermediate rendering buffers to enhance images, with layout-aware patch sampling and an adversarial objective for multi-level perceptual supervision.

    Source: §1 · p. 1

  • Results

    The approach consistently produces the most realistic results by a wide margin and outperforms all baselines in the reported experiments.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Enhancements match the target dataset’s style while remaining geometrically and semantically consistent with inputs, with less convincing changes for uncommon objects such as close-up pedestrians.

    Source: §5 · p. 12

    The compared photo-style-transfer methods depend on favorable reference images and may produce unrealistic color shifts or temporal instability as inputs change.

    Source: §4.2 · p. 10

Abstract

from arXiv · show

We present an approach to enhancing the realism of synthetic images. The images are enhanced by a convolutional network that leverages intermediate representations produced by conventional rendering pipelines. The network is trained via a novel adversarial objective, which provides strong supervision at multiple perceptual levels. We analyze scene layout distributions in commonly used datasets and find that they differ in important ways. We hypothesize that this is one of the causes of strong artifacts that can be observed in the results of many prior methods. To address this we propose a new strategy for sampling image patches during training. We also introduce multiple architectural improvements in the deep network modules used for photorealism enhancement. We confirm the benefits of our contributions in controlled experiments and report substantial gains in stability and realism in comparison to recent image-to-image translation methods and a variety of other baselines.

1 INTRODUCTION

The paper combines conventional rendering with deep learning, using rendering buffers to enhance realism while preserving scene content. It also targets artifacts through layout-aware patch sampling and a multi-level adversarial objective, and reports stronger realism than baselines.

1 INTRODUCTION

  • The paper frames photorealism as an unresolved gap: even sophisticated real-time games retain a perceptible difference from reality.Deep-learning methods capture photographic aspects but are often disconnected from rendering pipelines, hard to control, and artifact-prone.Source: §1 · p. 1
  • The network uses G-buffers containing scene information to modulate rendered-image features during photorealism enhancement.The buffers provide information about geometry, materials, and lighting.Source: §1 · p. 1
  • Scene-layout differences between commonly used datasets can explain artifacts such as hallucinated objects, motivating patch sampling that aligns the datasets.A new adversarial objective also encourages geometric and semantic consistency with the input.Source: §1 · p. 1
  • The approach enhances rendered images with dataset-specific styles, including gloss on cars, greener hills, and smoother roads.Training with Cityscapes, KITTI, or Mapillary Vistas produces outputs reflecting the respective dataset’s visual style.Source: §1 · p. 1
  • The authors introduce metrics designed to reduce the influence of mismatched scene layouts and assess realism at multiple levels.Their analysis suggests standard metrics can confound differences in style and content.Source: §1 · p. 1
  • Across experiments, the approach produces the most realistic results by a wide margin and outperforms all baselines.The authors report a new state of the art in photorealism enhancement.Source: §1 · p. 1

2 RELATED WORK

Related work spans data-driven synthesis, image-based and style-transfer methods, unpaired translation, and hybrids with rendering. The paper positions its approach as enhancing rendered images with scene buffers while addressing artifacts through semantic-aware training and sampling.

2 RELATED WORK

  • Conditional image synthesis from label maps is underconstrained, can produce artifacts and temporal inconsistency, and requires labor-intensive real-data annotations.This approach instead enhances already-rendered images and does not require real-data annotations.Source: §2 · p. 2
  • Image-based rendering requires prior scene capture and can artifact when views depart from recorded camera trajectories, while photo style transfer depends on a favorable reference image.For changing inputs, photo style transfer may produce unrealistic color shifts or temporal instability.Source: §2 · pp. 2–3
  • Unpaired translation avoids reference images but must infer cross-domain correspondences, which can yield realistic-looking outputs inconsistent with their inputs.Prior work has explored cycle consistency, attention, temporal regularization, geometric constraints, and other ways to improve consistency.Source: §2 · p. 3
  • Semantic guidance can aid category-specific processing, but annotated-data requirements and limited dataset diversity constrain common approaches.The paper uses multiple discriminators on feature maps from different layers of a pretrained VGG network.Source: §2 · pp. 3–4
  • The paper addresses layout mismatch with a training-free patch-level sampling strategy, retaining more diverse samples than image-level matching.The cited image-level matching approach requires a dedicated network and reduces the available training samples.Source: §2 · p. 4
  • Earlier rendering hybrids used G-buffers for shading or adversarial realism; this work introduces a different network structure, discriminator, and training objective.The authors report significantly better results than those prior hybrid approaches.Source: §2 · p. 4
  • Unlike an earlier indoor-scene approach requiring paired low- and high-quality renderings, this work assumes higher-quality modern-game inputs and uses auxiliary rendering buffers.The authors report temporal consistency and higher measured realism than high-production-value commercial games.Source: §2 · p. 4

3 METHOD

The method enhances rendered images by conditioning a convolutional network on rendering buffers, then trains it with perceptual and adversarial supervision. It also addresses dataset layout mismatch through patch sampling and uses multiple discriminator levels to assess realism.

3.1 Overview

  • Rendered images and multiscale G-buffer features feed a modified HRNet, trained with LPIPS structure preservation and a perceptual discriminator for realism.The network preserves full-resolution detail and replaces batch normalization with rendering-aware denormalization.Source: §3.1 · p. 4

3.2 Leveraging conventional rendering pipelines

  • The G-buffer encoder processes scene buffers in multiple streams, fuses them by object ID, and supplies multiscale features that RAD uses to scale and shift image features.Controlled experiments find RAD performs better than SPADE for ingesting G-buffer features, and substantial enhancement remains possible with only a subset of buffers.Source: §3.2 · pp. 5–6

3.3 Perceptual discriminator

  • A perceptual discriminator combines pretrained semantic segmentation and VGG features, with separate discriminator networks specializing in different perceptual levels.The segmentation and feature-extraction networks are frozen during training; semantic predictions also allow training without ground-truth annotations.Source: §3.3 · p. 6

3.4 Layout differences cause artifacts

  • GTA and Cityscapes have different positional scene-layout distributions, and GAN translation can hallucinate trees in the sky or star logos at the image bottom.The method shrinks crops to about 7% of an image and matches cross-dataset patches using VGG feature cosine similarity above 0.5; the crop-size change alone significantly improves results.Source: §3.4 · p. 7

3.5 Implementation & training details

  • Discriminator updates are randomly throttled according to performance to balance learning and reduce overfitting; training uses Adam for 1M iterations, or 600K in controlled experiments.The training setup also uses an LPIPS regularizer and gradient penalties for all discriminators.Source: §3.5 · p. 7

4 EVALUATION

The evaluation introduces layout-aware, multilevel metrics, compares the method with varied baselines, and tests its design choices in controlled experiments. Across comparisons, the method is judged more realistic than the baselines, while the experiments identify benefits from patch matching, G-buffers, RAD modules, and the perceptual discriminator.

4.1 Metrics

  • To address KID’s layout mismatch, sKVD computes squared MMD between VGG-16 features on semantically corresponding patches, with metrics from six relu layers.The six layers are relu1-2, relu2-2, relu3-3, relu4-3, relu5-3, and the listed relu layer family.Source: §4.1 · p. 8

4.2 Comparison to prior work

  • Among baseline types, image-to-image translation methods perform best and Cycada gives the best results within this group; despite using more explicit semantic information than other methods using a perceptual loss, it still hallucinates extraneous objects.Source: §4.2 · p. 10

4.3 Perceptual experiment

  • In 50,000 randomized comparisons, our enhancements were judged more realistic than every baseline, significantly for all methods.The experiment paired enhancements for 500 GTA images, with each pairing shown 10 times.Source: §4.3 · p. 10, Figure 15 · p. 11

4.4 Controlled experiments

  • Smaller uniform patches reduce mismatch and hallucinations; matching patches further reduce medium-to-high-level sKVD while slightly raising the lowest-level sKVD.Larger uniform patches show stronger hallucination artifacts.Source: §4.4 · p. 11, Figure 16 · p. 12
  • Without G-buffers, realism falls at all but the lowest level; VIPER buffers improve all levels, while the full set favors mid- and high-level metrics.The strongest VIPER effects are at low and medium levels.Source: §4.4 · p. 11
  • RAD modules yield consistently high quality unlike volatile SPADE; PatchGAN is significantly less realistic, and projection and adaptive backpropagation help at different perceptual levels.The combination of projection and adaptive backpropagation benefits results when all levels are considered.Source: §4.4 · pp. 11–12, Figure 17 · p. 12

5 CONCLUSION

The method enhances rendered images while keeping them geometrically and semantically consistent with their inputs and matching the style of the target dataset. Its strongest results are on objects and scenes with clear real-data correspondences; less common content is modified less convincingly.

5 CONCLUSION

  • The approach produces high-quality enhancements that remain geometrically and semantically consistent with inputs while matching the respective dataset’s style.Reported examples include road textures, cars, and vegetation.Source: §5 · p. 12
  • Enhancements are strongest for objects and scenes with clear real-dataset correspondences; close-up pedestrians are modified less convincingly.Source: §5 · p. 12
  • Current unoptimized inference takes half a second on a GeForce RTX 3090; native GPU G-buffer inputs could allow deeper engine integration and possibly greater efficiency.Source: §5 · p. 12
  • Structural consistency can facilitate reuse of available ground-truth annotations for synthetic data, and enhanced GTA V and VIPER images will be released.Source: §5 · p. 12

Related papers

Loading 2105.04619v1…