Source-linked AI summary
You Only Need Adversarial Supervision for Semantic Image Synthesis
Vadim Sushko, Edgar Schönfeld, Dan Zhang, Juergen Gall, Bernt Schiele, Anna Khoreva
TL;DR
GAN-based semantic image synthesis has struggled to achieve high image quality with adversarial supervision alone, motivating perceptual losses that can constrain progress. OASIS uses a segmentation-based discriminator and globally or locally sampled 3D noise to produce high-quality, diverse outputs, improving the state of the art by an average of 6 FID and 5 mIoU points.
Problem
GAN-based semantic image synthesis often produces poor image quality with adversarial supervision alone, while VGG-based perceptual loss can limit progress in synthesis quality and diversity.
Method
OASIS trains a semantic-segmentation discriminator directly from label maps and injects a spatially sensitive 3D noise tensor into the generator for global or local resampling.
Results
OASIS improves the state of the art by an average of 6 FID and 5 mIoU points across ADE20K, Cityscapes, and COCO-stuff using adversarial supervision alone.
Takeaways & Limitations
OASIS synthesizes images with higher fidelity, better label-map alignment, greater diversity, and color and texture distributions closer to real images without perceptual loss.
Abstract
from arXiv · showhide
Despite their recent successes, GAN models for semantic image synthesis still suffer from poor image quality when trained with only adversarial supervision. Historically, additionally employing the VGG-based perceptual loss has helped to overcome this issue, significantly improving the synthesis quality, but at the same time limiting the progress of GAN models for semantic image synthesis. In this work, we propose a novel, simplified GAN model, which needs only adversarial supervision to achieve high quality results. We re-design the discriminator as a semantic segmentation network, directly using the given semantic label maps as the ground truth for training. By providing stronger supervision to the discriminator as well as to the generator through spatially- and semantically-aware discriminator feedback, we are able to synthesize images of higher fidelity with better alignment to their input label maps, making the use of the perceptual loss superfluous. Moreover, we enable high-quality multi-modal image synthesis through global and local sampling of a 3D noise tensor injected into the generator, which allows complete or partial image change. We show that images synthesized by our model are more diverse and follow the color and texture distributions of real images more closely. We achieve an average improvement of $6$ FID and $5$ mIoU points over the state of the art across different datasets using only adversarial supervision.
1 INTRODUCTION
OASIS addresses limitations of adversarially supervised semantic image synthesis with a segmentation-based discriminator and spatially controllable 3D noise sampling. It reports improved quality, diversity, and alignment while using only adversarial supervision.
- Semantic image synthesis renders realistic images from user-specified layouts for applications including content creation, image editing, and training-data generation.
- OASIS redesigns the discriminator as an encoder-decoder semantic segmentation network trained directly with semantic label maps.Its (N+1)-class loss provides semantically aware pixel-level feedback to the generator.
- 3D noise sampling enables multi-modal synthesis with either global scene changes or local changes to specific semantic regions and arbitrary areas.Noise is sampled globally channel-wise or locally pixel-wise.
- 6 FID and 5 mIoU points are the average improvements over the state of the art across ADE20K, Cityscapes, and COCO-stuff.The reported results rely only on adversarial supervision.
2 RELATED WORK
Prior semantic image synthesis methods use diverse generator and discriminator designs, often relying on image encoders and perceptual losses. OASIS instead uses a segmentation-based discriminator and direct 3D noise resampling for multi-modal outputs.
- Semantic image synthesis has evolved from Pix2pix’s encoder-decoder generator and PatchGAN discriminator to multiple generator and discriminator modifications.
- Discriminator architectures: OASIS replaces conventional multi-scale image classifiers with a pixel-wise semantic segmentation discriminator supervised directly by semantic label maps.The paper presents this as the first application of adversarial semantic segmentation loss to semantic image synthesis.
- Generator architectures: SPADE injects label-map information through spatially adaptive normalization, while earlier multi-modal approaches use image encoders to extract image style.
- Comparison with SPADE: OASIS outperforms SPADE while using only adversarial supervision, a single segmentation-based discriminator, and direct 3D noise resampling instead of an image encoder.
- Perceptual losses: VGG-based perceptual loss became a default component for training semantic image synthesis generators after its introduction by CRN and adoption by Pix2pixHD.
3 OASIS MODEL
OASIS simplifies semantic image synthesis by replacing classification-based discrimination with segmentation-based feedback and by using spatially sampled 3D noise for controllable multimodal generation.
- 3.1 THE SPADE BASELINE: SPADE is used as a state-of-the-art baseline with multi-scale PatchGAN discriminators, an image encoder, and adversarial, feature-matching, and VGG perceptual losses.The baseline is resource demanding during training and testing.
- 3.2 OASIS DISCRIMINATOR: OASIS redesigns the discriminator as a semantic segmentation network that directly uses label maps as ground truth, strengthening spatially aligned feedback.The discriminator predicts per-pixel classes rather than making only a global real/fake decision.
- 3.2 OASIS DISCRIMINATOR: The discriminator assigns real pixels to N semantic classes and all fake pixels to an additional class, using class-balanced (N+1)-class cross-entropy.Class balancing gives semantic classes equal contribution and encourages learning of less-represented classes.
- 3.2 OASIS DISCRIMINATOR: LabelMix mixes real and fake images with a label-conditioned binary mask and penalizes inconsistent discriminator logits with an L2 consistency loss.Unlike random CutMix masks, label-conditioned mixing preserves consistency between pixel classes and scene layout.
- 3.3 OASIS GENERATOR: OASIS replaces SPADE’s image encoder with a 64×H×W noise tensor concatenated with the label map and injected at every spatially adaptive normalization layer.Global, per-channel, per-segment, or per-pixel resampling enables whole-scene or localized changes.
- 3.3 OASIS GENERATOR: Qualitative ADE20K comparisons report better perceptual quality and structure from OASIS trained with only adversarial supervision.The comparison emphasizes finer textures, more natural colors, and improved rendering of small or rare semantic classes.
4 EXPERIMENTS
Experiments show that OASIS improves image quality, label-map alignment, and diversity using adversarial supervision alone. Its segmentation-based discriminator is the main source of the performance gain, while 3D noise enables global or local multimodal variation.
- Main results: OASIS improves over the state of the art by an average of 6 FID and 5 mIoU points across ADE20K, COCO-stuff, and Cityscapes.
- Main results: OASIS produces finer textures, more natural colors, and plausible images for small or rarely occurring semantic classes.
- Multi-modal synthesis: Global or local resampling of 3D noise changes the whole image or selected objects while preserving the rest of the scene.The noise modulates generator features at multiple layers and scales.
- Multi-modal synthesis: OASIS improves diversity and quality over SPADE+ with an image encoder, whereas SPADE+ exhibits a quality-diversity tradeoff between 3D noise and perceptual loss.For SPADE+, 3D noise improves diversity at the cost of quality, while perceptual loss improves quality at the cost of diversity.
- Ablations: Replacing the SPADE+ discriminator with the OASIS discriminator improves FID and mIoU by more than 30 points while keeping the generator fixed.Changing to the lighter OASIS generator causes only a 0.3 FID and 0.5 mIoU degradation.
- Ablations: Alternative discriminators perform well only with perceptual supervision, while the OASIS discriminator performs strongly without it.
5 CONCLUSION
OASIS is presented as a simpler semantic image synthesis model that uses adversarial supervision alone. Its segmentation-based discriminator and 3D noise sampling support high-fidelity, diverse outputs without perceptual loss.
- OASIS uses a segmentation-based discriminator that provides spatially and semantically aware feedback without perceptual-loss supervision.
- The model generates diverse multimodal outputs by globally or locally resampling 3D noise to change whole scenes or individual objects.
- OASIS improves image quality and diversity while being simpler and more lightweight than previous methods.
A.1 SUMMARIZED MAIN ABLATION OVER TWO DATASETS
The ablation attributes the main performance gain to the OASIS discriminator, while 3D noise targets diversity and perceptual loss can reduce performance and diversity. OASIS improves rare-class IoU, with balancing contributing substantially to that gain.
- Main findings: The OASIS discriminator provides the main performance gain, while the lighter generator does not improve performance despite using significantly fewer parameters.LabelMix is identified as a second source of improvement.
- Noise and perceptual loss: 3D noise can lower mIoU because diversity complicates evaluation by the pretrained segmentation network, while its purpose is to improve MS-SSIM diversity rather than FID.
- Noise and perceptual loss: Perceptual loss can hurt performance and diversity by biasing the generator toward ImageNet features and encouraging more standard semantic features.This can nevertheless raise mIoU by making external pretrained segmentation easier.
- Noise and perceptual loss: OASIS achieves high quality without perceptual loss, whereas SPADE+ depends strongly on it because its discriminator provides a weaker generator training signal.The loss curves support this explanation during training.
- Rare-class performance: OASIS’s mIoU improvement mainly comes from better IoU on less-represented semantic classes.The per-class analysis orders classes by their pixel-wise frequency in the training images.
- Rare-class performance: On ADE20K, rare classes covering less than 3% of images receive over 40% relative gain over the baseline, mainly from per-class balancing in the OASIS loss.
A.4 ABLATION ON LABELMIX
LabelMix improves consistency regularization by generating binary masks that respect semantic-class boundaries, unlike CutMix. The comparison evaluates their effects on FID and mIoU for OASIS on Cityscapes.
- Mask construction: LabelMix respects boundaries between semantic classes in the label map, whereas CutMix does not.Both methods generate binary masks for consistency regularization.
- Interpretation: CutMix may introduce label noise because it creates a dense patchwork that conflicts with semantic classes and real-fake identities.LabelMix instead generates masks according to the label map, providing natural borders.
A.5 ABLATION ON FEATURE MATCHING LOSS
Feature matching affects FID notably mainly without perceptual loss, stabilizing SPADE+ but worsening OASIS under the tested settings. OASIS does not exhibit the training collapses observed for SPADE+.
- Overall effect: Feature matching affects FID notably only when perceptual loss is absent for both SPADE+ and OASIS.
- SPADE+: Without perceptual loss, feature matching prolongs SPADE+ collapse and improves FID from 60.7 to 49.7.The passage also states that mIoU improves in this setting.
- OASIS: OASIS shows no training collapse without extra losses, but feature matching worsens FID by 0.8 points without perceptual loss.
- OASIS: With perceptual supervision present, feature matching degrades OASIS mIoU by 1.1 points.This indicates interference with OASIS’s strong semantic-segmentation adversarial supervision.
A.6 ABLATION ON USING MORE THAN ONE OASIS DISCRIMINATOR
A single OASIS discriminator matches the tested multi-discriminator alternative because its U-Net already integrates multi-scale information. The ablation also explains how spatial 3D noise supports global and regional manipulation.
- Discriminator count: Adding a second OASIS discriminator at scale 0.5 does not improve performance and slightly worsens it on Cityscapes.
- Discriminator count: The OASIS U-Net discriminator already captures multi-scale information through skip connections across encoder, decoder, and individual blocks.This explains why OASIS does not need SPADE’s two different-scale discriminators.
- Noise sampling: Image-level sampling broadcasts one 1D noise vector across the label map, producing a 3D noise tensor used with the generator.
- Noise injection: Noise is injected through conditional normalization at every generator layer, making OASIS spatially sensitive to both labels and noise.
- Noise manipulation: Because noise affects features at multiple resolutions, image-level training noise can support region-level manipulation at inference.
- Experimental scope: Extensive ablations were performed on ADE20K and Cityscapes, while COCO-stuff received only essential experiments because training can take up to four weeks.
B.1 COMPARISON TO OTHER METHODS
OASIS often produces more plausible images than prior methods, with finer textures and more diverse colors, while its noise sampling supports global and local variation. Its discriminator also enables image segmentation and subsequent image recreation from predicted label maps.
- OASIS often produces more visually plausible images than SPADE and CC-FPSE, commonly generating finer textures and more natural colors.
- OASIS usually generates brighter and more diverse colors, although some objects can receive outlier colors or textures that appear unnatural.
- Global or local resampling of OASIS’s 64×H×W noise tensor changes the entire image or selected regions while keeping the label map fixed.
- Latent-space interpolation with a fixed label map produces smooth, semantically meaningful transitions such as winter-to-summer and day-to-night changes.
- The OASIS discriminator reaches 40.0 mIoU on ADE20K validation and can predict label maps for unseen images before generating multiple recreations.
B.5 LABELMIX
OASIS combines LabelMix consistency regularization with a segmentation-based discriminator and a lighter 3D-noise generator. The model supports globally or locally varied outputs and has a documented failure mode involving outlier colors and textures.
- MULTI-MODAL IMAGE SYNTHESIS: Noise can be resampled globally or locally in marked regions, and the resulting images are generated in single forward passes rather than stitched together.
- LIMITATION: OASIS produces diverse outputs but can generate objects with outlier colors and textures.
- LABELMIX: LabelMix mixes real and fake images with a label-map-based binary mask and enforces consistency between discriminator logits for the mixed image and mixed individual predictions.
- LABELMIX: The OASIS discriminator uses a U-Net architecture with 22M parameters, exceeding SPADE’s 5.5M-parameter multi-scale PatchGAN to provide more informative generator feedback.
C.3 LEARNING OBJECTIVE AND TRAINING DETAILS
OASIS is trained with an (N+1)-class adversarial cross-entropy objective and LabelMix consistency regularization. Unlike SPADE, its objective omits VGG perceptual and GAN feature-matching losses, with VGG used only in ablations.
- OASIS uses (N+1)-class cross entropy as its adversarial loss and additionally regularizes the discriminator with LabelMix consistency.
- The objective uses discriminator logits on real and fake inputs, with noise z and label map t forming the generator input.
- SPADE instead uses hinge adversarial loss together with VGG perceptual and feature matching losses.
- Training uses 256×256 resolution for ADE20K and COCO-Stuff, 256×512 for Cityscapes, and Adam with learning rates 0.0001 for G and 0.0004 for D.