Source-linked AI summary
Local Class-Specific and Global Image-Level Generative Adversarial Networks for Semantic-Guided Scene Generation
Hao Tang, Dan Xu, Yan Yan, Philip H. S. Torr, Nicu Sebe
TL;DR
Semantic-guided scene generation remains difficult for small objects and detailed local texture, especially with global image-level generation. LGGAN uses semantic-guided class-specific local sub-generators alongside global generation, fused in a jointly trained network. The paper reports superior performance and new state-of-the-art results on cross-view image translation and semantic image synthesis, while noting failures for some categories with limited training samples.
Problem
Global image-level scene generation struggles with small objects, detailed local texture, class imbalance, and differences in object size.
Method
LGGAN combines semantic-guided class-specific local generation, image-level global generation, learned fusion weight maps, dual discrimination, and a classification module in an end-to-end network.
Results
LGGAN achieves new state-of-the-art results on both cross-view image translation and semantic image synthesis tasks.
Takeaways & Limitations
The joint local-global design supports richer scene details while addressing small-object and local-texture difficulties in semantic-guided generation.
Takeaways & Limitations
LGGAN fails to generate photo-realistic large objects on Cityscapes and several categories on ADE20K, likely because of few training samples.
Abstract
from arXiv · showhide
In this paper, we address the task of semantic-guided scene generation. One open challenge in scene generation is the difficulty of the generation of small objects and detailed local texture, which has been widely observed in global image-level generation methods. To tackle this issue, in this work we consider learning the scene generation in a local context, and correspondingly design a local class-specific generative network with semantic maps as a guidance, which separately constructs and learns sub-generators concentrating on the generation of different classes, and is able to provide more scene details. To learn more discriminative class-specific feature representations for the local generation, a novel classification module is also proposed. To combine the advantage of both the global image-level and the local class-specific generation, a joint generation network is designed with an attention fusion module and a dual-discriminator structure embedded. Extensive experiments on two scene image generation tasks show superior generation performance of the proposed model. The state-of-the-art results are established by large margins on both tasks and on challenging public benchmarks. The source code and trained models are available at https://github.com/Ha0Tang/LGGAN.
1. Introduction
Semantic-guided scene generation targets cross-view translation and semantic image synthesis, but global image-level methods struggle with local structure, detailed texture, and small objects. LGGAN addresses these issues by combining class-specific local generation with global generation through learned fusion.
- Motivation: Semantic-guided scene generation includes cross-view image translation and semantic image synthesis, using semantic maps to guide object relations or image generation.Cross-view translation is ill-posed from a single RGB image, while semantic maps provide target-view correspondences and transformations.
- Challenges: Global image-level generation treats classes similarly, allowing large, frequent classes to dominate learning and disadvantaging small objects.Shared parameters and unequal pixel counts make classes such as roads and skies easier to learn than poles and traffic lights.
- Proposed approach: LGGAN uses semantic guidance to extract class-specific feature maps and separately generate different class regions with dedicated sub-generators.The local generator models scene classes in a local context using a shared encoded feature map.
- Proposed approach: LGGAN combines an image-level global generator, a class-specific local generator, and learned pixel-level weight maps in an end-to-end joint network.The global branch learns overall appearance, the local branch separately generates object classes, and fusion weights combine their outputs.
- Results: Experiments on cross-view translation and semantic image synthesis report significantly better results than state-of-the-art methods on both tasks.The paper evaluates cross-view translation on Dayton and CVUSA and semantic image synthesis on Cityscapes and ADE20K.
2. Related Work
Related work applies semantic guidance to scene generation, but global image-level methods remain unsatisfactory for detailed local texture. LGGAN therefore emphasizes complementary local and global generation learned jointly.
- Semantic-guided generation: Generative adversarial methods for semantic-guided scene generation use semantic maps to improve image generation performance.The related discussion frames semantic maps as useful guidance for the two scene-generation tasks.
- Motivation: Global image-level generation methods often produce unsatisfactory detailed local texture, motivating a model that addresses both global structure and local details.LGGAN jointly learns local and global branches so their benefits can be used together during optimization.
3. The Proposed LGGAN
LGGAN combines semantic-guided class-specific local generation with global image-level generation, using learned pixel-level fusion and dual discriminators. Its local generator filters features by semantic class, reconstructs class outputs, and learns discriminative class representations while handling void classes in classification.
- LGGAN Structure: LGGAN comprises semantic-guided class-specific, global image-level, and weight-map generation components for jointly modeling local details and global layout.The weight-map generator fuses local and global outputs, while the framework also uses semantic-guided and image-guided discriminators.
- Class-Specific Local Generation: Each local branch has independent parameters and specializes in one semantic class, reducing interference from large object classes during joint optimization.The local network is intended to address training-data imbalance and object-size differences that make small objects and scene details difficult to generate.
- Class-Specific Local Generation: Semantic masks filter an upsampled feature map into class-specific maps Fi, which separate convolutional branches use to produce class-level outputs.For each class, the filtered map is formed as Fi = Mi * f′, and the resulting outputs can be added or concatenated before convolution.
- Class-Specific Discriminative Feature Learning: A classification-based module pools packed local feature maps and predicts object-class probabilities to make class-specific representations more discriminative.The module uses semantic-guided averaging pooling followed by a fully connected layer; void-class features are excluded from the classification loss.
- Global-Local Fusion and Discrimination: The fusion branch predicts pixel-level weights for global and local outputs, while dual discriminators use semantic maps and, for cross-view translation, conditional images.The two-channel weights are normalized with channel-wise softmax, and the total adversarial loss combines image-guided and semantic-guided discriminator losses.
4. Experiments
Experiments evaluate LGGAN on cross-view image translation and semantic image synthesis, showing stronger visual quality and competitive quantitative performance across benchmarks. Ablations indicate that class-specific generation, classification-based feature learning, and weight-map fusion each contribute to the full model.
- Cross-View Image Translation: LGGAN is evaluated on Dayton and CVUSA for cross-view image translation using IS, Acc., KL, SSIM, PSNR, and SD.The experiments use the a2g direction and compare against several state-of-the-art methods.
- Cross-View Image Translation: LGGAN outperforms Pix2pix++, X-Fork++, and X-Seq++ and achieves better results than SelectionGAN on most metrics.SelectionGAN is slightly better on SSIM, PSNR, and SD, while LGGAN produces more photo-realistic results qualitatively.
- Cross-View Image Translation: LGGAN produces clearer details on cars, buildings, roads, and trees than SelectionGAN in qualitative comparisons on Dayton and CVUSA.Figure 4 presents the qualitative comparison in the a2g direction.
- Semantic Image Synthesis: LGGAN outperforms competing methods on Cityscapes and ADE20K using mIoU, pixel accuracy, and FID, with SIMS the only method better on Cityscapes.The comparison includes Pix2pixHD, CRN, SIMS, and GauGAN.
- Ablation Study: The full model improves over the global baseline from 62.3 to 68.4 mIoU and reduces FID from 71.8 to 57.7.The ablation sequence adds local generation, convolutional local fusion, classification-based feature learning, and weight-map fusion.
- Semantic Image Synthesis: Users prefer LGGAN outputs over competing methods on both Cityscapes and ADE20K, while LGGAN also generates fewer visual artifacts than GauGAN.Table 3 reports the user preference study, and Figure 5 provides qualitative comparisons.
- Analysis: Class-specific feature maps highlight road, vegetation, and car regions, while generated semantic maps improve local texture and small-object representation.The visualizations emphasize cars, traffic signs, and poles as examples of the targeted details.
5. Conclusion
The paper concludes that LGGAN combines global image-level and local class-level generation through three generation branches. Experiments report superior performance and new state-of-the-art results on cross-view image translation and semantic image synthesis.
- Conclusion: LGGAN contains global image-level, local class-level, and pixel-level fusion weight-map generation branches.The framework also includes a class-specific local network and a classification module for discriminative feature representations.
- Conclusion: The class-specific local network addresses imbalanced training data and differences in scene-object size during joint learning.The conclusion identifies these factors as motivations for the local generation design.
- Conclusion: Experimental results establish new state-of-the-art results on both cross-view image translation and semantic image synthesis tasks.The conclusion reports superiority of the proposed approach across both task types.
6. More Results on Semantic Image Synthesis
LGGAN achieves visually better semantic image synthesis than competing methods and benefits from jointly combining local and global generation. Its advantages are most evident for local texture and small objects, while failures remain for some large objects and underrepresented categories.
- Comparison with Existing Methods: LGGAN achieves visually better results than Pix2pixHD, CRN, SIMS, and GauGAN on Cityscapes and ADE20K.The comparisons use randomly selected samples without cherry-picking for visualization.
- Ablation Results: The full Local + Global model generates visually better results than the Local and Global settings on both datasets.This supports joint adversarial learning across local and global contexts.
- Semantic Map Evaluation: LGGAN generates better semantic maps than GauGAN, especially for local texture and small objects, on Cityscapes and ADE20K.Semantic maps are produced using DRN-D-105 for Cityscapes and UperNet101 for ADE20K.
- Typical Failure Cases: LGGAN cannot generate photo-realistic appearances for some large Cityscapes objects, including cars, trucks, and buses.The reported Cityscapes failure cases concern large object categories.
- Typical Failure Cases: LGGAN cannot generate several ADE20K categories, including people, cars, houses, and food.The paper attributes these failure cases mainly to the small number of training samples.