Source-linked AI summary
TextureGAN: Controlling Deep Image Synthesis with Texture Patches
Wenqi Xian, Patsorn Sangkloy, Varun Agrawal, Amit Raj, Jingwan Lu, Chen Fang, Fisher Yu, James Hays
TL;DR
TextureGAN addresses the challenge of controlling object texture in deep image synthesis from sketches. It lets users place texture patches on sketched objects and uses local texture constraints to generate plausible images that follow the patches and sketched contours.
Problem
Existing deep image synthesis methods provide sketch and color control, but texture control requires propagating textures across object boundaries while respecting shape, lighting, and occluding parts.
Method
TextureGAN combines sketch-guided synthesis with user-placed texture patches and a local texture loss based on a texture discriminator and Gram matrix style loss.
Results
TextureGAN generates plausible images faithful to user controls, propagates textures across objects, and composes them along sketched contours across fashion datasets.
Takeaways & Limitations
The feed-forward system enables real-time editing while providing separate color and content controls and handling varied texture inputs.
Takeaways & Limitations
External texture fine-tuning can cause the network to forget knowledge learned during ground-truth pretraining, so ground-truth iterations are mixed into training.
Abstract
from arXiv · showhide
In this paper, we investigate deep image synthesis guided by sketch, color, and texture. Previous image synthesis methods can be controlled by sketch and color strokes but we are the first to examine texture control. We allow a user to place a texture patch on a sketch at arbitrary locations and scales to control the desired output texture. Our generative network learns to synthesize objects consistent with these texture suggestions. To achieve this, we develop a local texture loss in addition to adversarial and content loss to train the generative network. We conduct experiments using sketches generated from real images and textures sampled from a separate texture database and results show that our proposed algorithm is able to generate plausible images that are faithful to user controls. Ablation studies show that our proposed pipeline can generate more realistic images than adapting existing methods directly.
1. Introduction
TextureGAN addresses the limited investigation of fine-grained texture control in deep image synthesis by letting users guide sketched objects with localized texture patches. It combines a drag-and-drop interface with a local texture loss to synthesize textures that follow object structure and support interactive editing.
- Fine-grained texture control had received little investigation compared with sketch, color, and coarse style-transfer controls.
- TextureGAN lets users drag example textures onto sketched objects, where the network applies the textures to the indicated objects.
- The texture-fill task requires high-resolution detail synthesis, boundary-aware propagation, implicit object segmentation, and handling of foreshortening, lighting, and occlusion.
- Training pairs are automatically extracted from real photographs, while a local texture loss fine-tunes the network for diverse textures unseen on ground-truth objects.
- The proposed interface places localized texture patches on sparse sketched object boundaries, and the generative network directly operates on both inputs.
- The network uses novel losses, including a local texture loss, to encourage handling of new textures never seen on existing objects.
2. Related Work
Related work spans non-parametric texture and image-synthesis methods, GAN-based texture synthesis, conditional image synthesis, and sketch or color-guided generation. TextureGAN differs by combining structural sketches, color patches, and texture swatches while introducing new loss functions for texture control.
- Image Synthesis: Parametric deep-learning methods, including GANs, offer a different image-synthesis paradigm from non-parametric approaches.
- Controllable Image Synthesis and Conditional GANs: Conditional GANs improve controllability by conditioning synthesis on inputs beyond noise and adding discriminators or losses for desired properties.
- Controllable Image Synthesis and Conditional GANs: Prior sketch- and color-guided systems generate outputs consistent with sketches or color strokes, but TextureGAN supports structural sketches, color patches, and texture swatches simultaneously.
- Texture Synthesis and Style Transfer: Texture synthesis generates images with visually similar textures, while style transfer combines content layout with style-image texture.
- Texture Synthesis and Style Transfer: Deep style transfer uses feature correlations captured by a Gram matrix to represent texture characteristics and synthesize or transfer styles.
- Texture Synthesis and Style Transfer: TextureGAN’s texture discriminator compares generated and ground-truth texture patches, differing from prior adversarial texture methods that operate on VGG features or pixels.
- Texture Synthesis and Style Transfer: The task resembles image completion, but its missing regions are larger and constrained by sketch, color, or texture; related texture losses may operate only in feature space.
3. TextureGAN
TextureGAN generates images from sketches overlaid with localized texture and color controls, using a conditional network trained with complementary losses. Its local texture loss and external texture fine-tuning target faithful texture propagation while preserving semantic structure and color control.
- TextureGAN: TextureGAN lets users place arbitrary-size texture patches on sketched objects to guide localized appearance generation.The network operates on sketch, texture, and color controls, using five input channels with separate sketch, texture, and color representations.
- TextureGAN: A local texture loss addresses uncertain correspondence between input patches and unconstrained sketch regions by encouraging faithful texture reproduction and propagation.The loss uses texture discrimination and Gram-matrix style matching, supplemented by pixel loss on local patches.
- TextureGAN: External texture fine-tuning uses a separate texture-only database to improve propagation of diverse textures, including textures unseen on ground-truth objects.The method first reproduces shoe, handbag, and clothing photos from synthetically sampled controls, then fine-tunes for broader texture generalization.
- TextureGAN: The training objective combines feature, adversarial, style, pixel, and color losses to jointly support sketch-conforming structure, realistic details, texture fidelity, and color constraints.Feature loss uses deeper VGG-19 representations, while color loss compares Lab ab channels with the ground truth.
- TextureGAN: Ablations show that removing texture loss harms texture propagation, while removing adversarial loss produces blurry results.With the complete objective, color and texture are propagated across the handbag; local adversarial loss also tends to improve texture consistency throughout the object.
4. Training Setup
TextureGAN is trained on object-centric fashion datasets using automatically generated sketches, texture patches, and segmentation-aware foreground sampling, followed by external texture fine-tuning.
- Datasets: Training uses handbags, shoes, and clothes datasets containing varied colors, materials, and patterns for product-design applications.Supervised pairs are constructed from ground-truth photographs converted to Lab color space.
- Data preparation: Segmentation masks identify foreground regions so local texture loss compares output patches with input textures rather than background patches.White backgrounds are used for handbags and shoes, while clothes use segmentation information.
- Data preparation: Sketches are generated from edge detection, clothing segmentation boundaries, xDoG variation, and additional synthetic sketches.Different procedures are used for handbags and shoes versus clothes.
- Data preparation: Texture constraints are created by randomly cropping one or two foreground patches, varying their locations and sizes, and converting them to normalized Lab values.For clothes, patches are extracted from specified semantic regions.
- Fine-tuning: External fine-tuning uses 6,300 leather-like textures sampled from 130 manually filtered high-resolution images.Roughly 50 crops of size 256x256 are sampled from each source image.
- Optimization: Pre-training uses Adam with learning rate 1e-2, while fine-tuning lowers the rate to 1e-3 and changes the loss weights.Most models use 128x128 inputs; one clothes model uses 256x256.
5. Results and Discussions
Experiments show that local texture losses and external texture fine-tuning improve texture propagation, while the system preserves sketch boundaries, supports multiple patches, and generalizes to human-drawn sketches.
- Ablation Study: The complete objective propagates color and texture across handbags, whereas removing texture loss preserves only details inside the input patch.The ablation identifies texture propagation as a distinct benefit of the texture loss.
- External Texture Fine-tuning Results: External texture fine-tuning is critical for propagating sharp regular textures on sophisticated handbag and clothes datasets, while shoes work well without it.Shoes have limited structural variation, leaving more network capacity for texture propagation.
- External Texture Fine-tuning Results: Fine-tuning with local style, pixel, and texture losses improves consistency across objects; local texture discriminator loss resembles the input texture more than style loss.Both local style loss and local texture discriminator loss outperform no fine-tuning or local pixel loss for propagation.
- Held-out sketches: Generated textures follow object boundaries exactly even when an input patch extends beyond the sketch, with center textures larger and lower regions darker.The shading is described as consistent with top lighting or ambient occlusion.
- Multiple texture patches: At 256x256 resolution, multiple texture patches are propagated within semantic clothing regions while respecting sketch boundaries.The clothes dataset contains substantial structural and texture variation across multiple semantic regions.
- Human-drawn sketches: Results on human-drawn handbag sketches remain high quality despite differing from the synthetically generated training sketches.This evaluates the system beyond its synthetic sketch-generation procedure.
6. Conclusion
TextureGAN enables sketch-based object synthesis with precise texture-patch control and real-time editing, producing texture compositions that follow sketched contours on fashion datasets.
- Conclusion: TextureGAN lets users sketch object structure and control generated details by placing texture patches.The system is feed-forward, allowing users to see edit effects in real time.
- Conclusion: The pipeline handles varied texture inputs and generates compositions that follow sketched contours on fashion datasets.It also operates in Lab color space to separate color and content controls.