Source-linked AI summary
Diversified Texture Synthesis with Feed-forward Networks
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, Ming-Hsuan Yang
TL;DR
Existing feed-forward texture methods trade efficiency against generality, diversity, and visual quality. The paper introduces a controllable feed-forward network for diverse multi-texture synthesis, supported by diversity loss, recentered Gram losses, and incremental training. Experiments report comparable results to single-texture networks while greatly reducing model size, with extensions to stylization.
Problem
Existing feed-forward methods often build one network per texture, produce visually identical outputs, and generate less satisfying visual effects.
Method
The paper uses a generator conditioned on noise and a one-hot texture selection unit, with diversity loss, recentered Gram losses, and incremental training.
Results
The model synthesizes diverse multiple textures in one network, achieves comparable results to existing single-texture networks, greatly reduces model size, and extends to multi-style transfer.
Takeaways & Limitations
A single user-controllable feed-forward model can synthesize many textures, interpolate between them, and support image stylization.
Abstract
from arXiv · showhide
Recent progresses on deep discriminative and generative modeling have shown promising results on texture synthesis. However, existing feed-forward based methods trade off generality for efficiency, which suffer from many issues, such as shortage of generality (i.e., build one network per texture), lack of diversity (i.e., always produce visually identical output) and suboptimality (i.e., generate less satisfying visual effects). In this work, we focus on solving these issues for improved texture synthesis. We propose a deep generative feed-forward network which enables efficient synthesis of multiple textures within one single network and meaningful interpolation between them. Meanwhile, a suite of important techniques are introduced to achieve better convergence and diversity. With extensive experiments, we demonstrate the effectiveness of the proposed model and techniques for synthesizing a large number of textures and show its applications with the stylization.
1. Introduction
The paper targets feed-forward texture synthesis methods that sacrifice generality, diversity, and visual quality. It proposes one controllable network for multiple textures, with interpolation and techniques for diversity and training stability.
- Existing feed-forward methods typically build one network per texture, produce visually identical outputs, and can generate less satisfying visual effects.
- The proposed generative network synthesizes diverse outputs for multiple textures within a single network.
- A one-hot selection unit provides user control over the texture type synthesized by the network.
- Interpolating selection units enables feed-forward generation of new textures between learned texture types.
- A diversity loss correlates outputs with input noise by penalizing feature similarities among different mini-batch samples.
- The work combines the generative network, diversity loss, and incremental learning to synthesize multiple textures in a user-controllable manner.
2. Related Work
Prior texture synthesis methods use statistical or patch-based procedures, while neural approaches use learned feature statistics for reconstruction or generation. Feed-forward networks improve efficiency, and concurrent work extends one network to multiple styles through normalization parameters.
- Traditional parametric methods represent textures with statistical models and iteratively optimize random noise to match relevant image statistics.
- Non-parametric methods grow images from seeds by repeatedly sampling similar pixels or patches from an example texture.
- Neural texture synthesis uses intermediate CNN outputs as image statistics, including feature statistics matched during noise optimization.
- Feed-forward methods accelerate synthesis by learning networks that minimize differences between ground-truth and generated feature statistics.
- Dumoulin et al. handle multi-style transfer in one network by specializing scaling and shifting parameters after normalization for each texture.
3. Proposed Algorithm
The proposed network synthesizes multiple textures from noise and a one-hot selection unit, while texture, diversity, and incremental-training techniques improve stability, output variation, and convergence.
- Network architecture: The network combines a noise vector and one-hot selection unit to generate multiple textures, with VGG features providing texture-loss supervision.The selection unit controls texture choice, while the generator uses up-convolutions and a fixed 19-layer VGG loss network.
- Texture loss: Mean-subtracted Gram matrices address scale discrepancies across textures and improve synthesis quality over the original Gram-matrix loss.Without recentering, losses and gradients vary drastically across textures, biasing learning toward Gram-matrix scale and producing artifacts and color mixing.
- Diversity loss: The diversity loss compares feature representations of outputs generated from different noise vectors for the same selected texture.It operates at VGG conv4_2 features to encourage higher-level variation rather than simple pixel shifts.
- Diversity loss: With diversity loss, outputs remain perceptually similar to the target texture while exhibiting rich visual variation.Without this loss, different noise inputs produce nearly identical results because texture loss alone does not enforce noise-output correlation.
- Incremental training: Incremental training introduces textures sequentially, retains previously introduced textures in later sampling, and then switches to random sampling after all textures are included.The experiments use K = 1000 iterations for each incremental stage.
- Incremental training: In 60-texture synthesis, incremental training achieves better converged texture loss and visual quality than random training.Random training leaves artifacts, poorly preserved geometry, and color mismatches, whereas later textures also converge relatively faster under incremental training.
4. Experimental Results
The experiments evaluate a single network for many textures, showing diverse synthesis, texture interpolation, and extension to multi-style transfer.
- Multi-texture synthesis: The experiments test synthesizing many textures with one network, including a 300-texture model trained on DTD images.Figure 8 presents 20 example textures from the 300-texture network.
- Diversity: Different noise vectors produce diverse results for each texture, while the diversity loss supports more natural variation than TextureNet [32].The paper attributes TextureNet's limited diversity to the absence of explicit diversity constraints.
- Interpolation: At test time, interpolating selection-unit bits creates new textures or smooth transitions between existing textures.The reported transitions include Texture 20 to Texture 19 and Texture 19 to Texture 12.
- Interpolation: Compared with, which overlays two textures through interpolated Gram matrices, the proposed method generates new textural effects.The comparison is shown in Figure 11.
- Multi-style transfer: The multi-texture synthesis design extends to multi-style transfer by combining style-specific noise maps with encoded content features.A selection unit controls which style-specific noise map is randomly initialized.
- Multi-style transfer: The multi-style transfer model achieves improved or comparable results against existing methods under similarly adjusted style weights.The comparison appears in Figure 15.
- Multi-style transfer: Style-transfer diversity appears in local visual structures while preserving the global image structure required by content preservation.Figure 16 highlights differences such as the beak and sky.
5. Discussion
The discussion examines selector and embedding mechanisms and presents multi-style transfer results, interpolation, comparisons, and diverse outputs.
- Selector network: The selector network injects guidance at every upsampling scale to help distinguish textures during synthesis.The paper compares training a 60-texture network with and without the selector network.
- Multi-style transfer: The 16-style network demonstrates transferred results for eight example styles, including color-independent transferred results.The content image is shown at the left, transferred results in the middle, and style images at the top.
- Comparison: Figure 15 compares the proposed style-transfer results with existing methods.The accompanying discussion reports improved or comparable results after adjusting style weights.
- Multi-style transfer: The 16-style network produces diverse transferred results while preserving content-driven global structure and varying local structures.The paper specifically notes differences in the beak and sky.
- Embedding: Embedding reduces the 60-D and 300-D one-hot selection units to 32-D and 128-D representations while retaining texture discrimination.The reported synthesized results indicate that the hand-crafted one-hot representation is redundant.
6. Conclusion
The proposed multi-texture feed-forward network synthesizes diverse results in one model, with incremental learning supporting effective training. Experiments report comparable results to single-texture networks while reducing model size, and extend the model to multi-style transfer.
- The framework targets diverse synthesis of multiple textures within one single network.
- Figure 17 compares loss curves for the framework without and with the selector network.
- Experimental results show comparable results to existing single-texture networks while greatly reducing model size.
- The work extends the multi-texture synthesis model to multi-style transfer for image stylization.