Source-linked AI summary
Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks
Chuan Li, Michael Wand
TL;DR
Deep Markovian texture synthesis produces strong visual results but can require minutes for low-resolution images because generation relies on iterative deconvolution. MGANs precompute a strided feed-forward decoder trained adversarially on Markovian patches, achieving fast synthesis while preserving comparable quality, with applications to texture synthesis and stylization.
Problem
Deep Markovian models synthesize detailed high-resolution textures, but deconvolution-based generation requires costly iterative optimization.
Method
MGANs train a strided convolutional generator and discriminator adversarially, using fixed VGG features to learn Markovian patch mappings and directly decode features to pixels.
Results
Once trained, MGANs preserve coherent texture for complex image content and decode 512×512 images in 40ms, about 500 times faster than Gatys et al.’s method.
Takeaways & Limitations
The precomputed model enables extremely fast texture synthesis and supports applications including artistic style transfer and video stylization.
Takeaways & Limitations
The method works less well for non-texture data and weak-content regions, where semantic understanding or stronger VGG activations are needed.
Abstract
from arXiv · showhide
This paper proposes Markovian Generative Adversarial Networks (MGANs), a method for training generative neural networks for efficient texture synthesis. While deep neural network approaches have recently demonstrated remarkable results in terms of synthesis quality, they still come at considerable computational costs (minutes of run-time for low-res images). Our paper addresses this efficiency issue. Instead of a numerical deconvolution in previous work, we precompute a feed-forward, strided convolutional network that captures the feature statistics of Markovian patches and is able to directly generate outputs of arbitrary dimensions. Such network can directly decode brown noise to realistic texture, or photos to artistic paintings. With adversarial training, we obtain quality comparable to recent neural texture synthesis methods. As no optimization is required any longer at generation time, our run-time performance (0.25M pixel images at 25Hz) surpasses previous neural texture synthesizers by a significant margin (at least 500 times faster). We apply this idea to texture synthesis, style transfer, and video stylization.
1 Introduction
Deep Markovian texture models produce detailed high-resolution results by modeling local patches, but their deconvolution-based generation is prohibitively slow. MGANs replace iterative inversion with a precomputed feed-forward network while preserving quality and supporting arbitrary image dimensions.
- Traditional texture synthesis uses Markov random fields to characterize images through local pixel-patch statistics.
- Deep generative models improve visual quality, but full-image models remain limited to small images and deep Markovian models lack non-trivial global structure without guidance.
- Several minutes are required to synthesize a 512×512 image with higher-level deep Markovian deconvolution, because generation requires iterative back-propagation.
- MGANs precompute network inversion by fitting a strided convolutional network, enabling feed-forward synthesis of arbitrary-sized images without optimization or blending.
- 40ms is sufficient for the GPU implementation to compute a 512×512 image while maintaining quality similar to the original optimization-based approach through adversarial training.
- A strided convolutional network replaces pooling layers with subsampled convolution filters that learn pooling during training.
2 Related Work
Related work spans deep-feature visualization, full-image GANs, and fast perceptual texture decoders. MGANs differ from concurrent fast methods by using Markovian feature-patch statistics rather than Gaussian distributions of individual feature vectors.
- Deconvolutional networks reconstruct pixels from deep neural activations and have been developed for visualizing features and improving reconstruction efficiency.
- Full-image generative methods use specially trained auto-encoders, including GAN variants with pyramid and conditional extensions.
- Ulyanov et al. and Johnson et al. precompute decoders trained with perceptual texture loss to accelerate Gatys et al.’s approach.
- MGANs use Li et al.’s feature-patch statistics instead of Gaussian models of individual feature vectors, providing benefits for faithful texture reproduction.
- The motivation contrasts Gaussian distribution matching with adversarial projection onto complex nonlinear manifolds of contextually related patches.
3 Model
The model combines adversarially trained Markovian patch discrimination with a feed-forward generator to replace slow iterative texture synthesis. MDANs support guided and un-guided synthesis, while MGANs directly decode VGG features and retain comparable visual quality with substantially higher speed.
- MDANs: Adversarial training distinguishes real and synthesized neural patches, replacing explicit patch-dictionary matching in the Markovian synthesis pipeline.The discriminator operates on VGG 19 Relu3_1 patches; optional higher-layer Relu5_1 comparison provides content guidance.
- MGANs: MGANs add a strided convolutional generator that decodes VGG 19 Relu4_1 features directly to pixels, while discriminator and generator are trained simultaneously.The generator extends from fixed-size training inputs to arbitrary-size images and avoids per-image iterative deconvolution.
- MDANs: MDANs iteratively optimize pixels from random noise or a guidance image using texture, optional content, and smoothness losses.Content loss uses mean squared error between VGG feature maps, enabling contextually related guided outputs.
- MGANs: Adversarial training improves style reproduction over pixel-level and neural VAEs, avoiding the over-smoothed outputs produced by pixel Euclidean loss.The comparison uses Picasso-style painting synthesis on training and testing photos.
- MGANs: 25,000 times faster: MGANs produce results comparable to MDANs, although MDANs preserve input content better and generate more natural backgrounds.MGANs are more stylized because training across many images emphasizes frequent features.
- MGANs: MGANs improve progressively during training, learning color and contrast regions before brush-stroke textures, eyes, and reduced ghosting artifacts.The model also generalizes well to testing data during the observed training progression.
4 Experimental Analysis
The experiments examine how discriminator and generator design choices affect synthesis quality, including feature layers, patch sizes, network complexity, and initialization. They also assess learned decoder features and the role of pretrained VGG-19 representations.
- Decoder features: Decoder features are visually similar to, but not faithful reproductions of, example texture patches, yet remain sufficient for synthesizing new images.The features are decoded from one-hot input vectors.
- Parameter influence: Lower VGG-19 layers produce sharper appearances but lose texture form and structure.The study compares relu2_1, relu3_1, and relu4_1 using approximately equal receptive fields.
- Network complexity: Removing a discriminator convolutional layer reduces synthesis quality, while additional layers provide no obvious improvement in these examples.Restoring one convolutional layer produces smoother synthesis.
- Network complexity: Reducing discriminator channels generally worsens results, but 64 and 128 channels show no significant difference.For the simpler ivy texture, the difference between 4 and 64 channels is marginal.
- Network complexity: Increasing generator complexity yields limited improvement, while removing an initial decoder layer or reducing channels causes some quality loss.The observed differences are described as not very significant.
- Initialization: Pretrained discriminator initialization produces good results after 50 iterations, whereas random initialization does not achieve comparable quality as quickly.The pretrained network had previously transferred 50 face images.
- VGG-19: Training the discriminator from pixels to class labels performs significantly worse than using pretrained VGG-19 features.The paper attributes VGG-19’s statistical power to shared feature cascades learned across diverse images.
5 Results
The results show coherent texture transfer, fast feed-forward decoding, and applications to unguided synthesis and video. Compared with prior methods, the approach trades some distributional fidelity for coherence while achieving substantial speed improvements, with memory remaining a practical constraint.
- Texture transfer: Our method transfers textures more coherently than global-statistics methods, while Li et al.’s local patch approach produces the most coherent synthesis.Li et al.’s method generates a 384-by-384 image in 110 seconds because it requires patch matching.
- Speed: 500 times faster than Gatys et al. and 5000 times faster than Li et al., our method runs at the same speed level as Ulyanov et al.These comparisons are reported for the synthesis results discussed in Figure 11.
- Texture transfer: The Gaussian-based method preserves color distributions and textures backgrounds better, whereas ours handles coherent transfer in complex regions more effectively.The paper notes that Gaussian modeling can produce too much or too little texture in complex regions.
- Applications: Unguided synthesis is possible by decoding brown-noise-derived VGG-19 feature maps into plausible textures.The model trained with random ImageNet images demonstrates this generalization ability.
- Applications: The feed-forward method produces video decoding that is relatively more temporally coherent than deconvolutional methods.Figure 14 reports 8Hz decoding for 1080-by-810 video.
- Speed: Our method and Ulyanov et al. decode 512-by-512 images at 25Hz, with runtime scaling linearly with image pixel count.Our method takes 10 ms for 256-by-256, 40 ms for 512-by-512, and 160 ms for 1024-by-1024 images.
- Memory: The generative model uses 70 MB for parameters and approximately 2.5 GB at runtime for a 512-by-512 image.Runtime memory scales linearly with image size, though scanline block processing can reduce usage.
6 Limitation
MGANs have distinct strengths and weaknesses across texture types and content regions. They perform better for deformable or recognizable textures but remain limited for non-texture data, flat regions, and highly stationary backgrounds.
- MGANs work less well with non-texture data, such as transferring facial features between photographs.The authors attribute this limitation to the need for semantic understanding of expression, pose, and gender.
- MGANs may deviate from the example texture’s global color distribution, unlike Gaussian-based models that better preserve its global appearance.The authors note that this deviation can be acceptable when the content image matters more.
- VGG-19 feature invariance makes MGANs weaker on highly stationary backgrounds with weak activations, where statistics-based methods often generate better textures.The method performs better on areas containing recognizable features.
- MGANs often produce outputs more consistent with example textures but show weaknesses in flat regions because they lack iterative optimization.The authors suggest recurrent architectures with state information as a possible improvement.
7 Conclusion
The paper applies adversarial generative networks in a Markovian setting to learn mappings between depictions of the same content. The resulting model supports fast texture and style synthesis, while broader generative image modeling remains future work.
- MGANs learn mappings between different depictions of the same content using adversarial generative networks in a Markovian setting.The model trains from one texture example and randomly selected ImageNet images.
- After training, MGANs decode brown noise into realistic textures and photos into artworks while preserving coherent texture for complex image content.The authors report advantages over statistics-based methods for this preservation.
- Training takes about an hour per example, after which synthesis is extremely fast and provides attractive style-transfer invariance.
- Future work would extend the framework with coarse-scale structure models and larger datasets to improve generalization across broader image classes.The stated long-term goal is a directly decoding generative model for large classes of real-world images.