Source-linked AI summary
Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
Chuan Li, Michael Wand
TL;DR
The paper addresses data-driven image synthesis that must preserve style while changing image structure, a challenge for both classic MRFs and dCNN inversion. It combines a dCNN with an MRF regularizer over higher-level features to control abstract layout and local feature consistency. Across photographic and artistic transfers, the method improves plausibility and mesostructure, while remaining restricted when content and style structures differ strongly.
Problem
Data-driven synthesis seeks variants that resemble an example while differing in structure, but classic MRFs struggle to generalize sparse local patches and dCNN inversion can produce implausible feature layouts.
Method
The method combines dCNN feature representations with an MRF regularizer on higher network layers, using an energy term and EM optimization to constrain local feature layouts.
Results
The combined method generalizes among image patches beyond classic MRFs and produces more plausible mesostructures, including plausible photorealistic style transfers.
Takeaways & Limitations
Combining discriminative neural representations with classical MRF synthesis enables style transfer that preserves better mesostructure and supports photorealistic results with some plausibility.
Takeaways & Limitations
The method is restricted to cases where style and content contain similarly shaped elements; strong perspective, size, or shape changes can produce artifacts, while rigidity can hurt pure artistic styles.
Abstract
from arXiv · showhide
This paper studies a combination of generative Markov random field (MRF) models and discriminatively trained deep convolutional neural networks (dCNNs) for synthesizing 2D images. The generative MRF acts on higher-levels of a dCNN feature pyramid, controling the image layout at an abstract level. We apply the method to both photographic and non-photo-realistic (artwork) synthesis tasks. The MRF regularizer prevents over-excitation artifacts and reduces implausible feature mixtures common to previous dCNN inversion approaches, permitting synthezing photographic content with increased visual plausibility. Unlike standard MRF-based texture synthesis, the combined system can both match and adapt local features with considerable variability, yielding results far out of reach of classic generative MRF methods.
1. Introduction
The paper combines dCNN feature representations with an MRF prior for data-driven image synthesis, controlling abstract layout while preserving adaptable local style patterns. It targets both photorealistic and non-photorealistic transfers and reports improved plausibility over prior approaches.
- Problem: Data-driven synthesis creates a structurally different variant from example images under user-provided constraints, separating style building blocks from content layout.The method supports constraints ranging from changed dimensions to detailed layout specifications.
- Limitations of prior methods: MRF texture synthesis struggles to learn and generalize plausible high-dimensional image patches from sparse examples, producing mismatched pieces that require stitching or blending.The missing capability is interpolation and adaptation beyond simple patch sampling and nearest-neighbor estimation.
- Method: The method applies an MRF regularizer to higher dCNN layers, where feature activations encode increasingly invariant and approximately spatially coherent arrangements of abstract image content.The implementation uses relu3_1 and relu4_1 of the 19-layer VGG network to prescribe plausible local object layouts.
- Method: The additional MRF prior is implemented as an energy term and optimized with Kwatra et al.'s EM algorithm within a variational framework.The M-step uses linear blending, which matches the paper's claim that higher-level neural encodings are more perceptually linear.
- Results: The combined method generalizes among image patches beyond classic MRFs, automatically matches semantically related portions, and improves local plausibility by reducing hallucinatory artifacts.The paper reports more plausible mesostructure and says photorealistic photograph synthesis becomes plausible where previous methods could not achieve it.
- Implementation: Using 3 × 3 patches at relu4_1 produced the best synthesis results in practice, while enabling overlapping MRFs and increased synthesis detail.The paper notes similar matching performance to a slightly higher layer but better practical synthesis with relu4_1 patches.
2. Related Work
Related work spans neural image reconstruction and synthesis, generative adversarial networks, and classic MRF-based texture synthesis. These approaches differ in how they recover image structure, adapt examples, and scale generation.
- Neural synthesis: Deconvolutional and inversion methods reconstruct images by back-projecting or optimizing intermediate neural representations, while Gatys et al.'s approach also supports unguided synthesis.The cited neural methods exploit intermediate-layer encodings, and Gatys et al. reported results exceeding traditional parametric texture synthesis.
- Generative adversarial networks: GANs generate images through adversarial optimization, but cited work reports limited output quality, weak structure, expensive large-image training, or restricted domains.Extensions improve quality or condition generation, including a Laplacian pyramid and a face-focused conditional model.
- MRF-based synthesis: Classic MRF synthesis models local patch statistics, but struggles both to adapt patches beyond stitching or blending and to capture complex global layouts.The resulting local plausibility can coexist with an unstructured global arrangement described as a texture soup.
3. Model
The model combines neural feature matching, content guidance, an MRF style prior, and smoothness regularization in an energy minimized over the synthesized image.
- The synthesized image matches the content image in high-level feature space while matching local neural patches from the style image.The content term uses squared feature-map distance, while the MRF prior maintains style through local patches.
- The energy also includes a content loss and a smoothness prior that penalizes squared image gradients to encourage smooth reconstructions.The content loss constrains the synthesized image toward the guidance image, while the smoothness term addresses low-level information discarded by discriminative training.
- The MRF loss extracts k × k × C neural patches and matches each synthesized patch to its best style-image counterpart using normalized cross-correlation.The matched patches are then used in the style energy, whose Euclidean distance is minimized during synthesis.
- The objective is minimized by back-propagation with Limited-memory BFGS, and neural-level MRF optimization is required for comparable synthesis quality.The method uses neural patches rather than pixel patches in the texture optimization process.
- The content and natural-image-prior weights control guidance and smoothness, with α1 set to zero for non-guided synthesis and α2 fixed to 0.001.For style transfer, the default content weight is α1 = 1 and can be tuned to interpolate between content and style.
4. Analysis
The analysis finds that middle-level neural features improve patch matching and blending, while the MRF prior improves local consistency by reducing artifacts and preserving meso-scale style features.
- Combining MRF priors with dCNNs significantly improves synthesis quality by addressing implausible feature mixtures and inconsistent patch placements.The analysis identifies offsets between patches found at relu3_1 and relu4_1, along with blending artifacts such as distorted faces and ghosting eyes.
- 4.1. Neural Matching: Middle-level VGG features at relu3_1 and relu4_1 provide better patch matches than pixels, lower layers, or the increasingly invariant relu5_1.Lower layers remain sensitive to appearance variation, whereas the middle layers better capture object-part structure.
- 4.2. Neural Blending: Neural blending at middle layers often produces more meaningful results than pixel averaging, which can create strong ghost artifacts.The comparison blends two semantically related, structurally similar patches that differ substantially in pixel values.
- 4.3. MRF Prior: The MRF prior improves local consistency relative to Gram-matrix style constraints, producing patches visually consistent with the style image instead of distortions and smears.This improvement reduces flexibility in local adaptation in favor of reproducing meso-scale features more faithfully.
5. Implementation Details
The implementation uses a pretrained 19-layer VGG network, multi-resolution optimization, dense neural patch matching, and transformed style-image copies to handle variation.
- Synthesis starts from random noise and iteratively minimizes the objective using back-propagation with relu3_1 and relu4_1 MRF layers and relu4_2 content guidance.
- The method uses densely sampled 3 × 3 patches with stride one, implementing patch matching as an additional convolutional layer for fast computation.Style patches act as filters, and the best match is the filter producing the maximum response.
- A two-scale-factor image pyramid performs 200 iterations at each resolution, initializing each level by bilinear upsampling the previous output.Optimization stops when the synthesized image’s longest dimension is below 64 pixels.
- To reduce perspective and scale differences, the method samples style patches from seven scales and five rotations per scale.Rotational copies are used only for deformable objects such as faces because additional patches increase computational cost.
6. Results
The evaluation compares the method with Gatys et al. across artistic and photorealistic style transfer, using examples and an informal user study rather than a formal study. The method often improves style fidelity and local plausibility, but remains sensitive to content–style compatibility and can lose sharpness or structure.
- The evaluation uses typical successful and failure cases, comparisons with Gatys et al., and an informal user study because input-data restrictions make a formal study less meaningful.The authors refer readers to supplementary material for additional un-guided synthesis results.
- Artistic synthesis: In artistic synthesis, the method produces more plausible facial features and better preserves characteristic artwork shapes than Gatys et al. in the shown examples.The comparison includes Picasso and Kandinsky style transfers, where the baseline introduces unnecessary details, unnatural eyes, or blends with the content exemplar.
- Photorealistic synthesis: In photorealistic synthesis, the MRF constraint reduces smears and missing photorealistic details compared with Gatys et al.The examples transfer a vintage-car style to two modern vehicles.
- User study: The informal user study found that Gatys et al. usually preserves content better, whereas the proposed method produces more accurate styles.This establishes a content–style trade-off rather than a uniformly better result.
- Failure cases: The method performs best when content and style patches match, but can introduce artifacts or deviate from content when MRFs do not fit.When no car match is found, it replaces the car with texture synthesis; Gatys et al. retains the car but renders artifacts.
- Limitations: The method is restricted to input data that can be reassembled by its MRFs, making strong perspective or structural differences unsuitable.The authors also report structural artifacts for subjects with strict symmetry, such as architecture.
- Limitations: Although photorealistic synthesis improves, results remain less sharp than the original photos because the network loses non-discriminative image details.The authors suggest retraining the dCNN or combining it with stitching-based texture synthesis for pixel-level photorealism.
7. Conclusions
The paper combines deep convolutional neural networks with classical MRF texture synthesis and finds that the resulting method often preserves better mesostructures. Its main limitation is that stricter mesostructure control can fail when style and content elements differ substantially in perspective, size, or shape.
- The paper’s key insight is combining the discriminative power of deep neural networks with classical MRF-based texture synthesis.
- The method often preserves better mesostructures and permits plausible transfer of photo-realistic styles.The paper compares successful and failure cases against a state-of-the-art neural style-transfer method.
- The MRF prior is advantageous when style and content images contain similarly shaped elements without strong perspective, size, or shape changes.These conditions are covered by the invariance of the high-level neural encoding.
- For pure artistic styles, the method’s increased rigidity can be a disadvantage.
- Future work includes resolving incompatibility between structure guidance and MRFs, transferring middle-level style across large datasets, and generating pixel-level photorealistic images.