Source-linked AI summary
Style Transfer by Relaxed Optimal Transport and Self-Similarity
Nicholas Kolkin, Jason Salavon, Greg Shakhnarovich
TL;DR
Style transfer requires separating style from content despite the difficulty of defining either concept and the lack of established ground truth for evaluation. STROTSS models style with relaxed optimal transport over deep features and content with self-similarity, while supporting point- and region-level guidance. Human evaluations indicate higher stylization quality than prior work at any given content-preservation level.
Problem
Style transfer lacks clear semantic definitions of style and content, and its outputs are difficult to compare quantitatively because no established ground truth exists.
Method
STROTSS defines style using an efficient relaxed Earth Movers Distance over deep features, content using self-similarity, and extends optimization with point-to-point and region-to-region guidance.
Results
For any given content score, STROTSS provides a higher style score than prior work in large-scale human evaluations.
Takeaways & Limitations
The proposed formulations compare favorably to prior work in both stylization quality and content preservation, while user guidance supports controlled style transfer.
Takeaways & Limitations
The method uses a simplified EMD formulation in which all features have equal mass rather than allowing general non-uniform mass distributions.
Abstract
from arXiv · showhide
Style transfer algorithms strive to render the content of one image using the style of another. We propose Style Transfer by Relaxed Optimal Transport and Self-Similarity (STROTSS), a new optimization-based style transfer algorithm. We extend our method to allow user-specified point-to-point or region-to-region control over visual similarity between the style image and the output. Such guidance can be used to either achieve a particular visual effect or correct errors made by unconstrained style transfer. In order to quantitatively compare our method to prior work, we conduct a large-scale user study designed to assess the style-content tradeoff across settings in style transfer algorithms. Our results indicate that for any desired level of content preservation, our method provides higher quality stylization than prior work. Code is available at https://github.com/nkolkin13/STROTSS
1 Introduction
STROTSS addresses the difficulty of defining style and content by modeling style through feature distributions and content through self-similarity, while adding intuitive spatial guidance. Human evaluations compare stylization and content preservation across algorithms and hyperparameter settings.
- 1 Introduction: STROTSS defines style as a distribution over deep-network features and measures style distance with an efficient Earth Movers Distance approximation.The formulation aims to transport visual attributes from the style image while minimally distorting the content layout and semantics.
- 1 Introduction: Content similarity is based on self-similarity, preserving perceived semantics and spatial layout while allowing substantial pixel-level differences.This definition is less tied to precise pixel values than prior content formulations.
- 1 Introduction: Figure 1 contrasts unconstrained and guided outputs, with region-to-region guidance shown beneath the corresponding content and style images.Images are arranged as content, output, and style.
- 1 Introduction: Users can impose point-to-point or region-to-region constraints to control which style-image features influence corresponding output locations.Examples include stylizing content hair with style-image clouds or matching eyes across images.
- 1 Introduction: Human evaluations use 662 Amazon Mechanical Turk workers to measure content preservation and stylization quality separately across multiple hyperparameter settings.The study evaluates the style-content trade-off rather than relying only on subjective qualitative comparisons.
2 Methods
The method optimizes an image using feature-based style and self-similarity content losses, with relaxed transport for computational efficiency and optional spatial constraints. It combines deep hypercolumn features, color and moment matching, multiscale optimization, and sampled pairwise distances.
- 2 Methods: STROTSS minimizes an objective over the output image using a content loss and multiple style-loss terms with RMSprop.The method takes a content image and style image as inputs and optimizes the output iteratively.
- 2 Methods: Hypercolumns from VGG16 combine low-level edge and color, mid-level texture, and high-level semantic features at each pixel.Features are bilinearly upsampled to the original image dimensions before concatenation.
- 2.2 Style Loss: Relaxed EMD replaces computationally expensive optimal transport by using auxiliary distances that retain only one transport constraint each.Exact EMD optimization costs O(max(m, n)^3), making repeated computation during gradient descent impractical.
- 2.2 Style Loss: The transport cost uses cosine distance between feature vectors, while moment matching addresses artifacts caused by ignoring feature magnitudes.The authors report that Euclidean feature distance produced significantly worse results and that cosine distance can cause over- or under-saturation.
- 2.2 Style Loss: The color-palette loss matches output and style-image colors using relaxed transport with Euclidean distance in a decorrelated color space.This term is weighted because palette shifting conflicts with content preservation.
- 2.3 Content Loss: The content loss preserves normalized pairwise cosine distances between feature vectors at image coordinates rather than directly matching content pixels.This constrains output structure while allowing pixel values to differ substantially from the content image.
- 2 Methods: User guidance modifies the relaxed-transport ground metric so paired output and style regions receive low style loss and mismatched assignments receive infinite cost.Point guidance is augmented with eight automatically generated points forming a uniform 9x9 grid.
- 2.5 Implementation Details: Multiscale optimization upsamples outputs across resolutions, optimizes a Laplacian pyramid, and resamples 1024 style and 1024 content coordinates during RMSprop updates.The authors report that Laplacian-pyramid optimization dramatically speeds convergence.
3 Related Work
Prior style-transfer work varies in how it represents style, performs transfer, enables control, and evaluates outputs without established ground truth.
- Neural Style Transfer represents style with Gram matrices and content with deep feature tensors, using an optimization-based formulation.
- Later style losses match output features to style-image distributions using translated Gram matrices, nearest neighbors, histograms, or feature pairings.
- Real-time feed-forward methods replace optimization with trained networks, supporting arbitrary styles at test time but generally sacrificing output quality for speed.
- Prior control methods preserve global properties or impose spatial correspondences through matched masks between style and content images.
- Because style transfer lacks established ground truth, evaluations commonly rely on subjective comparisons, aesthetic rankings, classifiers, or expert judgments.
- The study evaluates content-style trade-offs across hyperparameter settings, aiming to provide a more complete quantitative comparison than prior efforts.
4 Experiments
Experiments evaluate STROTSS across style-transfer regimes, human judgments, ablations, approximation quality, and runtime. The results show favorable style-content tradeoffs, complementary loss contributions, and a simple but effective Relaxed EMD approximation.
- Large-Scale Human Evaluation: The evaluation covers paired, unpaired, and texture regimes, with 30 style/content pairings per regime.Paired inputs depict similar things, unpaired inputs depict different things, and texture inputs combine a face photograph with a homogeneous texture.
- Large-Scale Human Evaluation: Workers compare outputs for content preservation and stylization quality against the same inputs, enabling separate scores along both axes.The study uses AMT comparisons between algorithm outputs and either the content or style input.
- Ablation Study: Combining Relaxed EMD terms produces higher-quality outputs than either term alone, while moment matching and palette matching address saturation and palette issues.The full combination of ℓr, ℓm, and ℓp improves limitations observed in individual style-loss components.
- Relaxed EMD Approximation Quality: Across 900 unpaired image pairs, REMD has mean ratio 0.60 and standard deviation 0.04 relative to exact EMD.REMD is a lower bound that is simple to compute, empirically easy to optimize, and effective for style transfer.
- Runtime: Timing comparisons use CPU and GPU measurements and mark methods that run out of memory or require projected results.The timing table compares methods across square image sizes and records unavailable or out-of-memory cases.
5 Conclusion and Future Work
The paper concludes that STROTSS's formalizations of style and content compare favorably with prior work, while identifying more accurate approximations and feed-forward training as future directions.
- Conclusion: STROTSS compares favorably to prior work in both stylization quality and content preservation.The conclusion attributes this result to novel formalizations of style and content.
- Conclusion: The ablation study indicates that style-similarity losses measuring feature-distribution distances more accurately lead to better style transfer.The conclusion identifies distribution-distance accuracy as a factor associated with improved transfer quality.
- Future Work: Future work includes more accurate Earth Movers Distance approximations and faster feed-forward methods trained with the proposed objective.The current approximation is described as simple but effective, while speed improvements are left for future work.
6 Appendix
The appendix provides qualitative guidance comparisons and an extended ablation study examining optimization choices and style-loss components.
- Spatial Guidance: A qualitative comparison evaluates STROTSS's spatial guidance against a prior method using identical content, style, and guidance masks.The comparison uses an example from the prior method's paper.
- Extended Ablation Study: The extended ablation study varies pixel versus Laplacian-pyramid optimization, multiscale versus single-scale updates, and the Relaxed EMD ground metric.The ℓ2 ground-metric variant replaces the default cosine distance, while single-scale optimization performs 800 final-resolution updates.