Source-linked AI summary
Preserving Color in Neural Artistic Style Transfer
Leon A. Gatys, Matthias Bethge, Aaron Hertzmann, Eli Shechtman
TL;DR
Neural artistic style transfer can undesirably copy the style image’s colors into the content scene. The paper introduces two simple linear color-preservation methods and finds that both produce perceptually interesting results, while each has distinct trade-offs.
Problem
Neural artistic style transfer transfers the style image’s colors, which can undesirably alter the content scene’s appearance.
Method
The paper compares color histogram matching and luminance-only transfer as simple linear methods for preserving the content image’s colors during style transfer.
Results
Both color-preservation methods produce perceptually interesting results, but they have different advantages and disadvantages.
Takeaways & Limitations
Color histogram matching works reasonably well and performs better when applied before style transfer, while luminance-only transfer preserves content colors perfectly.
Takeaways & Limitations
Color matching may leave output colors mismatched and replicate content structures from the style scene, whereas luminance-only transfer loses dependencies between luminance and color channels.
Abstract
from arXiv · showhide
This note presents an extension to the neural artistic style transfer algorithm (Gatys et al.). The original algorithm transforms an image to have the style of another given image. For example, a photograph can be transformed to have the style of a famous painting. Here we address a potential shortcoming of the original method: the algorithm transfers the colors of the original painting, which can alter the appearance of the scene in undesirable ways. We describe simple linear methods for transferring style while preserving colors.
1 Introduction
Neural artistic style transfer combines a content image with a style image by matching CNN feature statistics, reproducing painterly structures such as brushstrokes and geometric shapes. This note addresses the undesirable transfer of the style image’s colors by proposing color-preserving alternatives.
- 1 Introduction: Neural artistic style transfer produces a new image depicting the content image’s objects in the style of a separate style image.The method matches feature-response statistics in a convolutional neural network trained for object recognition.
- 1 Introduction: The original method reproduces stylistic structures such as brushstrokes and geometric shapes, but its transformed output copies the painting’s color scheme.Figure 2 contrasts the original transfer with color-preserving variants.
- 1 Introduction: Two simple approaches preserve the source image’s colors while transferring style: color histogram matching and luminance-only transfer.The methods are compared for their advantages and disadvantages.
2 Approach #1: Color histogram matching
The approach uses linear color transformations to match the style image’s color statistics to the content image before neural style transfer. It compares Cholesky and Image Analogies transfers and finds better results when color matching precedes synthesis.
- Pipeline: The method transforms the style image to match the content image’s colors, then uses the transformed image as input while leaving neural style transfer otherwise unchanged.This preserves the original style-transfer pipeline apart from replacing S with the transformed style image S′.
- Linear color transformation: The transformation matches the style image’s mean and covariance of RGB values to those of the content image.An affine pixel transform uses a 3 × 3 matrix A and a 3-vector b, with constraints µS′ = µC and ΣS′ = ΣC.
- Variants: The approach considers Cholesky decomposition and Image Analogies color matching as two variants among the family of valid transformations.The Cholesky variant uses Σ = LL^T, while the Image Analogies variant is described using covariance eigendecomposition and a matrix square root.
- Comparison: Image Analogies color transfer generally looks better than Cholesky transfer and avoids Cholesky’s dependence on channel ordering.Results from the Monge-Kantorovitch linear transform were essentially indistinguishable from Image Analogies.
- Overall result: Color matching works reasonably well with neural style transfer, despite producing poor synthesis results in Image Analogies.The comparison highlights that the color-matching procedure behaves differently across these style-transfer settings.
- Color matching placement: Matching the color distribution before style transfer leads to better results than matching the output afterward.The authors attribute this to reduced competition between reconstructing content and matching style texture.
3 Approach #2: Luminance-only transfer
The luminance-only variant applies neural style transfer to luminance while retaining the content image’s color channels. When luminance distributions differ substantially, matching them before synthesis can help.
- Luminance-only transfer: Style transfer is performed only on luminance, while the content image’s I and Q color channels are retained for the final output.The style and content luminance channels produce output luminance LT, which is combined with the content colors in YIQ space.
- Luminance-only transfer: Matching the style luminance histogram to the content luminance histogram before transfer can help when their distributions differ substantially.The linear adjustment matches the second-order statistics through the images’ mean luminances and standard deviations.
4 Comparison and discussion
The paper compares linear color transfer before style synthesis with luminance-only transfer, finding complementary trade-offs in color fidelity and luminance–color structure. The discussion also notes a possible optimization-size advantage for luminance-only synthesis and identifies directions for future work.
- Methods: The two proposed methods are linear color transfer onto the style image before synthesis and style transfer performed only in the luminance channel.Both methods aim to preserve the content image’s colors during neural style transfer.
- Comparison: Both methods produce perceptually interesting results but differ in their advantages and disadvantages.The comparison focuses on how well each method preserves colors and relationships between color and luminance structure.
- Trade-offs: Color transfer is limited because the output colors may not exactly match the content image, and it can replicate content structures from the style scene.The paper gives vertical yellow reflection stripes from the van Gogh scene as an example.
- Trade-offs: Color-matching synthesis preserves dependencies between luminance structure and colors, whereas luminance-only synthesis preserves colors perfectly but can misalign colors with brushstrokes.With luminance-only synthesis, a single brushstroke may contain multiple colors; color matching preserves stroke-like paint structure but not exact colors.
- Computational considerations: Luminance-only synthesis has one-third fewer optimization parameters, although the paper finds no clear practical advantage in typical GPU implementations.The reduced dimensionality is presented as a potential rather than established computational benefit.
- Future work: Future work could unify color statistics with CNN activations and investigate more sophisticated color-transfer and adjustment procedures.