Source-linked AI summary
Visual Attribute Transfer through Deep Image Analogy
Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, Sing Bing Kang
TL;DR
Visual attribute transfer lacks a general solution for images that look very different but share semantic structure, especially when dense correspondences are needed. The paper introduces Deep Image Analogy, which combines image analogies with CNN features and coarse-to-fine nearest-neighbor search. It demonstrates transfers across style, texture, color, sketch or painting to photo, and time-lapse cases, while identifying boundaries involving semantic representation, geometric variation, and geometry style transfer.
Problem
Existing methods do not reliably provide dense correspondences for images from different domains with extreme visual variation despite semantic similarity.
Method
Deep Image Analogy adapts image analogy to CNN feature space and uses feature-domain PatchMatch with coarse-to-fine processing to guide semantic dense matching.
Results
The method is effective across style/texture transfer, color/style swap, sketch/painting-to-photo, and time-lapse applications, including cases with substantial appearance variance.
Takeaways & Limitations
Semantic-based dense correspondences support more structure-preserving visual attribute transfer across semantically related images with different appearances.
Takeaways & Limitations
The method is less successful for style-to-photo transfer and does not handle semantically unrelated images, large scale or viewpoint changes, or geometry style transfer.
Abstract
from arXiv · showhide
We propose a new technique for visual attribute transfer across images that may have very different appearance but have perceptually similar semantic structure. By visual attribute transfer, we mean transfer of visual information (such as color, tone, texture, and style) from one image to another. For example, one image could be that of a painting or a sketch while the other is a photo of a real scene, and both depict the same type of scene. Our technique finds semantically-meaningful dense correspondences between two input images. To accomplish this, it adapts the notion of "image analogy" with features extracted from a Deep Convolutional Neutral Network for matching; we call our technique Deep Image Analogy. A coarse-to-fine strategy is used to compute the nearest-neighbor field for generating the results. We validate the effectiveness of our proposed method in a variety of cases, including style/texture transfer, color/style swap, sketch/painting to photo, and time lapse.
1 Introduction
The paper targets visual attribute transfer between images that differ greatly in appearance but share semantic structure. Deep Image Analogy uses CNN features and image-analogy ideas to establish dense correspondences, supporting several transfer applications while requiring related content and limited geometric variation.
- Visual attribute transfer copies color, texture, or style from one image to another.
- The method addresses pairs that differ in appearance but contain semantically similar high-level visual content.Examples include nature scenes, headshots, and indoor scenes with corresponding object classes.
- Existing low-level and sparse-feature matching methods struggle with extreme visual variation or fail to produce dense cross-domain correspondences.
- Deep Image Analogy constructs a CNN feature space for image analogies because deep features better represent semantic structures.
- Pre-trained CNN representations progress from low-level details to high-level semantics while approximately preserving spatial correspondence, enabling coarse-to-fine nearest-neighbor search.
- The technique extends PatchMatch and reconstruction into the feature domain and applies them to style, texture, color, sketch-to-photo, painting-to-photo, and time-lapse transfers.It generates two results rather than the typical one-way transfer result, but is designed for similar content composition and not large geometric variation.
2 Related Work
Related work spans specialized visual-attribute transfer, image analogy, dense correspondence, and neural style-transfer methods. The paper positions its approach as more general and structure-preserving for visually different but semantically related images.
- Prior color, texture, and style-transfer methods commonly target particular visual attributes rather than handling them generally.
- Image analogy provides supervised dense mappings between registered image pairs, while this work uses only an original image A and example image B′ as weak supervision.
- Traditional dense correspondence methods rely on brightness consistency, locality, smoothness, or low-level features and can fail under large displacements or extreme appearance changes.
- PatchMatch efficiently computes dense nearest-neighbor fields, but its conventional low-level formulation does not reliably match visually different yet semantically similar images.
- CNN-based neural style-transfer methods can produce impressive stylization, but do not guarantee structure preservation and may distribute texture randomly.
- The proposed technique uses semantic-based dense correspondences to transfer style more structure-preservingly across multiple visual attributes.
3 Motivation
The method formulates transfer between appearance-different but semantically similar images as a bidirectional image analogy with latent images and mappings. CNN reconstruction supplies fused content and appearance features, while deep PatchMatch estimates dense correspondences coarse-to-fine.
- Given A and B′, the goal is to find a mapping between them for visual attribute transfer despite different visual attributes.
- The analogy A : A′ :: B : B′ introduces latent images A′ and B, enforcing spatial correspondence within pairs and appearance similarity across pairs.
- The difficult mapping A → B′ is decomposed into an in-place mapping A → A′ followed by a similar-appearance mapping A′ → B′.
- Forward and reverse mappings are optimized with bidirectional consistency constraints so that composing them returns the original pixel location.
- CNN reconstruction builds A′ by combining content structure from A with corresponding visual details from B′ through weighted feature combinations across layers.
- Deep PatchMatch computes forward and reverse nearest-neighbor fields in feature space, using semantic representations compatible with CNN-based latent-image reconstruction.
4 Deep Image Analogy
Deep Image Analogy transfers visual attributes by combining image analogy with deep CNN features and progressively refined dense correspondences. Its pipeline reconstructs latent images while moving from coarse semantic structure to fine visual detail.
- System overview: Deep Image Analogy computes forward and reverse nearest-neighbor fields between CNN feature maps of the input and latent image pairs.The system uses these correspondences to reconstruct latent images and repeats the process across layers.
- Nearest-neighbor field search: Bidirectional matching discourages ambiguous mappings, while single-direction matching is prone to misalignment.The comparison is illustrated in Figure 5(b)(c).
- Latent image reconstruction: Latent feature reconstruction fuses content from the source with sampled visual details using weighted feature-map combinations.The reconstruction pipeline includes feature-map warping, deconvolution for the next layer, and fusion.
- Latent image reconstruction: Directly upscaling mappings fails to preserve structures because adjacent CNN layers do not maintain exact spatial correspondence through ReLU and max-pooling.The proposed alternative warps features at the previous layer before deconvolving them for the current layer.
5 Evaluations on Matching
The evaluation compares dense correspondence methods across increasingly different visual conditions, from similar scenes to semantically related images with substantially different styles. Deep Image Analogy is reported to outperform baselines overall on PASCAL 3D+ and to produce acceptable results for highly varied photo–painting pairs.
- Evaluation Setup: The evaluation covers same-scene pairs, appearance-changing pairs, and semantically related scenes with vastly different styles.These categories include video-like motion, changes in pose, illumination, and season, and photo–painting pairs.
- Categories I–II: On Middlebury, the method performs better than flow methods in reconstruction error, although PatchMatch produces slightly better-looking reconstructions.The authors attribute the visual difference to weaker consideration of color similarity in their method.
- Category III: On PASCAL 3D+, correspondence accuracy is measured with PCK over 20 randomly selected image pairs per category, and the method outperforms three baselines overall.The dataset supplies annotated sparse correspondences for semantically related objects with remarkably different appearances.
- Categories I–II: For category (II), the method is visually comparable to NRDC or RFM in high-confidence regions and provides a better guess elsewhere.PatchMatch can fail across colors, while SIFT Flow and DeepFlow have mismatches where sparse feature points are absent; RFM and NRDC have complementary limitations.
- Category III: For photo–painting pairs in category (III), existing matching methods produce noticeable artifacts, whereas Deep Image Analogy automatically generates visually acceptable results.Halfway morphing can address the problem but may fail without user-provided high-level matching.
6 Applications
The applications span photo-to-style, style-to-style, style-to-photo, and photo-to-photo transfer. Across these settings, the method emphasizes local or pixel-level structural correspondence, while its success depends on image relatedness and matching quality.
- Applications: Deep Image Analogy is evaluated on four transfer tasks: photo to style, style to style, style to photo, and photo to photo.Comparisons use author-released implementations, default settings, or submitted images to competing services and apps.
- 6.1 Photo to Style: For photo-to-style transfer, the method provides higher-quality content-specific stylization that better preserves structures than global-statistics-based approaches.The authors note that competing methods can place style textures in semantically inappropriate regions, such as flowers on a face.
- 6.1 Photo to Style: The method directly samples reference patches using an estimated NNF, trading off faithful content capture at times for better style reproduction.This is contrasted with methods that optimize pixel color using combined content and style losses.
- 6.2 Style to Style: For content-related artworks with vastly different styles, the method can swap styles, including transforming a Pharaoh statue into Peking Opera facial makeup.The paper presents this as a new and effective application.
- 6.3 Style to Photo: Style-to-photo transfer is generally less successful than photo-to-style, but produces more plausible results when the sketch or painting and photo are very related.The task is harder because artworks may lack detail and photorealism is highly sensitive to slight structural misalignments.
- 6.3 Style to Photo: Compared with CG2Real, the pixel-to-pixel mapping better maintains input-CG structures while achieving visually comparable results using one reference photo instead of multiple photos.CG2Real performs region-to-region transfer after retrieving and co-segmenting a small set of photos.
- 6.4 Photo to Photo: For photo-to-photo color or tone transfer, WLS refinement preserves transferred colors while discarding transferred details and filling in source-image details.Without refinement, reference details can distort fine-grained source structures such as eyes and noses.
- 6.4 Photo to Photo: Local color transfer can improve region-to-region results and can transfer colors across different scenes through semantic correspondences, but mismatches can create inaccurate mappings.The paper gives tree-to-tree and mountain-to-mountain correspondences as examples and notes a saturated sky caused by mismatched regions.
7 Discussion
The method has several scope boundaries: it struggles with missing objects, large scale or viewpoint changes, textureless regions, and geometry style transfer because it prioritizes content preservation.
- Limitations: A pre-trained VGG network has limited capability for representing all objects and fine-grained structures, constraining semantic-level matching.The authors suggest training on a domain-specific dataset as a possible improvement.
- Limitations: When an object appears in one image but not the other, semantic matching is unclear and can produce mismatched regions such as a hat.The top-left failure case in Figure 22 illustrates this problem.
- Limitations: Large differences in scale and viewpoint can prevent the method from building correct correspondences between semantically related scenes.Addressing this would require pre-scaling images or rotation- and scale-aware NNF search, which is difficult in the feature domain.
- Limitations: Textureless regions with very low neural activation, such as backgrounds, may lack usable correspondences.The paper suggests analyzing pixel intensity or explicitly enforcing smoothness in the energy function.
- Limitations: Photo-to-style transfer cannot produce geometry style transfer because the method assumes maximal preservation of content structure.The authors identify relaxing this assumption as future work.
8 Concluding Remarks
The paper presents Deep Image Analogy for transferring visual attributes across semantically related images using dense correspondences in deep feature space. It reports broad applicability across appearance variations and transfer challenges, while also showing failure cases.
- Concluding Remarks: The method transfers visual attributes across semantically related images by adapting image analogy to a deep feature space for dense correspondence finding.
- Concluding Remarks: The approach outperforms previous methods on image pairs with substantial variation in lighting, color, texture, and style.
- Concluding Remarks: The method applies to real-world visual attribute transfer and challenges including content-specific stylization, style-to-photo, and style-to-style transfer.
- Concluding Remarks: Figure 22 presents examples of failure cases that delimit the method's demonstrated scope.