Source-linked AI summary
The Contextual Loss for Image Transformation with Non-Aligned Data
Roey Mechrez, Itamar Talmi, Lihi Zelnik-Manor
TL;DR
Many image-transformation tasks lack spatially aligned training targets, making common pixel-comparison losses unsuitable. The paper introduces Contextual Loss, which matches semantically similar features while incorporating image-wide context. It reports state-of-the-art or comparable results across four applications, including unpaired domain translation without GANs.
Problem
Pixel-to-pixel losses assume spatial alignment, while global losses cannot constrain generated-image content in several non-aligned image-transformation tasks.
Method
Contextual Loss represents images as feature collections, matches similar features regardless of position, and incorporates global context through all generated-image features.
Results
The method achieves state-of-the-art or comparable results across four applications without GANs, including domain translation that outperforms CycleGAN.
Takeaways & Limitations
Contextual Loss supports semantic region matching and realistic-looking image generation for non-aligned transformation tasks with a feed-forward architecture.
Abstract
from arXiv · showhide
Feed-forward CNNs trained for image transformation problems rely on loss functions that measure the similarity between the generated image and a target image. Most of the common loss functions assume that these images are spatially aligned and compare pixels at corresponding locations. However, for many tasks, aligned training pairs of images will not be available. We present an alternative loss function that does not require alignment, thus providing an effective and simple solution for a new space of problems. Our loss is based on both context and semantics -- it compares regions with similar semantic meaning, while considering the context of the entire image. Hence, for example, when transferring the style of one face to another, it will translate eyes-to-eyes and mouth-to-mouth. Our code can be found at https://www.github.com/roimehrez/contextualLoss
1 Introduction
Many image-transformation tasks lack spatially aligned targets, limiting pixel-based losses and leaving global losses unable to constrain image content. The Contextual Loss compares semantically corresponding features while incorporating whole-image context, enabling non-aligned transformation results without GANs.
- 1 Introduction: Pixel-to-pixel losses assume spatial alignment, which is unavailable in semantic style transfer, animation, puppet control, and unpaired domain translation.Global Gram loss can compare non-aligned images but cannot constrain the generated image’s content.
- 1 Introduction: The Contextual Loss matches features across images while ignoring their spatial positions and considering all generated-image features as global context.It is defined over feature collections extracted from images, allowing semantic rather than coordinate-based comparison.
- 1 Introduction: Feature matching lets generated images deform spatially relative to targets while preserving semantic correspondences such as eyes-to-eyes and mouth-to-mouth.The approach is less global than Gram loss because it compares features, and therefore regions, based on semantics.
- 1 Introduction: The Contextual Loss tends to maintain the target image’s appearance, enabling realistic-looking generation without GANs.The paper contrasts this with GAN training, which can be difficult to fine-tune.
- 1 Introduction: Across four applications, the method achieves state-of-the-art or comparable results without GANs, including unpaired domain translation that outperforms CycleGAN.It also improves puppet control and single-image animation over prior pixel-to-pixel approaches and enables semantic style transfer without segmentation.
2 Related Work
Prior image-transformation methods commonly rely on paired, aligned data or global objectives that do not preserve content. Related work spans pixel-based translation, cycle-consistent unpaired transfer, global style transfer, and semantic feature matching.
- 2 Related Work: Pix2pix and related feed-forward approaches use pixel-to-pixel losses and require paired, aligned input-target images.These methods include L1 or L2 losses, sometimes combined with adversarial training.
- 2 Related Work: Unpaired domain-transfer methods replace simple feed-forward architectures with cycle-consistent systems; CycleGAN requires four networks.The cycle maps images between domains and back toward their starting point.
- 2 Related Work: Classical CNN style transfer uses perceptual loss to maintain input similarity and Gram loss to capture target style globally.The approach was originally optimized with a gradient-based solver.
- 2 Related Work: Semantic style-transfer methods match features across corresponding regions, using feature replacement and inversion or decoder-based reconstruction.These methods target region-level correspondences such as sky-to-sky and trees-to-trees rather than spreading style globally.
3 Method
The Contextual Loss compares feature sets rather than spatially corresponding pixels, using global image context to match semantically similar regions in non-aligned images. It aggregates contextual feature similarities into a differentiable loss for image transformation networks and is analyzed theoretically and in a toy denoising experiment.
- Motivation: The method targets comparison of non-aligned images, where pixel-to-pixel losses require spatial alignment and global Gram loss ignores pixel locations.The paper motivates a loss for tasks including non-aligned image transformation, while noting that existing alternatives do not address all such cases.
- Contextual similarity: Each image is represented as a set of features, and each target feature is matched to its most similar feature in the other image.The contextual similarity aggregates the matched feature similarities rather than comparing features at corresponding spatial positions.
- Contextual similarity: Feature-level contextual similarity considers whether a feature is substantially closer to one target feature than to all others, making similarity low when no particular match exists.The construction is intended to remain robust to distance scale: features far from all target features receive low contextual similarity.
- Properties: Contextual similarity lies in [0, 1] and equals 1 when an image is compared with itself.The paper also relates binarized contextual similarities to nearest-neighbor matching and template matching.
- Training loss: The contextual loss applies a negative logarithm to contextual similarity computed from VGG19 feature maps and trains a generator using losses against target and, when needed, source images.The paper uses the resulting loss in multiple image transformation applications, with the perceptual-network layer selected per application.
- Analysis and experiment: In a 1D Gaussian analysis, contextual similarity is maximized for matching distributions and falls rapidly as their distributions move apart.A toy non-aligned denoising experiment reports that L1 produces a drastically blurred image, whereas contextual-loss optimization removes noise without ruining image details.
4 Applications
Across four image-transformation applications, the Contextual loss is evaluated by changing the loss functions while using established architectures, enabling semantic transfer and non-aligned training. The reported results include sharper animation, fewer artifacts, effective semantic style transfer, and successful unpaired domain translation without GANs.
- Experimental setup: The experiments use state-of-the-art architectures and modify only the loss functions, with additional comparisons to recent alternatives.The application settings distinguish paired or unpaired data and semi-aligned or non-aligned data.
- Robustness to Misalignments: Optimizing a noisy image with Contextual loss removes noise without ruining image details, whereas L1 produces a drastically blurred result on non-aligned targets.The figure attributes this behavior to robustness against misalignments and spatial deformations.
- Semantic Style Transfer: In semantic style transfer, Contextual loss transfers style between semantically corresponding regions rather than globally across the image.The method is used for both style and content terms, allowing positional deformations while preserving similarity to the source.
- Semantic Style Transfer: Compared with Gatys et al. and CNNMRF, the method produces semantic regional transfer without the global spreading or severe artifacts reported for the alternatives.Examples include eyes-to-eyes, hair-to-hair, and other corresponding facial regions.
- Single Image Animation: For single-image animation, Contextual loss maintains the source animation layout and target appearance despite generated images being inherently non-aligned with the target.The experiment uses the CRN architecture and trains on approximately 1,000 input frames for 10 epochs.
- Single Image Animation: In single-image animation, the Contextual-loss animations are more faithful to target appearance and input motion than the baseline, with the loss function as the only difference.The baseline uses the same CRN architecture with a Perceptual-plus-Gram objective.
- Puppet Control: In puppet animation, the Contextual-loss model is sharper, has significantly fewer artifacts, and captures driver poses without GANs.The comparison includes the same CRN architecture with L1, Pix2pix with adversarial training, and CycleGAN.
- Unpaired Domain Transfer: For unpaired domain translation, the method reports successful results compared with CycleGAN using one feed-forward generator instead of four networks and adversarial training.The approach trains on non-aligned random pairs because Contextual loss does not require aligned data.
5 Conclusions
The paper concludes that the Contextual loss handles non-aligned training data and achieves state-of-the-art or comparable results across four applications.
- The proposed loss function naturally handles image-generation tasks with non-aligned training data.
- Across four applications, the method achieves state-of-the-art or comparable results on all tasks.
- The follow-up work applies Contextual loss to super-resolution and surface normal estimation and connects it theoretically to KL-divergence.
- The supplementary material reports limitations, ablations, and variations of the proposed loss.