PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Image Blending

Lingzhi Zhang, Tarmily Wen, Jianbo Shi

arXiv:1910.11495v1cs.CV

TL;DR

Image blending must make a source object fit naturally onto a target, but Poisson blending can limit texture adaptation and let target colors seep through the source. The paper combines a differentiable Poisson blending loss with deep style and content losses in a two-stage, training-free method. User studies report that the method was preferred over other methods 80–90% of the time across real-world and painting-style targets.

  • Problem

    Image blending must make a source object look natural on a target, while Poisson blending is described as insufficiently adaptive to target texture and prone to target colors seeping through the source.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    A two-stage algorithm uses a differentiable Poisson-purpose loss with deep style and content losses to refine a blending region from a source image, coarse mask, and target without training data.

    Source: §1 · p. 2, §4.1 · p. 4, §4.2 · p. 5

  • Results

    80–90% of the time, users preferred the method over other methods across real-world and painting-style target images.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The authors conclude that the method could be applied to common image-editing tasks and help users compose artistic works.

    Source: §6 · p. 8

    The method assumes a source image cropped using a coarse mask and aligned with the target, with user-provided offsets used when alignment is needed.

    Source: §4 · p. 4

Abstract

from arXiv · show

Image composition is an important operation to create visual content. Among image composition tasks, image blending aims to seamlessly blend an object from a source image onto a target image with lightly mask adjustment. A popular approach is Poisson image blending, which enforces the gradient domain smoothness in the composite image. However, this approach only considers the boundary pixels of target image, and thus can not adapt to texture of target image. In addition, the colors of the target image often seep through the original source object too much causing a significant loss of content of the source object. We propose a Poisson blending loss that achieves the same purpose of Poisson image blending. In addition, we jointly optimize the proposed Poisson blending loss as well as the style and content loss computed from a deep network, and reconstruct the blending region by iteratively updating the pixels using the L-BFGS solver. In the blending image, we not only smooth out gradient domain of the blending boundary but also add consistent texture into the blending region. User studies show that our method outperforms strong baselines as well as state-of-the-art approaches when placing objects onto both paintings and real-world images.

1. Introduction

Image blending must make a cropped source region look natural against a target despite imprecise boundaries. The paper proposes a two-stage, training-free algorithm that combines a differentiable Poisson-style loss with deep style and content losses.

1. Introduction

  • Image blending must adapt a possibly imprecise crop to its new background while making the boundary seamless.Poisson blending is popular but has a closed-form solution that is difficult to combine with other reconstruction objectives.Source: §1 · p. 1
  • A supervised GAN-based approach requires scarce paired examples, limiting its generalization and application domain.Image harmonization instead assumes a precisely delineated foreground and adjusts its illumination, color, and texture.Source: §1 · p. 2
  • The proposed two-stage algorithm first creates a seamless boundary, then refines the region toward the target's style and texture.It combines a differentiable Poisson-purpose loss with deep content and style losses, using only a source image, coarse mask, and target, without training data.Source: §1 · p. 2

2. Related Work

Prior image-composition approaches address boundary artifacts through blending or appearance adjustment, while deep image-editing work also uses learned priors and feature-based style transfer.

2.1. Image Blending

  • Alpha blending layers images using per-pixel alpha values, but overlapping source and target content can produce a ghost effect.It improves on direct copy-and-paste.Source: §2.1 · p. 2
  • Gradient-domain blending propagates source gradients while using target boundary pixels, aiming for smooth transitions even with imprecise object masks.Poisson image editing is an early method in this family.Source: §2.1 · p. 2
  • GP-GAN combines a Gaussian-Poisson equation with a Wasserstein GAN, but its supervised paired-data requirement makes generalization difficult.The paired examples include a source, target, and ground-truth blended image.Source: §2.1 · p. 2

2.2. Other Image Editing Techniques

  • Image harmonization adjusts a foreground to fit another image's background; methods range from global color statistics to deep networks and painterly transfer.The painterly approach adapts transfer parameters to the painting.Source: §2.2 · pp. 2–3
  • Style transfer matches deep content features and style-feature Gram matrices, while other deep editing tasks use learned priors for upsampling or filling missing pixels.The figure also illustrates Poisson editing's Laplacian consistency goal with a target-boundary constraint.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

3. Background - Poisson Image Editing

Poisson image editing seeks to reduce abrupt boundary changes by reconstructing the blending region to follow source-image gradients while respecting target-image boundary values.

3. Background - Poisson Image Editing

  • Poisson blending reduces visible artifacts by smoothing abrupt intensity changes at the composite boundary.Its goal is smooth transition across the blending boundary.Source: §3 · p. 3
  • The formulation interpolates the blending region using a guidance vector field, taken here from source-image gradients, with target-image boundary constraints.The solution is computed independently for each color channel.Source: §3 · p. 3
  • Discretizing the image yields a quadratic optimization and simultaneous linear equations; the cited early solvers include Gauss-Seidel and V-cycle multigrid.The linear system is described as sparse and symmetric positive-definite.Source: §3 · p. 3

4. Methods

The method reconstructs a source region in two stages: first creating a seamless blend, then refining its style and texture toward the target. A differentiable Poisson-gradient loss is combined with deep-network content and style losses, with histogram and total-variation losses for regularization.

4.4. Two-Stage Algorithm

  • The two-stage algorithm first synthesizes a preliminary blend with Poisson-gradient, style, and content losses, then refines its style toward the target image.The first stage starts from random noise; the second optimizes a copy of the first-stage blend.Source: §4 · p. 3, Figure 3 · p. 4, §4.4 · pp. 5–6

4.1. Poisson Gradient Loss

  • The Poisson-gradient loss converts the boundary and gradient constraints into a differentiable objective that can be combined with other pixel-reconstruction objectives.It constructs the blending image from reconstructed pixels inside the mask and target pixels outside, then compares its Laplacian gradients with source-plus-target gradients.Source: §4.1 · pp. 4–5

4.2. Style and Content Loss

  • Content and style losses use deep-network features to preserve source semantics and align the blend’s texture with the target.In stage two, the refined image is optimized for content similarity to the first-stage blend and style similarity to the target.Source: §4.2 · p. 5

4.3. Regularization Loss

  • Histogram matching and total-variation losses regularize the generated image by stabilizing style transformation and encouraging spatial smoothness.Histogram matching aligns feature-map distributions with the target; total variation removes unwanted details while preserving important information.Source: §4.3 · p. 5

4.4. Two-Stage Algorithm

  • The method optimizes the blending and refined images with L-BFGS for up to 1,000 iterations in both stages; a 512 × 512 image takes about five minutes on one NVIDIA GTX 1080Ti.Source: §4.4 · p. 6

5. Experimental Results

The full method outperforms its baseline variants in ablations and produces strong blending results on paintings and real-world images. In user studies, it was preferred over other methods 80–90% of the time.

Ablation study

  • The full model outperforms baseline variants; omitting blending loss leaves boundary artifacts, omitting content loss wipes out visual content, and omitting style loss yields inconsistent illumination and texture.The two-stage algorithm adds more style and texture than the single-stage baseline.Source: §5 · p. 7

Painting comparisons

  • On paintings, the method produces the most consistent style and texture while maintaining source content, unlike Style Transfer + Poisson Blending, which loses some source content.Poisson Blending + Style Transfer yields more consistent style and texture than Poisson Blending but produces unpleasant color illumination.Source: §5 · p. 7

Real-world comparisons

  • On real-world images, the method produces the best visual results for blending boundary, texture, and color illumination among the compared approaches.Figure 7 compares the method with Deep Harmonization, Poisson Blending, and GP-GAN.Source: §5 · p. 7, Figure 7 · p. 8

User study

  • 80–90% of the time, users preferred the method over other methods; it received the highest votes for 9 of 10 real-world images and 8 of 10 painting-style images.Thirty users evaluated 20 image sets; Figure 8 shows the painting and real-world comparisons.Source: Figure 8 · p. 8, §5 · p. 8

6. Conclusion

The paper concludes that its gradient blending loss and two-stage algorithm generate blending images on-the-fly with an L-BFGS solver. It reports that the approach needs no training data, generalizes to real-world images and paintings, and outperforms strong baselines and state-of-the-art approaches in user studies.

  • The paper proposes a novel gradient blending loss and a two-stage algorithm that generates blending images on-the-fly with an L-BFGS solver.Source: §6 · p. 8
  • The algorithm does not rely on training data and can generalize to real-world images and paintings.Source: §6 · p. 8
  • User studies show that the algorithm outperforms strong baselines and state-of-the-art approaches.Source: §6 · p. 8
Loading 1910.11495v1…