PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving the Harmony of the Composite Image by Spatial-Separated Attention Module

Xiaodong Cun, Chi-Man Pun

arXiv:1907.06406v3cs.CV

TL;DR

Image harmonization must reconcile a pasted region with its background while preserving shared content and non-spliced appearance. The paper introduces S2AM, a mask-guided regional attention module integrated into Unet low-level features, and reports better quality than competing attention and harmonization methods.

  • Problem

    Image harmonization must reduce appearance differences in pasted regions while preserving shared high-level content and non-spliced appearance.

  • Method

    S2AM uses hard-coded masks and separate attention pathways to learn masked-region changes and other features within two Unet-based framework variants.

  • Results

    The experiments report better results than other attention mechanisms and state-of-the-art image harmonization methods across quality and quantitative criteria.

  • Takeaways & Limitations

    The proposed regional attention design also supports harmonizing composite images without a specific mask and may extend to other tasks with regional differences.

Abstract

from arXiv · show

Image composition is one of the most important applications in image processing. However, the inharmonious appearance between the spliced region and background degrade the quality of the image. Thus, we address the problem of Image Harmonization: Given a spliced image and the mask of the spliced region, we try to harmonize the "style" of the pasted region with the background (non-spliced region). Previous approaches have been focusing on learning directly by the neural network. In this work, we start from an empirical observation: the differences can only be found in the spliced region between the spliced image and the harmonized result while they share the same semantic information and the appearance in the non-spliced region. Thus, in order to learn the feature map in the masked region and the others individually, we propose a novel attention module named Spatial-Separated Attention Module (S2AM). Furthermore, we design a novel image harmonization framework by inserting the S2AM in the coarser low-level features of the Unet structure in two different ways. Besides image harmonization, we make a big step for harmonizing the composite image without the specific mask under previous observation. The experiments show that the proposed S2AM performs better than other state-of-the-art attention modules in our task. Moreover, we demonstrate the advantages of our model against other state-of-the-art image harmonization methods via criteria from multiple points of view. Code is available at https://github.com/vinthony/s2am

I. INTRODUCTION

Image harmonization seeks to remove visual discrepancies between a pasted region and its background. The paper proposes learning regional appearance changes separately while preserving shared high-level and non-spliced features.

  • Image compositing artifacts arise from color, texture, and luminance differences that make synthesized images visibly unrealistic.
  • The paper observes that harmonization should model dissimilarity in the spliced region while preserving consistency elsewhere and in high-level features.
  • S2AM uses hard-coded masks and channel attention to learn masked and unmasked features separately within an encoder-decoder framework.
  • The framework inserts S2AM into low-level Unet features through two variants while retaining high-level feature consistency.
  • Experiments report that S2AM outperforms other attention mechanisms and that the full harmonization method surpasses state-of-the-art methods.

II. RELATED WORKS

Prior work includes hand-crafted, neural, attention-based, and image-to-image transformation approaches. This paper frames harmonization as supervised image transformation using synthesized composites and a masked encoder-decoder.

  • Earlier harmonization methods used hand-crafted color or texture features, but their narrow focus produced unreliable results when appearances differed substantially.
  • Attention mechanisms weight informative channels or spatial features, while self-attention captures long-range dependencies.
  • Image harmonization can be treated as image-to-image transformation because it changes appearance between an input composite and a target image.
  • The proposed training setup synthesizes composites by modifying a region of a natural image, using the composite and mask as inputs and the natural image as target.
  • The method uses a Unet encoder-decoder with interpolated attention blocks to learn spliced and background regions separately.

A. Spatial-Separated Attention Module (S2AM)

S2AM separates feature processing by spatial region using mask-guided channel attention and a learnable transformation block. It is designed to model masked-region appearance changes while preserving background information.

  • S2AM addresses the limitation of shared convolution kernels by using hard-coded masks to process region-specific appearance changes.
  • Three channel gates weight features for background reconstruction, masked-region differences, and masked-region features that need not change.
  • The module combines the masked-region outputs of Gfg and Gmix with the non-spliced output of Gbg according to the binary mask.
  • S2AM is presented as a general separated-learning form whose channel-attention and learnable blocks can be adapted to other regional-difference tasks.
  • The learnable block L applies convolutional transformations to model appearance changes in the spliced region while preserving feature details.
  • The masks are smoothed with a uniform 7×7 Gaussian kernel to help generate boundaries when hard-coded masks are inaccurate.

B. Generalizing S2AM to Image Harmonization without Mask

The paper removes the hard-coded mask by learning a spatial attention map, then uses S2AM within a U-Net to separate spliced and non-spliced features. Two insertion strategies place S2AM in skip connections or after encoder-decoder feature concatenation, while post-processing combines the network output with the input image.

  • Spatial attention without a mask: A spatial attention block replaces the hard-coded mask and learns the spatial position of the spliced region automatically.Max and average channel pooling feed a convolutional-sigmoid module that produces spatial feature weights.
  • Spatial attention without a mask: An attention loss guides the spatial attention output toward the ground-truth spliced mask at each attention level.The ground-truth mask is resized to supervise the sigmoid outputs because coarse spatial attention may be inaccurate.
  • Network structure: The U-Net backbone preserves high-level consistency while S2AM focuses separately on low-level appearance changes.The method uses U-Net's fully convolutional structure rather than the linear-layer design used by DIH.
  • Network structure: S2ASC replaces original skip connections, whereas S2AD applies S2AM after concatenating encoder and decoded features.Both variants are inserted only at three coarser levels to preserve similar high-level features.
  • Post-processing: The final output combines the network prediction and input image using the upsampled coarse attention mask.For masked harmonization this operation is post-processing; without a mask, the post-processing is trained jointly.

D. Loss Function

The method uses different loss designs for masked and mask-free harmonization because the latter has weaker prior information. Pixel, attention, and adversarial objectives supervise image fidelity, spatial focus, and artifact reduction.

  • Loss design: Masked and mask-free harmonization use different loss functions because mask-free harmonization has weaker prior information.
  • Pixel loss: The pixel-level objective uses an L2 criterion to match predicted and target image values at corresponding spatial positions.The output is denoted Ipredict and the target image I.
  • Attention loss: Attention loss supervises each spatial attention block by comparing its mask with the corresponding ground-truth spliced mask.The paper chooses L2 loss for this attention supervision.
  • Adversarial loss: Adversarial loss uses a Least-Square GAN with Unet plus S2AD as generator and a Patch-based pix2pix discriminator.It is intended to remove artifacts in the predicted image.
  • Combined loss: The total mask-free loss combines attention, pixel, and GAN terms with experimentally fixed α = 90 and β = 100.

E. Data Acquisition

Because no public image-harmonization dataset is available, the paper constructs two synthesized datasets for complementary robustness evaluations. S-COCO uses instance masks and transformed regions, while S-Adobe5K targets inaccurate masks and manually pasted content.

  • Dataset construction: The paper builds two synthesized datasets because no publicly available image-harmonization dataset exists.
  • S-COCO: S-COCO evaluates robustness on tampered images with ground-truth instance masks from COCO.The dataset uses COCO's natural images and human-annotated instance masks.
  • S-COCO: S-COCO transforms a natural image, randomly adjusts brightness, color, and contrast, then crops and pastes the transformed region using its instance mask.
  • S-Adobe5K: S-Adobe5K evaluates robustness to inaccurate ground-truth masks and manually copy-and-pasted regions using Adobe5K raw photographs.Adobe5K contains 5k raw photos, each with five styles produced by different experts.
  • Dataset release: The authors plan to publicly release both synthesized datasets for further research.The dataset-building workflow is summarized in Fig. 6.

IV. EXPERIMENTS

The experiments compare the proposed harmonization variants with established baselines on synthesized datasets, including masked and mask-free settings. Training details, baseline configurations, and numerical comparisons are organized across the experimental sections.

  • Experimental design: The evaluation covers image harmonization and mask-free harmonization on several datasets, followed by detailed S2AM analysis.
  • Training and testing: Models are trained at 256×256 resolution with Adam at learning rate 0.001, and the proposed model runs in 0.012 seconds per image on an NVIDIA 1080 GPU.
  • Baselines: The baseline set includes DIH, RealismCNN, pix2pix, U-Net, and the two S2AM U-Net variants.The comparisons use the same synthesized datasets for the relevant baselines.
  • Numerical comparison: Table I reports numerical comparisons on the synthesized datasets, distinguishing S2ASC, S2AD, and post-processing variants.

C. Comparison

On synthesized datasets, the proposed S2AM-based models outperform comparison methods in numerical, visual, and pretrained-realism evaluations, while S2AD provides stronger global consistency than S2ASC.

  • C. Comparison: Unet+S2ASC achieves the best results among compared methods on S-COCO, with clearer harmonized images and absolute-difference colormaps.The comparison uses a common learning framework while varying model structure.
  • C. Comparison: S2AM-based methods also improve results on S-Adobe5K, including cases with limited data and inaccurate masks.The reported advantage is attributed to separately filtering regional features in the skip-connection.
  • C. Comparison: The pretrained R-CNN realism scores generally improve with S2AM, and Unet+S2AM performs best among the compared methods.The R-CNN model was not trained on the evaluated dataset, while its behavior differs on S-Adobe5K.
  • C. Comparison: S2AD produces more realistic global results than S2ASC by using both high-level and low-level information.The comparison attributes S2AD’s global consistency to upsampled high-level features and global context.

2) Comparison on Real Dataset with User Study.:

On real composite images, the proposed method outperforms DIH and Zhu in the reported comparison, with a larger margin on the 99-image dataset.

  • 2) Comparison on Real Dataset with User Study.: On a real dataset of 99 composite images, the proposed method outperforms DIH and Zhu by a larger margin.The images cover varied scenes and include provided spliced masks.
  • 2) Comparison on Real Dataset with User Study.: Figure 10 compares the proposed method with DIH and Zhu using the authors’ provided pretrained model.The figure reports better results for the proposed method.

1) Comparison on Synthesized Dataset:

The method is also evaluated without a ground-truth mask: it predicts reasonable attention maps and improves harmonization, while attention-loss experiments favor L2 with a 1:10 loss proportion.

  • 1) Comparison on Synthesized Dataset:: The study evaluates the proposed mask-free method on S-COCO and reports its results in Table III.The setup feeds only the color image to the network.
  • 1) Comparison on Synthesized Dataset:: Without the tampered mask as input, the full method produces more realistic harmonized images than the baseline Unet in visual and numerical evaluations.The method also predicts a reasonable intermediate attention map.
  • 2) Evaluation of Attention Loss:: L2 is more suitable than Binary Cross Entropy for attention loss under the same pixel-loss ratio.The comparison uses a 10k-image S-COCO subset trained four times faster than the original dataset.
  • 2) Evaluation of Attention Loss:: The best reported proportion between attention loss and pixel-level loss is 1 : 10.The tested attention-loss proportions include 0.1×, 0.01×, 10×, and the selected setting.

1) Evaluation of Attention Modules:

S2ASC outperforms the tested attention modules, while ablations show contributions from regional separation, Gaussian filtering, low-level placement, and the learning-able block.

  • 1) Evaluation of Attention Modules:: Unet+S2ASC outperforms CONV-Block, SE-Block, CBAM, and the stronger-parameter baseline by a large margin.The reported explanation is that hard-coded masks support separate channel-attention purposes for spliced and non-spliced regions.
  • 2) Ablation Study:: Removing Gbg, Gmix, and Gfg makes the model slightly worse because the learning block must jointly learn necessary and unnecessary features.The ablation is compared with the baseline.
  • 2) Ablation Study:: Removing Gmix slightly reduces performance, consistent with Gmix selecting original spliced-region features that need not be transferred.The comparison is made against S2ASC.
  • 2) Ablation Study:: Gaussian filtering improves performance, particularly at the boundaries of the spliced region.The improvement is reported both visually and numerically.
  • 2) Ablation Study:: Replacing all skip-connections with S2ASC slightly worsens performance, although the six-layer variant remains much better than the baseline.This supports placing style-related processing in lower-level features while preserving high-level features.
  • 2) Ablation Study:: Removing the learning-able block worsens results, especially SSIM, because its convolutions learn additional image detail.SSIM evaluates luminance, contrast, and structure.

3) Attention Analysis:

The attention analysis examines how S2AM weights foreground, mixed, and background features across network levels, and compares its harmonization with alternative attention modules. The conclusion reports better qualitative and quantitative results and points to adaptation beyond image harmonization.

  • Attention Analysis:: Unet+S2AM produces better boundary harmonization and absolute-difference colormaps than Unet+SE-Block and Unet+CBAM.The comparison highlights the boundary of the arms and the corresponding difference maps.
  • Attention Analysis:: S2ASC allocates channel weights according to the input images by comparing response maps between two images.Darker blocks indicate lower channel weighting, while lighter blocks indicate greater importance for filtered features.
  • Attention Analysis:: S2ASC visualizes Gfg, Gmix, and Gbg channel responses across three coarse-to-fine network levels.The visualizations show the first 18 attention channels at S2ASC1, S2ASC2, and S2ASC3.
  • Conclusion: The proposed method achieves better image-harmonization results than other methods in both quality and quantity.The conclusion also states that S2AM can support mask-free harmonization through spatial-attention interpolation and attention loss.
  • Conclusion: The attention module is presented as adaptable to regional-difference tasks including Free-Form Image Inpainting and Semantic Image Synthesis.These applications are identified as promising future work.
Loading 1907.06406v3…