Source-linked AI summary
Deep Image Harmonization
Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, Ming-Hsuan Yang
TL;DR
Image harmonization must make extracted foregrounds compatible with new backgrounds, but prior hand-crafted appearance statistics can be unreliable when layer contents differ. This paper trains an end-to-end CNN that combines context and semantic information and introduces scalable training-pair synthesis; it reports favorable results on synthesized datasets and real composite images.
Problem
Hand-crafted foreground-background appearance statistics are unreliable when the contents of the two composite layers differ, limiting realistic image harmonization.
Method
An end-to-end CNN takes a composite image and foreground mask, combines contextual features with scene-parsing semantic guidance, and uses synthesized high-quality training pairs.
Results
The method performs favorably against existing state-of-the-art algorithms on synthesized datasets and real composite images.
Takeaways & Limitations
Joint context-semantic learning and efficient training-data generation provide a unified approach for harmonizing composite images.
Abstract
from arXiv · showhide
Compositing is one of the most common operations in photo editing. To generate realistic composites, the appearances of foreground and background need to be adjusted to make them compatible. Previous approaches to harmonize composites have focused on learning statistical relationships between hand-crafted appearance features of the foreground and background, which is unreliable especially when the contents in the two layers are vastly different. In this work, we propose an end-to-end deep convolutional neural network for image harmonization, which can capture both the context and semantic information of the composite images during harmonization. We also introduce an efficient way to collect large-scale and high-quality training data that can facilitate the training process. Experiments on the synthesized dataset and real composite images show that the proposed network outperforms previous state-of-the-art methods.
1. Introduction
Image harmonization adjusts extracted foreground appearances to match a new background, addressing limitations of hand-crafted feature statistics. The paper proposes an end-to-end CNN with context and semantic guidance, supported by synthesized training data and evaluations on synthetic and real composites.
- 1. Introduction: Hand-crafted color and texture statistics can produce unreliable harmonization because they ignore composite-image contents.The mismatch is especially problematic when foreground and background contents differ substantially.
- 1. Introduction: The proposed end-to-end deep CNN captures both context and semantic information while directly producing a harmonized image from a composite and foreground mask.It preserves input contents and adjusts appearances on the foreground region.
- 1. Introduction: A scene-parsing decoder and joint training scheme provide semantic guidance to the harmonization decoder without requiring high-level semantic-label supervision.The shared encoder propagates semantic information so the model can use scene cues when adjusting foreground appearance.
- 1. Introduction: The paper develops an efficient method for synthesizing large-scale, high-quality training pairs because expert-created harmonized composites are difficult to collect.The data-generation approach is intended to facilitate training of the end-to-end CNN.
- 1. Introduction: Experiments compare the method with existing approaches on synthesized datasets and real composite images, with favorable performance reported in both settings.The real-image evaluation uses a user study, while synthesized data supports quantitative comparison.
- 1. Introduction: The work contributes an end-to-end harmonization approach, a joint context-semantic CNN, and an efficient training-data collection method.These contributions are presented as the paper’s main advances in image harmonization.
2. Related Work
Prior image-harmonization methods match foreground and background appearances using statistics, gradients, multiscale features, semantic information, or realism models. The paper extends learning-based image editing with an end-to-end CNN that explicitly incorporates scene semantics.
- 2. Related Work: Existing harmonization methods match appearances using global, gradient-domain, multiscale, or semantic techniques, while realism-based methods optimize appearance using predicted realism.These approaches address appearance compatibility through increasingly varied feature and realism objectives.
- 2. Related Work: Earlier methods include color and tone transfer, realism prediction, and optimization of foreground appearances against background statistics.The cited approaches differ in whether they directly match appearances or use a learned realism score.
- 2. Related Work: Neural image-editing methods for colorization, inpainting, and filtering learn feature representations from raw input data using unsupervised schemes.The proposed method shares this learning-based concept but targets image harmonization.
- 2. Related Work: The proposed architecture is described as the first end-to-end trainable CNN designed specifically for image harmonization.Its distinction from prior learning-based editing methods is the task-specific harmonization design.
- 2. Related Work: Because reconstruction pipelines may miss fine-level semantics, the paper adds pixel-wise scene parsing and propagates that information to the harmonization model.The full framework remains end-to-end trainable.
3. Deep Image Harmonization
The method builds large-scale synthesized training pairs and an end-to-end joint CNN that harmonizes foreground appearances using contextual and semantic information. Experiments and analyses support realistic outputs, quantitative evaluation, and improved performance from larger inputs and semantic joint training.
- Data Acquisition: Training pairs are synthesized by editing realistic images, preserving realistic ground truth while creating composite inputs with mismatched foreground appearances.The process begins with a real image as ground truth, edits a selected region, and uses the edited image as the composite input.
- Data Acquisition: The data collection pipeline combines color transfer for COCO and Flickr with style-based editing for MIT-Adobe FiveK.COCO and Flickr use reference-based luminance and color-temperature transfer, while FiveK exploits multiple professional retouching styles.
- Data Acquisition: Large-scale, high-quality pairs make ground-truth realism learnable and enable quantitative comparison against realistic reference images.The synthesized data support error measurements even though image harmonization may have no single best solution.
- Context-aware Encoder-decoder: The encoder-decoder network uses skip links to retain image details and textures while learning representations from low-level details to high-level context.The encoder has no pooling layers, and skip links are introduced because L2 optimization can produce blurry outputs missing details.
- Joint Training with Semantics: The joint architecture adds a scene parsing decoder sharing the encoder and passes semantic feature maps to the harmonization decoder.Joint training propagates semantic information so foreground appearances can be adjusted according to categories and surrounding background semantics.
4. Experimental Results
Experiments on synthesized and real composite images show that the proposed joint network improves harmonization quality, realism, semantic adjustment, mask generalization, and runtime over prior approaches.
- Synthesized Data: The joint network consistently outperforms the single network without scene parsing and other state-of-the-art methods on all three synthesized datasets in MSE and PSNR.The baseline without semantics also outperforms other existing methods.
- Synthesized Data: Semantic information enables the network to adjust foreground appearance according to scene content, producing more realistic results than the version without semantics.An example recovers skin color and obtains a higher PSNR score with semantics.
- Real Composite Images: The network achieves the highest B-T realism score on both the created real-image test set and examples from prior work.It is also the only method reported to improve harmonization significantly over the cut-and-paste input.
- Real Composite Images: The joint network produces realistic outputs on real composites containing varied scenes and objects, supporting direct learning from synthesized data with realistic ground truths.The evaluated real-image examples include composite foreground regions that are objects or scenes.
- Generalization to Background Masks: The model generalizes to arbitrary foreground masks, including inverted masks, while adjusting the selected region using context and semantic information.The reported example changes the harmonized region from a child to the inverted child mask.
- Runtime Performance: 0.1 seconds on a Titan X GPU and 3 seconds on a CPU are reported for harmonizing a 512 × 512 image, versus more than 10 seconds for prior methods on CPU.The comparison concerns methods based on statistic matching or adjustment-function optimization.
5. Concluding Remarks
The paper presents an end-to-end network that captures context and semantics for image harmonization and introduces efficient large-scale training-pair collection. Experiments show favorable performance on synthesized datasets and real composite images against state-of-the-art algorithms.
- Concluding Remarks: The proposed joint network captures context and semantic information for image harmonization in an end-to-end trainable framework.The semantic decoder branch provides semantics to help harmonization.
- Concluding Remarks: An efficient method for collecting large-scale, high-quality training pairs facilitates training for image harmonization.The contribution addresses the need for substantial training data.
- Concluding Remarks: Experiments show favorable performance on synthesized datasets and real composite images compared with other state-of-the-art algorithms.The conclusion reports results across both evaluation settings.