Source-linked AI summary
Learning a Discriminative Model for the Perception of Realism in Composite Images
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, Alexei A. Efros
TL;DR
Because realism depends on many visual factors, the paper learns its perception from automatically generated composites rather than direct human realism labels. A CNN distinguishes natural photos from composites, predicts human-perceived realism, and guides color adjustment that outperforms existing methods in human ratings.
Problem
Assessing realism is challenging because many visual factors must align, and direct realism labels are costly to obtain at scale.
Source: §1 · pp. 1–2Method
Results
The model predicts human-perceived composite realism; RealismCNN + SVM reaches 0.88 accuracy, and the color-adjustment method outperforms existing methods on average human ratings.
Source: §6 · p. 6, §6.1 · p. 7, §7 · p. 8Takeaways & Limitations
The learned realism model can guide automatic color adjustment and object selection for image compositing.
Source: §7 · p. 8The model does not capture high-level scene cues such as semantics, layout, or perspective because composites use same-category, similarly shaped object replacements.
Source: §7 · p. 8
Abstract
from arXiv · showhide
What makes an image appear realistic? In this work, we are answering this question from a data-driven perspective by learning the perception of visual realism directly from large amounts of data. In particular, we train a Convolutional Neural Network (CNN) model that distinguishes natural photographs from automatically generated composite images. The model learns to predict visual realism of a scene in terms of color, lighting and texture compatibility, without any human annotations pertaining to it. Our model outperforms previous works that rely on hand-crafted heuristics, for the task of classifying realistic vs. unrealistic photos. Furthermore, we apply our learned model to compute optimal parameters of a compositing method, to maximize the visual realism score predicted by our CNN model. We demonstrate its advantage against existing methods via a human perception study.
1. Introduction
Judging realism is difficult because many visual details must align, while computers struggle to distinguish natural images from the broad range of unrealistic ones. The paper trains a CNN on natural photos and automatically generated composites to predict perceived realism and guide composite adjustment.
1. Introduction
- The problem is difficult because unrealistic images span a much broader range than natural images, and no existing method had reliably identified images on the natural-image manifold.Human judgments of realism depend on many factors, where a single mismatch can undermine the impression.Source: §1 · p. 1
- The study narrows its focus to appearance by swapping similarly shaped objects of the same category, keeping scene layout and semantics constant.The authors describe this as a limited domain and aim to predict whether people will perceive the resulting composites as realistic.Source: §1 · p. 1
- The paper trains a CNN to distinguish natural photos from automatically generated composites, using that pretext task to predict whether composites will look realistic to people.The model is trained without costly human realism labels and is reported to capture color, lighting, and texture compatibility.Source: §1 · p. 2
- The learned realism score can optimize color adjustment, outperforming previous color-adjustment methods in a human study, and can help select objects for a background.Source: §1 · p. 2
2. Related Work
Prior realism and compositing approaches use generative image statistics or task-specific heuristics, while forensic methods pursue a different goal. This work instead learns from automatically generated composites to predict which ones appear realistic.
2. Related Work
- Generative natural-image-statistics models capture local filter or patch properties but are inadequate for the higher-level visual information needed to assess photo realism.Source: §2 · p. 2
- Earlier discriminative methods can outperform generative ones by simulating examples labeled with parameters of the data-generation process; this approach instead uses non-task-specific generated negatives without recording those parameters.Source: §2 · p. 2
- Traditional compositing methods suppress artifacts with hand-crafted heuristics but do not directly measure or improve realism; the cited outdoor-composite work focuses only on lighting direction inconsistencies.The described heuristics address color transitions, color and texture inconsistencies, and bleeding artifacts, often under mask or background conditions.Source: §2 · p. 2
- Lalonde and Efros rank composites for visual realism using hand-crafted color-histogram features rather than learning.Source: §2 · p. 2
- Digital image forensics detects manipulations that may be imperceptible, whereas this work predicts which fake images will look realistic.Source: §2 · p. 2
3. Learning the Perception of Realism
The method learns realism prediction from a natural-versus-composite pretext task using generated object-replacement examples. Its composite-generation variants provide training data with different annotation requirements, and the model ranks examples in a way reported to match human perception.
3. Learning the Perception of Realism
- A CNN learns a natural-versus-generated pretext classification task using free labels, and experiments show that it performs well on a manually annotated realistic-versus-unrealistic test set.Source: §3 · pp. 2–3
3.1. Automatically Generating Composites
- FullySupervised generation replaces an object with a shape-matched source from another image, using detailed segmentation masks for both source and target.The rough shape matching is intended to keep generated composites close to the natural-image manifold.Source: §3.1 · p. 3, Figure 3 · p. 3
- PartiallySupervised generation uses labeled target masks with proposal-based source objects, while Unsupervised generation uses proposals for both source and target objects.Source: §3.1 · p. 3
- The trained CNN ranks visually appealing composites above unrealistic photos with visible artifacts, with high-ranked examples harder for people to spot and low-ranked examples showing segmentation or color problems.Source: Figure 5 · p. 4, §3.1 · p. 4
- Unsupervised generation performs only slightly worse than FullySupervised against human annotations, while random unaligned cuts lead the CNN to rely on high-frequency edge artifacts and perform significantly worse.The training set uses about 11,000 natural images and 25,000 object instances from the largest 15 LabelMe categories.Source: §3.1 · p. 4
- Generated composites can be artifact-free and realistic-looking, requiring the CNN to learn visual compatibility as well as segmentation and blending artifacts; the three generation approaches have distinct failure patterns.FullySupervised examples can have lighting or color inconsistencies, PartiallySupervised examples may paste objects onto other objects, and Unsupervised examples may have incorrect semantics.Source: §3.1 · p. 4
4. Improving Image Composites
The section uses the learned realism score to guide color adjustment and object choice in compositing. Its adjustment balances realism against preserving the source object's identity, under assumptions of a well-segmented foreground and reasonable placement.
4. Improving Image Composites
- Object composition blends an adjusted foreground with a background through an alpha mask, assuming the foreground is well segmented and reasonably placed.The adjustment is intended to make foreground visual properties compatible with the background.Source: §4 · p. 4
- The color-adjustment objective rewards composite realism while regularization keeps the result close to the original object's identity, with the regularization weight set to 50.The paper uses a brightness-and-contrast mapping independently for each color channel; its regularizer penalizes large changes and discourages independent channel variation.Source: §4 · pp. 4–5
- The adjustment is optimized by gradient descent when the color mapping is differentiable, using L-BFGS-B and multiple random starts because the objective is nonconvex.The solution with minimum cost is selected.Source: §4 · p. 5
- The learned model's color adjustment generates perceptually better composites than existing methods, while the model can also select more suitable objects for composition.The comparison is reported for perceptual evaluation; object selection is described separately.Source: §4 · p. 5
5. Implementation
The implementation fine-tunes an ImageNet-pretrained VGG model for natural-photo versus composite classification and restricts dataset objects by size and annotation criteria.
5. Implementation
- The CNN is an ImageNet-pretrained VGG model fine-tuned for binary classification of natural photos versus composites using SGD.Training uses a 0.0001 initial learning rate, reduced tenfold after 10,000 iterations, with fc8 learning rate ten times that of lower layers, momentum 0.9, batch size 50, and at most 25,000 iterations.Source: §5 · p. 5
- Dataset generation includes objects occupying 5%–50% of image pixels; human annotation excludes occluded objects with specified label-string terms.Excluded terms are “part”, “occlude”, “regions” and “crop”.Source: §5 · p. 5
6. Experiments
The experiments show that the realism model outperforms prior approaches on classification and can guide color adjustment and object selection. Human ratings favor its color adjustment and selected objects in the tested settings.
6. Experiments
- RealismCNN + SVM achieves 0.88 AUC, outperforming prior low-level-statistics methods and AlexNet (0.75), PlaceCNN (0.73), and VGG (0.76).The evaluation classifies 359 unrealistic photos versus 360 realistic photos, including natural images and realistic composites.Source: §6 · pp. 5–6, Table 1 · p. 5
- Human annotators reach 0.91 AUC, near the model’s agreement level; FullySupervised training performs best without realism labels, while SVM supervision narrows gaps between generation procedures.The study also reports 0.83 AUC for RealismCNN alone on an indoor-photo dataset.Source: §6 · pp. 6–7, Table 2 · p. 6
6.1. Optimizing Color Compatibility
- Our color adjustment method earns the highest average human ratings overall and notably improves unrealistic composites, while no method notably improves realistic composites.For realistic composites and natural photos, the model typically changes little; other methods continue matching low-level statistics.Source: §6.1 · p. 7
- Hard negative mining avoids extreme colors and improves human ratings across iterations, reaching 0.117 for CNNIter3.The final color-adjustment results use CNNIter3 with regularization restored.Source: Figure 8 · p. 7, §6.1 · p. 8
6.2. Selecting Suitable Object
- For object selection, RealismCNN receives an average human rating of 0.285, versus −0.033 for Shape and −0.252 for Random.Candidates are drawn from 25 same-category objects with similar shapes for each background.Source: §6.2 · p. 8
7. Conclusion
The conclusion presents a learned model for predicting whether composites appear realistic to human observers. It also identifies omitted high-level scene cues as a limitation and points to color adjustment and object selection as applications.
7. Conclusion
- The learned model predicts whether a composite will be perceived as realistic by a human observer.The model characterizes the space of natural images using automatically created composites.Source: §7 · p. 8
- The model can guide automatic color adjustment and object selection for image compositing.The paper illustrates object selection by comparing RealismCNN, shape matching, and random selection.Source: §7 · p. 8, Figure 9 · p. 8
- The model captures visual cues such as color, lighting, and segment compatibility, but not high-level semantics, layout, or perspective.The composites replace objects with same-category, similarly shaped objects; the authors identify high-level cues as requiring further investigation.Source: §7 · p. 8