Source-linked AI summary
Hybrid Generative-Discriminative Object Placement
Siyuan Zhou, Li Niu
TL;DR
Object placement must predict plausible foreground locations and scales, while existing generative and discriminative methods do not balance efficiency and effectiveness well. The paper proposes an anchor-based semi-generative model that fuses foreground and background features to score anchors and generate placement sets, achieving diverse, plausible results with a favorable efficiency-effectiveness balance on OPA.
Problem
Object placement must predict plausible foreground location and scale, but existing generative and discriminative methods do not balance efficiency and effectiveness well.
Method
The method assigns anchors, fuses foreground and background features, predicts rationality for each anchor, and generates placement sets for positive anchors.
Results
Experiments on OPA show high-quality rationality maps, diverse and plausible placement sets, and comparable or better results with significantly faster inference than FOPA and GracoNet.
Takeaways & Limitations
The anchor-based semi-generative design combines advantages of generative and discriminative placement methods while achieving a balance between efficiency and effectiveness.
Takeaways & Limitations
The method may predict unreasonable scales and locations when backgrounds are highly cluttered and offer limited space for plausible placement.
Abstract
from arXiv · showhide
As an important operation of image composition, object placement aims to predict the plausible placement (location, scale) for the inserted foreground object. Previous object placement methods can be divided into generative methods and discriminative methods, both of which cannot balance efficiency and effectiveness well. In this work, we propose a semi-generative method in the middle ground between them. In particular, we assign uniformly distributed anchors on the background. Then, we fuse foreground and background features to predict the rationality score for each anchor and predict plausible placement sets for positive anchors. Extensive experiments on the OPA dataset show that our method can strike a good balance between efficiency and effectiveness.
1. Introduction
Object placement learning predicts plausible foreground locations and scales, but generative and discriminative paradigms trade coverage, quality, and efficiency. The proposed semi-generative approach combines anchor-based rationality prediction with diverse placement generation and adaptive foreground-background fusion.
- Object placement predicts a foreground object's plausible location and scale within a foreground-background pair.
- Generative methods produce placements but may suffer mode collapse, limited location coverage, and lower prediction quality.
- Discriminative methods score rationality across locations and scales, covering locations with better quality but requiring substantially higher computational cost.
- The semi-generative model assigns anchors, predicts rationality for each anchor, and generates placement sets for positive anchors.
- A dynamic fusion module uses foreground and background information to adaptively aggregate contextual information at different scales through deformable convolution.
- Experiments on the OPA dataset indicate that the method balances efficiency and effectiveness at lower cost than discriminative methods.
3. Our Method
The method uses uniformly distributed background anchors to predict anchor rationality and plausible placement sets, combining discriminative coverage with generative placement prediction. It fuses foreground and background features dynamically, then trains with sparse annotations, matching, assessment, and self-training strategies.
- Anchor Representation: The semi-generative approach treats each background feature-map location as an anchor and predicts whether its neighborhood contains a plausible foreground placement.Anchors are uniformly distributed by dividing the background into h × w cells; rationality scores classify anchors as positive or negative.
- Dynamic Feature Fusion: Dynamic Fusion Module adaptively aggregates foreground-conditioned background context before producing the fused feature map used for placement prediction.Foreground features are spatially replicated and concatenated with background features to predict deformable-convolution offsets, whose output is fused again with foreground features.
- Placement Prediction: For each positive anchor, a prediction head maps fused features and random vectors to placement parameters consisting of center offset and foreground scale.Sampling multiple random vectors produces multiple placements; a shared scale preserves the foreground object's aspect ratio.
- Training Objectives: The training procedure combines rationality classification, mode seeking, placement matching, and placement assessment losses for predicted positive-anchor placements.Predicted placement sets are matched to ground-truth sets using bipartite matching, while SOPA supervises the plausibility of resulting composite images.
- Training Strategies: Self-training addresses sparse and noisy OPA annotations by promoting high-scoring unannotated anchors and augmenting positive placement sets.Negative anchors may contain unannotated positive placements, while annotated positive anchors may have additional plausible placements.
4. Experiments
Experiments on OPA evaluate the method’s data setting, metrics, baselines, efficiency, ablations, and qualitative behavior. The results indicate comparable effectiveness to FOPA at substantially lower computational cost, while self-training and model components improve performance.
- Dataset: OPA contains 62,074 training and 11,396 testing composite images, with non-overlapping foregrounds and backgrounds between splits.The dataset includes 4,137 foreground objects from 47 categories and 1,389 background images.
- Evaluation Metrics: Evaluation measures plausibility with user study, accuracy, and FID, and measures diversity with LPIPS.Efficiency is reported using FLOPS and average inference time per foreground-background pair.
- Comparison with Baselines: The method significantly outperforms generative methods and achieves comparable results with FOPA, while requiring efficiency comparable to GracoNet and much less computational cost than FOPA.FOPA’s higher computational cost is attributed to its complex network structure.
- Ablation Studies: Replacing deformable convolution worsens performance, removing mode-seeking loss significantly degrades diversity, and both matching and classification losses contribute to performance.The classification loss is reported to be more important than the matching loss because it pushes generated composites from positive anchors to be positive.
- Ablation Studies: The self-training stage could boost performance by refining annotations when compared with the first training stage.The model uses transformed annotations that associate placements with positive or negative anchors.
- Visualization Results: Qualitatively, the method produces diverse and plausible placements, while GracoNet tends toward large unreasonable foregrounds and FOPA can produce unreasonable occlusions or locations.The method is reported to be significantly faster than FOPA.
- Visualization Results: The method and FOPA identify broadly coinciding positive anchors, while the method can identify free space that FOPA includes with extra unreasonable locations.The comparison uses rationality maps across 16 discrete foreground scales and max pooling to obtain an anchor-level map.
5. Conclusion
The paper develops an anchor-based semi-generative object placement method to combine advantages of generative and discriminative approaches. It produces rationality score maps and diverse, plausible placement sets, with experiments on OPA demonstrating the method’s superiority.
- Conclusion: The method combines generative and discriminative object placement through an anchor-based semi-generative learning approach.It produces high-quality rationality score maps and diverse, plausible placement sets.
Supplementary for Hybrid Generative-Discriminative Object Placement
The supplementary material examines hyper-parameters, user-study details, additional visualizations, dynamic-fusion offsets, and limitations. It also discusses cases where clutter and limited space make reasonable placement difficult.
- Supplementary Material: The supplementary material covers hyper-parameters, user-study details, additional visualizations, dynamic-fusion offsets, further method visualizations, and limitations.Its limitation discussion reports failures on reasonable scale and location prediction in cluttered backgrounds with limited plausible space.
1. Hyper-parameter Analyses
The analysis varies anchor count, sampling times, self-training thresholds, and loss weights to assess their effects on performance. Results indicate benefits from sufficiently many samples and carefully chosen thresholds, while loss weights are relatively robust.
- Sampling times: As T increases from 5 to 25, performance first improves and then stabilizes, motivating the default T = 10.OPA test pairs contain at most three positive placements per anchor, and sufficiently large T can improve plausibility and diversity.
- Self-training thresholds: Low τc or τs reduces accuracy because noisy positive anchors or placements can mislead self-training, although LPIPS becomes high.The default thresholds are τc = 0.6 and τs = 0.85.
- Loss weights: The method is relatively robust to λ1 and λ2 across the tested range [10^-2, 10^2].Each weight is varied while fixing the other at its default value: λ2 = 5 or λ1 = 10.
2. Details of User Study
The user study compares plausibility judgments for composite images generated by five object-placement methods. Participants select the most plausible image for each test sample, and the selection proportions are computed.
- Study design: Fifty voluntary participants compare images generated by TERSE, PlaceNet, GracoNet, FOPA, and the proposed method.For each test sample, every participant selects the method producing the most plausible composite image.
- Study design: The study reports each method's proportion of selections as the best-performing method among the 50 participants.The supplied study description specifies the aggregation procedure but not the resulting proportions.
3. More Visual Comparison with Baselines
Visual comparisons examine generated composites from GracoNet, FOPA, and the proposed method across foreground-background pairs. The proposed method is described as matching or exceeding FOPA in plausibility and diversity while being more efficient.
- Visual comparison: GracoNet often generates overly large foregrounds and insufficient location diversity in the visual examples.These issues are identified in rows 1, 2, 3, and 5 of Figure 2.
- Visual comparison: FOPA produces more plausible and diverse composites, but some examples have unreasonable locations or scales.Examples include a cup placed on food and a bus placed too small in front of a truck.
- Additional examples: Figure 4 shows ten composites per foreground-background pair, with two images from the same positive anchor grouped together.The figure illustrates multiple outputs associated with positive anchors.
- Failure cases: Figure 5 presents two example failure cases of the proposed method.The supplied caption identifies the examples as failures without specifying their causes.
- Visual comparison: The proposed method's generated composites are comparable or better than FOPA in plausibility and diversity, while being much more efficient.The efficiency comparison is referenced to Table 1 in the main paper.
4. Offset Visualization
Offset visualizations show that the Dynamic Fusion Module adapts convolution-kernel context to foreground properties and local background conditions. Larger objects induce larger offsets, while nearby occluders receive increased attention.
- Dynamic Fusion Module: The Dynamic Fusion Module predicts convolution-kernel offsets from concatenated background features and replicated foreground features.Offsets are visualized for selected anchor locations to examine adaptive contextual aggregation.
- Foreground-dependent offsets: Larger foreground physical size produces larger predicted offsets at the same location on the same background.This reflects the need to inspect larger-scale contextual information for large foregrounds, including possible occlusions.
- Location-dependent offsets: Kernel-center location changes the predicted offsets for the same foreground and background.When an unreasonable occluder is near the center, offsets attend more to the occluder.
- Interpretation: The visualizations indicate that predicted offsets depend jointly on foreground information and background context.This supports dynamically aggregating contextual information at different scales.
5. More Visualization Results of Our Method
The method generates diverse and plausible composite images by sampling from multiple positive anchors and random vectors. This covers different reasonable object locations and scales.
- Five positive anchors are randomly selected using rationality scores above 0.5.Two composite images are then generated for each selected anchor by sampling the random vector twice.
- The generated composite images are usually diverse and plausible across reasonable locations and scales.
6. Failure Case
The method can fail in challenging placement cases, particularly when clutter leaves little room for a plausible object placement.
- Cluttered backgrounds with limited plausible placement space can produce unsatisfactory scale and location predictions.The paper gives a desk example where the available space is limited.