Source-linked AI summary
presto: Efficient, Training-free, and Open-world Object Placement via Imaginary Search
Weixuan Ding, Shang Liu, Hanyu Pei, Zeyan Liu
TL;DR
Object placement requires spatially and semantically plausible positioning of novel objects in unseen scenes, but existing methods rely on limited rules or supervision. Presto reformulates the task as MLLM-guided heuristic search, and achieves outstanding results without category-specific supervision while outperforming baselines on standard metrics and human alignment.
Problem
Object placement must resolve spatial and semantic coherence across diverse scenes, while open-world placement requires positioning novel objects in unseen scenes without task-specific training.
Method
Presto is a zero-shot, training-free framework that uses MLLM-guided decisions in an imaginary action space to iteratively refine object position and scale through multiscale search.
Results
Presto achieves outstanding results without category-specific supervision, outperforming baselines on both standard metrics and human alignment across OPA and OPAZ datasets.
Takeaways & Limitations
Presto’s plug-and-play design and strong generalization make it suitable for open-world object placement applications.
Takeaways & Limitations
Object placement lacks discrete ground-truth labels, and plausible placements vary continuously with position, scale, and semantic fit.
Abstract
from arXiv · showhide
Object placement is critical in image composition, requiring spatially and semantically coherent positioning of objects within diverse scenes. Existing approaches typically rely on hand-crafted rules or supervised learning on limited datasets, which restricts their generalization and interpretability, especially in open-world scenarios involving novel objects and scenes. In this work, we reformulate open-world object placement as a heuristic search task guided by reasoning from a Multimodal Large Language Model (MLLM). We introduce \textsf{presto}, a zero-shot, training-free framework that operates within an imaginary action space to iteratively refine object position and scale. Our coarse-to-fine search strategy ensures fast convergence, and we evaluate two decision-making variants: Metric-guided Selection and MLLM-as-a-judge. Experiments across multiple benchmarks show that \textsf{presto}~achieves state-of-the-art performance, particularly in previously unseen, open-world settings. Human studies further reveal that the MLLM-as-a-judge variant produces more perceptually coherent placements than metric-driven approaches, highlighting a gap between standard evaluation metrics and human visual judgment.
1 Introduction
Object placement requires spatial and semantic coherence, yet existing rules and supervised methods struggle with open-world generalization. Presto reframes placement as MLLM-guided heuristic search and reports strong benchmark and human-evaluation performance.
- Motivation: Object placement must resolve boundary, appearance, geometry, occlusion, and semantic inconsistencies when inserting foreground objects into scenes.The task supports image editing, augmented reality, artistic design, and synthetic data generation.
- Prior limitations: Rule-based methods use handcrafted constraints, whereas supervised methods depend on annotated datasets and incur substantial training and annotation costs.These approaches can perform well in controlled settings but are limited for open-world use.
- Motivation: Open-world generalization requires positioning novel objects in unseen scenes without task-specific training while satisfying spatial and semantic constraints.Valid placements form a continuous solution space, such as placing a cake on a table rather than a keyboard.
- Approach: Presto reformulates placement as MLLM-guided heuristic search, iteratively refining position and scale instead of making a one-shot prediction.The framework is training-free and zero-shot, using an imaginary action space and coarse-to-fine search.
- Findings: MLLM-as-a-judge scores lower on automatic metrics but performs better in human evaluations, indicating a disconnect between metrics and perceptual judgment.The paper compares Metric-guided Selection with MLLM-as-a-judge.
2 Related Works
Object placement methods evolved from handcrafted constraints to category-specific and instance-specific deep models, but practical precision and usability remain concerns.
- Task definition: Object placement inserts a foreground object into a background so the composite appears natural and contextually appropriate.A reasonable placement chooses location, size, and shape to avoid disproportionate objects, awkward occlusions, and physically impossible positioning.
- Traditional methods: Traditional approaches rely on geometric, depth, class-specific, or appearance-consistency constraints, but often lack precision and practical usability.These methods are primarily rule-based.
- Deep methods: More recent deep neural methods are categorized as category-specific predictors or instance-specific models for individual foreground-background pairs.Category-specific methods generalize across instances within a category, while instance-specific methods determine transformations for each pair.
3 Preliminaries
Object placement is formulated as optimizing position and scale from separate foreground and background inputs, with MLLM prompting explored as a training-free alternative. Initial evaluations show that direct prompting struggles because placement requires joint spatial-semantic reasoning over continuous, plausibly multiple solutions.
- Problem Definition: Object placement generates a composite image and mask by positioning and scaling a foreground object within a background scene.The placement parameters are ξ = (x, y, r), where x and y specify position and r specifies the foreground scaling factor.
- Problem Definition: The optimization objective is to find placement parameters that maximize a quality score such as SimOPA.The score evaluates visual and contextual alignment with the original scene.
- MLLM Prompting: MLLM prompting uses the foreground image, foreground mask, background image, and textual instruction to predict placement parameters.The instruction provides task context and encourages reasoning, while the model returns predicted (x, y, r).
- MLLM Prompting: Prompting offers resource, time, data, and open-world advantages because it avoids dedicated training and curated placement annotations.The supplied discussion also identifies deployment flexibility and potential interpretability through step-by-step responses.
- Key Observations and Motivation: GPT-4o reached 71% zero-shot accuracy and 78% with Zero-shot CoT on OPA, but prompting strategies still lacked consistently accurate placements or reliable rationales.The tested models included GPT-4o, o4-mini, and DeepSeek-VL2; results varied from 44% to 78%.
- Key Observations and Motivation: Direct prompting is difficult because placement jointly reasons over separate inputs, generates contextually coherent content, and lacks discrete ground-truth labels.Plausible placements vary continuously with position, scale, and semantic fit.
- Key Observations and Motivation: Presto addresses these difficulties by reformulating placement as heuristic search guided by incremental MLLM decisions rather than one-shot prediction.The placement is progressively refined through potential adjustments such as position and size.
4 The Presto Framework
Presto performs training-free local search over position and scale using MLLM-guided actions, coarse-to-fine refinement, and either metric-based or MLLM-based selection. Its constrained action choices reduce search complexity, while MLLM-based judgments align more closely with human perception.
- Framework Overview: Presto uses initialization, iterative heuristic optimization, and selection to refine open-world object placements.The framework resembles local hill climbing and integrates MLLM-driven decision-making.
- Initialization: Initialization generates multiple candidate placements, evaluates their semantic quality, and retains the highest-coherence candidate as the optimization start.The starting placement is represented by ξ^(0) = (x^(0), y^(0), r^(0)).
- Iterative Optimization: Presto separates movement and resizing because MLLMs struggle to optimize position and size simultaneously.Movement changes (x, y), while resizing changes r; position is prioritized because it affects semantic rationality more significantly.
- Iterative Optimization: Multiscale search begins with large adjustments and progressively reduces step sizes using decay factor α for coarse-to-fine refinement.The schedule supports broad exploration followed by fine-grained optimization in a few iterations.
- Iterative Optimization: Elitism retains the best candidate across iterations until convergence or early stopping when improvements cease.Each iteration explores controlled movement and resizing actions before selecting the next candidate.
- Imaginary Action Space: The imaginary action space represents normalized position (x, y) ∈ [0, 1] and scale r ∈ (0.0, 1.0].Position is measured proportionally from the background’s top-left corner, while r specifies size relative to background dimensions.
- Imaginary Action Space: The resized foreground dimensions are computed from the scale r and the original foreground and background dimensions.The notation uses wb and hb for background width and height, and wf and hf for foreground width and height.
- Imaginary Action Space: Bounding boxes are converted into edge coordinates from the object center and size to make placements more suitable for MLLM use.The coordinates represent left, bottom, right, and top edges.
5 Experiment
Presto is evaluated on standard and open-world placement benchmarks using automatic metrics, qualitative comparisons, human ratings, and ablations. It achieves strong accuracy and generalization, while MLLM-based judgments align better with human perceptual assessments than metric-driven selection.
- Settings and Metrics: Presto is evaluated on OPA and OPAZ using SimOPA accuracy, FID, LPIPS, qualitative comparisons, and human rationality ratings.The human study collected 1,320 ratings from 88 responses.
- Main Results: 0.958 accuracy makes Metric-guided Presto the top OPA method, while MLLM-as-a-judge reaches 0.926 and surpasses six of seven baselines.The baselines were supervised models trained on OPA, whereas Presto was zero-shot and training-free; average convergence took 1.02 and 2.61 iterations, respectively.
- Main Results: 23.05 FID lets Metric-guided Presto outperform four OPA baselines, while the MLLM-as-a-judge variant beats six baselines.Although CSENet has better FID, human evaluators find Presto’s placements more realistic.
- Main Results: On OPAZ, both Presto variants outperform every baseline in accuracy, with Metric-guided Selection improving accuracy by 23.9% to 41.7%.Metric-guided Presto trails only IOPRE in FID but has 17.1% higher accuracy; LPIPS diversity is lower than some baselines.
- Qualitative and Human Evaluation: Presto produces more plausible locations and scales in qualitative comparisons, including commonsense object placement and contextually appropriate relative sizing.Examples include clocks on walls, fire hydrants near sidewalks, smaller distant airplanes, and a teddy bear scaled among toys.
- Qualitative and Human Evaluation: Human ratings favor Presto over baselines, and MLLM-as-a-judge receives higher ratings than Metric-guided Selection despite lower accuracy.Presto averages 7.09 on OPA and 5.97 and 7.2 on OPAZ, whereas baselines average 3.11–4.54.
- Ablation Study: Removing MLLM guidance reduces performance by over 20%, while using MLLM only for optimization or initialization improves the baseline by 14% or 18%.The ablation separates MLLM contributions to initialization and iterative heuristic optimization.
- Ablation Study: Qwen-VL2.5-7B achieves SimOPA accuracies of 85% and 61%, outperforming four and six state-of-the-art methods, respectively.Presto also improves accuracy across Qwen-VL2.5-7B and DeepSeek-VL2-27B configurations.
6 Conclusion
The paper reformulates object placement as MLLM-guided heuristic search and introduces Presto, a zero-shot, training-free framework for iterative position and scale adjustment. Experiments on OPA and OPAZ report strong results without category-specific supervision, including performance on standard metrics and human alignment.
- Conclusion: Presto reformulates object placement as heuristic search in an imaginary search space, using MLLM-guided decisions to iteratively adjust position and scale.The framework is zero-shot, training-free, and does not require category-specific supervision.
- Conclusion: Experiments on OPA and OPAZ show that Presto outperforms baselines on standard metrics and human alignment while generalizing to open-world applications.The paper also emphasizes Presto’s plug-and-play design and strong generalization.