Source-linked AI summary
Scene Retargeting: Learning Object Placement with Analogical Transfer
Minkwan Kim, Junho Kim, Seungmin Lee, Changwoon Choi, Young Min Kim
TL;DR
Scene synthesis must preserve functional context while meeting the physical constraints of sparse, irregular layouts. Scene retargeting transfers an exemplar’s organization through cluster placement, foundation-model features, and physical refinement. It outperforms existing baselines on physical validity and semantic coherence metrics, with downstream applications including real-to-sim transfer, analogical trajectory transfer, and multi-reference composition.
Problem
Automated scene synthesis must reproduce functional context while maintaining geometric plausibility under sparse, irregular, scene-specific constraints.
Source: §1 · p. 1Method
Results
The method outperforms existing baselines across physical validity and semantic coherence metrics.
Source: §5 · p. 10Takeaways & Limitations
The pipeline applies to real-to-sim scene transport, analogical trajectory transfer, and multi-reference composition.
Source: §1 · p. 2The framework focuses on 2D floor-plane poses and comparable inventories, leaving vertical stacking and severe category mismatches outside its scope.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Interactive simulations of embodied AI or spatial computing applications build on realistic 3D scenes that support daily activities. However, sparse, irregular layout structures impose scene-specific physical constraints, making it hard to define a generalizable framework for generating similar functional context. We formalize Scene Retargeting as stably transferring the semantically coherent spatial organization across layouts, rather than relying on textual descriptions or pairwise relationships. Our cluster-wise transfer flexibly handles mismatched object instances and adapts to distinctive floor plans. We optimize to preserve the rich semantic context of individual clusters by respecting the spatial distribution of foundation features. We can then impose physical constraints to refine wall contacts, pairwise alignment, or clear passageways and openings. Our framework outperforms state-of-the-art methods on layout generation on the 3D-FRONT dataset, and demonstrates downstream applications including real-to-sim transfer, analogical trajectory transfer, and multi-reference composition.
1 INTRODUCTION
Scene retargeting addresses the challenge of reproducing a scene’s functional context while respecting sparse, irregular, scene-specific geometry. It transfers spatial organization from one exemplar through clusters and a three-stage framework that adapts to different floor plans and object combinations.
1 INTRODUCTION
- Scene synthesis must capture implicit scene function and complex geometric relationships despite sparse, irregular structural restrictions.Layouts need to preserve semantic context for daily interactions while remaining geometrically plausible under scene-specific constraints.Source: §1 · p. 1
- Nearby objects are grouped using spatial and semantic features, making clusters the transfer unit and preserving each group’s internal arrangement.This reduces degrees of freedom from one pose per object to one pose per group, supporting transfer despite unknown correspondences and differing object counts.Source: §1 · p. 2
- The three stages place clusters on the target floor plan, arrange objects within clusters using 3D features, then refine correspondences and physical constraints.Refinement addresses pairwise relations, wall contacts, passage clearance, and penetrations.Source: §1 · p. 2
- Scene retargeting transfers a single exemplar’s semantically coherent arrangement across varied layouts while respecting functional and geometric constraints.Unlike a learned generative prior, it uses one reference scene and can vary the floor plan and object instances while preserving similar function.Source: §1 · pp. 1–2
2 RELATED WORKS
Prior work generates layouts from learned distributions, language or visual-language inputs, or existing examples and optimization. Scene retargeting differs by transferring a single reference across room boundaries and object inventories.
2.1 DATA-DRIVEN 3D INDOOR SCENE LAYOUT GENERATION
- Data-driven layout methods range from autoregressive synthesis and denoising to scene-graph approaches and 3D diffusion, but do not faithfully reproduce a specific exemplar.Their variation comes from stochastic sampling rather than adherence to a target layout.Source: §2.1 · p. 2
2.2 LLM AND VLM-DRIVEN SCENE SYNTHESIS
- LLM- and VLM-driven methods offer intuitive control, but language captures coarse relations rather than exact metric 3D geometry.The paper instead uses pretrained 3D foundation-model features to preserve fine-grained metric relationships.Source: §2.2 · p. 3
2.3 EXEMPLAR-BASED SYNTHESIS AND LAYOUT OPTIMIZATION
- Unlike prior scene analogies that transfer trajectories between fully furnished scenes, scene retargeting arranges an empty target room across distinct room boundaries and mismatched inventories.Its analogical refinement derives optimization terms from the single reference rather than a dataset-derived prior.Source: §2.3 · p. 3
3 METHOD
Scene Retargeting transfers functional organization from a reference scene to a target floor plan through cluster placement, context-conditioned object arrangement, and parameter-free correspondence-based refinement. This hierarchy accommodates differing object inventories and room geometry while preserving spatial relations and physical validity.
3.1 OVERVIEW
- The three-stage pipeline places reference clusters on the target floor plan, arranges objects using transferred spatial context, then refines matched objects against relational and architectural constraints.The final stage also enforces physical validity, wall alignment, and clearance around openings.Source: §3.1 · p. 3, Figure 2 · p. 4
3.2 CLUSTER-LEVEL TRANSFER
- Co-located, semantically associated objects form functional clusters that are transferred as units, reducing placement to predicting each cluster’s location on the target floor plan.The placement network attends to boundary-point context and post-processes predictions for collisions, room boundaries, and opening clearances.Source: §3.2 · p. 4
3.3 OBJECT PLACEMENT VIA CROSS-SCENE FEATURE ATTENTION
- Target-object tokens cross-attend to spatial scene tokens from the transferred exemplar, conditioning pose prediction on analogous 3D configurations as well as object properties.The scene tokens are extracted from reference-object point clouds transformed into target space and combined with the target floor point cloud.Source: §3.3 · p. 5
3.4 CORRESPONDENCE AND ANALOGICAL REFINEMENT
- Within matched clusters, graph matching establishes object correspondences, then pose refinement preserves exemplar relations while adapting wall clearances and opening access to the target room.The parameter-free stage enforces physical validity through projection, clamping poses within the floor plan, resolving overlaps, and pruning low-confidence matches or unresolvable collisions.Source: §3.4 · pp. 5–6
4 EXPERIMENTS
The experiments compare Scene Retargeting with layout-generation baselines, test pipeline components and conditioning signals, and demonstrate downstream applications. The framework leads the reported metrics and supports several transfer scenarios.
4.1 IMPLEMENTATION DETAILS
- Networks learn pose recovery from perturbed ground-truth layouts and use iterative denoising at inference; retargeting takes 6.9 seconds end-to-end.Training uses mean squared error with a light L1 penalty; inference perturbs cluster centroids and target inventory poses.Source: §4.1 · p. 6
4.3 COMPARATIVE STUDIES
- Our framework achieves state-of-the-art performance across every metric, raising iRecall from 46.9 to 64.3 and PSA from 45.8 to 63.5.Figure 4 provides qualitative comparisons across room geometries.Source: Table 1 · p. 7, §4.3 · pp. 7–8, Figure 4 · p. 8
4.2 DATASETS & BASELINES
- Evaluation uses held-out 3D-FRONT living-room and bedroom pairs, with baselines receiving the reference through each method’s supported input channel.Retargeting pairs are formed by inventory similarity; baseline scores are reported for pruned and unpruned inventories, using the higher overall score.Source: §4.2 · p. 7
4.4 ABLATION STUDIES
- Ablations show that cluster placement and scene features support structural fidelity, analogical refinement improves relational alignment, and physical constraints strengthen plausibility and containment.Removing cluster placement reduces iRecall by 13.2, while refinement increases it by 7.3; removing physical constraints lowers CF by 8.6 and IB by 7.1.Source: §4.4 · pp. 8–9
4.5 APPLICATIONS
- Without fine-tuning, the framework supports human and camera trajectory transfer, real-to-sim generation from scans, and composition from multiple reference rooms.Object-attached waypoints map to target counterparts, and a single real-world capture can yield diverse synthetic environments.Source: §4.5 · p. 9, Figure 5 · p. 9
5 CONCLUSION
Scene Retargeting transfers a reference scene’s semantically coherent organization to target rooms with distinct floor plans and mismatched inventories, and the authors report stronger physical-validity and semantic-coherence metrics than existing baselines. The framework focuses on 2D floor-plane poses and comparable inventories; the authors also describe their use of AI assistance.
5 CONCLUSION
- The framework transfers a reference scene’s semantically coherent organization to rooms with distinct floor plans and mismatched object inventories, while enforcing relational fidelity, physical plausibility, and navigational affordances.It positions functional clusters, uses pretrained 3D foundation-model features for object arrangements, and adds analogical refinement.Source: §5 · p. 9
- The method outperforms existing baselines on physical-validity and semantic-coherence metrics and supports versatile downstream applications.The conclusion does not specify metric values or name the applications in this passage.Source: §5 · p. 10
- The framework is limited to 2D floor-plane poses and comparable inventories, leaving vertical stacking and severe category mismatches outside its scope.The authors identify extending to 3D structures and combining exemplar transfer with generative or retrieval-based completion as future directions.Source: §5 · p. 10
A AI USAGE STATEMENT
- AI assistance was used for grammar and sentence polishing, baseline implementation and adaptation, and debugging; the authors say they reviewed and verified the resulting text and code.The authors retain responsibility for the work’s content, claims, and implementations.Source: A AI USAGE STATEMENT · p. 13
B NETWORK ARCHITECTURE
The architecture separates placement of reference-scene clusters on a target floor plan from object-level pose prediction within the placed layout. Both learned networks use target-room geometry, while the object decoder also attends to reference-object and room point-cloud features.
B NETWORK ARCHITECTURE
- Both learned stages use Transformers to predict clean layouts from perturbed layouts, while frozen OpenShape and Concerto encoders provide features without being trained.The cluster and object networks are the only trained networks described here.Source: B NETWORK ARCHITECTURE · p. 13
- The cluster placement network represents each reference cluster with a query and attends to target-room boundary tokens and other cluster queries to predict cluster position and orientation.The room boundary is represented by 250 points carrying position and inward normal; the encoded boundary and cluster queries have width 256.Source: B NETWORK ARCHITECTURE · p. 13
- The object-level decoder predicts object poses from object tokens that combine size, class, OpenShape features, and perturbed pose with reference-scene and target-room features.The decoder receives a direct floor-plan token in addition to scene tokens formed from the reference objects and target-room floor point cloud.Source: B NETWORK ARCHITECTURE · p. 13
C TRAINING AND INFERENCE DETAILS
The learned networks are trained through perturbed-layout recovery, then used with iterative denoising and a parameter-free analogical refinement stage. Refinement transfers exemplar-derived relationships and clearances while directly enforcing room boundaries and resolving residual collisions.
C TRAINING AND INFERENCE DETAILS
- The learned networks are optimized, while clustering, graph matching, and analogical refinement are parameter-free.The optimizer settings match the reference implementation, so the authors say Table 1 comparisons do not reflect differences in training-time tuning effort.Source: C TRAINING AND INFERENCE DETAILS · pp. 13–14
- Both networks learn self-supervised layout recovery by reconstructing each room’s clean arrangement from a synthetically perturbed version, because paired retargeting ground truth is unavailable.Scene-level perturbation scales are drawn once per room and applied independently to objects, approximating transfer of a displaced arrangement.Source: C TRAINING AND INFERENCE DETAILS · p. 13
- A separate model is trained for each room type, with axis-aligned rotations augmenting each room together with its floor plan, boundary samples, and object poses.Openings remain unchanged under augmentation because they are represented as intervals on boundary edges; the boundary representation does not provide openings as learned conditioning.Source: C TRAINING AND INFERENCE DETAILS · p. 14
- Inference iteratively denoises layouts initialized from reference cluster centroids or target-inventory poses, stopping early when successive displacements remain below specified thresholds.Both stages run for at most 100 steps; the stopping thresholds are 0.01 in position and 0.005 in (cos θ, sin θ) orientation for three consecutive steps.Source: C TRAINING AND INFERENCE DETAILS · p. 15
D.1 OBJECTIVE FUNCTIONS
- The refinement loss combines relational and other exemplar-derived targets, computed once before optimization and held fixed so each term pulls the target layout toward a reference-supplied value.Orientation is represented by 2D unit vectors to avoid angle wrap-around while penalizing 180° flips.Source: §D.1 · p. 15
- Wall-clearance transfer matches an object’s reference gap to its assigned target wall, with wall matching based on cluster correspondence and wall orientation and span.Objects are assigned only to walls near their reference counterparts; corner objects can retain two walls when their normals differ by at least 60°.Source: §D.1 · p. 15
- Each opening creates a 0.75 m inward walk-through zone, and the loss penalizes object penetration into that zone using separating-axis depth.Elevated windows are considered only for objects tall enough to obstruct them; archways annotated as holes are excluded from the opening set.Source: §D.1 · pp. 15–16
D.2 PHYSICAL CONSTRAINTS
- Physical validity is enforced by clamping objects within the target floor plan and resolving verified point-cloud overlaps by translation; low-confidence matches or unresolvable collisions are pruned.Point-cloud checks leave non-overlapping objects untouched even when their oriented bounding boxes intersect, as can occur for a chair under a table.Source: §D.2 · p. 16
E EVALUATION DETAILS
The evaluation derives reference relations through a rule-based extraction pass and uses those facts both for scoring and to prepare text-conditioned baseline inputs. Its text instructions are fixed before layout generation and operate at category-level rather than instance-level.
E.1 RELATION TRIPLETS
- iRecall and PSA use a rule-based pass over the reference ground-truth layout to extract object-object and object-wall relations, with direction defined relative to object-facing frames rather than room compass directions.The relation thresholds were set on the validation split to align with human qualitative judgments.Source: §E.1 · p. 16
E.2 INSTRUCTION SYNTHESIS
- A deterministic verbalizer makes one clause per relation triplet, then a language model rewrites the clauses into a fluent two-to-four-sentence instruction reused across methods.The same cached text is shown to the judge, and it is fixed before layouts exist to avoid circularity between the specification and scores.Source: §E.2 · pp. 16–17
- The instruction avoids absolute directions, room dimensions, and instance numbers so it can be applied to differently shaped rooms and checked against renderings without identifiable instance labels.The reported compression turns geometric relations into category-level orderings without distances.Source: §E.2 · p. 17
- The instruction and iRecall evaluate category-level relations, so distinctions between individual instances of the same category are outside the evaluation’s stated scope.In the living-room example, 41 extracted facts reduce to 28 distinct category-level triplets.Source: §E.2 · p. 17
E.3 METRICS AND JUDGING PROTOCOL
The section defines relational fidelity and geometric validity measures, then explains how the paper adapts the judging protocol to use the available reference scene.
E.3 METRICS AND JUDGING PROTOCOL
- iRecall measures whether reference relations are preserved, matching category-level object pairs existentially after normalizing clear synonyms into functional groups.Triplets are compared as sets, and directional relations are evaluated independently.Source: §E.3 · pp. 17–18
- Table 4 reports pre-projection in-boundary rate for containment and iRecall for relational fidelity.Source: Table 4 · p. 18
- The judging protocol follows LayoutVLM’s geometric measures and rating rubric, but shows the reference rendering alongside the output because text instructions lose spatial detail.Source: §E.3 · p. 18
F.1 ANALYSIS
The analysis examines cluster granularity and decoder attention, then shows how the same framework relocates a furnished room to new floor plans without retraining.
F.1 ANALYSIS
- At three objects per cluster, iRecall is 56.7; pre-projection in-boundary rate falls from 93.9% at five objects to 87.9% at eleven, motivating a seven-object cluster size.Small clusters preserve too few relations; larger rigid clusters increasingly risk falling outside the floor boundary.Source: §F.1 · p. 18
- In 890 of 894 identity swaps, stage 2 attention moves toward the map produced by the newly assigned object identity, rather than position alone.The identity-preserving control leaves the attention map unchanged.Source: §F.1 · p. 18, Figure 7 · p. 19
F.2 RELOCATING A FURNISHED ROOM TO A NEW FLOOR PLAN
- For relocation, the reference supplies both arrangement and inventory while the target supplies its floor plan and openings; the framework reduces to cluster placement and analogical refinement.Stage 2 object decoding is unnecessary and stage 3 correspondence is the identity matrix.Source: §F.2 · p. 19
- Relocation produces physically valid rooms with the original object inventory across different floor plans and opening layouts.The same checkpoints and hyperparameters are used; cluster placement and analogical refinement adapt the arrangement to the new room.Source: §F.2 · pp. 19–20, Figure 8 · p. 19
F.3 QUALITATIVE RESULTS
Qualitative comparisons show preserved functional layouts and physical plausibility, while ablations and failure cases identify the roles and limits of the framework’s components.
F.3 QUALITATIVE RESULTS
- Without wall-clearance refinement, objects drift from reference wall distances; without opening refinement, they can block doorways and windows.The full model maintains wall-distance relationships and walkthrough affordances across openings.Source: §F.3 · p. 20, Figure 9 · p. 21
- Removing Stage 1 placement breaks layout structure under physical projection, removing scene features impairs spatial transfer, and removing Stage 3 leaves fine relations unadjusted.Stage 3 removal also fails to prune redundant assets.Source: §F.3 · p. 20, Figure 10 · p. 21
- Severe reference-target inventory gaps can produce awkward groupings, while highly constrained rooms can leave opening clearance, wall proximity, and collision demands unresolved.Source: §F.3 · p. 20, Figure 11 · p. 21
- Against InstructScene, LayoutVLM, and LaviGen, the framework preserves functional groups and spatial relations while fitting target boundaries without physical violations.Source: §F.3 · p. 20