HARMONY: Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis

1 INTRODUCTION

• Recovering a 3D scene from one image is ill-posed because the view captures partial geometry and occlusions, while object scales and spatial relationships lack explicit 2D cues.The problem has applications including AR/VR content creation, embodied AI, robotic navigation, and interactive scene editing.Source: §1 · pp. 1–2
• Prior methods relying on low-level perception struggle with occluded or small objects, while language-only and image-conditioned VLM approaches face limits in scene complexity or precise geometric placement.Text-conditioned methods may be restricted to simple, template-like arrangements; image-conditioned approaches can retain noticeable visual mismatch.Source: §1 · p. 2
• HARMONY hierarchically grounds the scene, places wall-mounted items, furniture, and decorations, refines positions using geometry signals, and checks partial-scene renderings after each stage.Furniture placement uses depth-first traversal so each new candidate appears in a clean, unoccluded view of the partial scene.Source: §1 · p. 2
• HARMONY achieves state-of-the-art performance on synthetic and challenging real-world inputs, with qualitative comparisons indicating more faithful arrangements and better scene-detail preservation than GPT-6 Astra.The authors attribute this to combining VLM reasoning about spatial and semantic relationships with point-cloud-based position adjustment.Source: §1 · p. 2

2 RELATED WORK

• Object-reconstruction models advance geometry and appearance, while HARMONY integrates such reconstruction with hierarchical reasoning and geometry-grounded refinement to assemble scenes aligned with the image.The cited examples include latent diffusion systems and human-in-the-loop reconstruction for heavily occluded in-the-wild images.Source: §2 · p. 3
• Geometry-grounded scene pipelines combine point-cloud estimation, segmentation, inpainting, object reconstruction, and layout optimization, while whole-scene generators can miss semantic spatial relations such as how a chair faces a desk.The passage also describes methods using scene parsing, pose optimization, physical correction, and joint diffusion of object shapes and poses.Source: §2 · p. 3
• Text-driven agentic methods plan hierarchical placements from scene descriptions, but often rely on asset retrieval and hand-designed priors because text lacks accurate 3D positions and orientations.SceneSmith is given as an example of floorplan initialization followed by hierarchical placement reasoning.Source: §2 · p. 3
• Image-conditioned VLMs improve grounding, but VLMs may not reason precisely about numerical quantities, and small-object point clouds can be incomplete or noisy.HARMONY uses hierarchical reasoning and checks at each stage to reduce error accumulation.Source: §2 · pp. 3–4

3 METHOD

• HARMONY integrates relational and spatial reasoning, 2D and 3D generation, and visual geometry-grounded models into a unified reconstruction pipeline.Its VLM-led stages begin with camera alignment, proceed through object reconstruction and hierarchical placement, and end with appearance estimation.Source: §3 · p. 4

3.1 3D ROOM LAYOUT AND CAMERA INITIALIZATION

• The room-layout stage combines VLM semantic understanding with VGGT geometric corner detection to align an empty room and camera pose with the reference image.The VLM estimates room dimensions, identifies a deep visible corner, and labels walls; VGGT supplies a Manhattan frame and corresponding room geometry.Source: §3.1 · p. 4, §3.1.1 · pp. 4–5, §3.1.2 · p. 5

3.1.1 VLM SEMANTIC INITIALIZATION.

• VLM estimates room dimensions, anchors the deepest visible corner, and labels visible walls to initialize a semantically aligned empty room box.If only walls are visible, the farthest wall endpoint serves as the anchor.Source: §3.1.1 · pp. 4–5

3.1.2 VGGT GEOMETRIC REFINEMENT.

• VGGT surface normals define a Manhattan frame whose planes and deepest floor corner anchor the room geometry to the camera view.The frame provides width, vertical, and depth axes, and identifies the floor and ceiling endpoints of the anchor corner.Source: §3.1.2 · p. 5

3.1.3 CAMERA CALIBRATION.

• A similarity transform re-expresses the VGGT camera in the canonical room frame by aligning rotation, scale, and the floor anchor.Metric scale uses the estimated room height, while translation places the floor anchor at its corresponding canonical corner.Source: §3.1.3 · p. 5

3.2 OBJECT SEGMENTATION AND RECONSTRUCTION

• Objects are detected by category and hierarchy, occluded regions are inpainted, and image-to-3D meshes are canonicalized and orientation-labeled.The hierarchy covers wall-mounted items, free-standing furniture and ceiling objects, then decorations; VLM checks inpainted objects against the reference.Source: §3.2 · pp. 5–6

3.3 HIERARCHICAL SCENE RECONSTRUCTION

• HARMONY places wall-mounted items, furniture, and decorations in stages, using depth-first ordering, relational prompts, geometry refinement, and reflective corrections.The VLM places objects from the deepest room corner outward; silhouette alignment refines lateral position and scale-to-depth ratio, while point clouds refine depth and metric scale.Source: §3.3 · pp. 6–7

3.4 LIGHTING

• For photorealistic rendering, the VLM infers per-object PBR materials and estimates emissive sources’ activation, color, and intensity, then refines them against the reference render.Material choices use per-category priors, and the reflective comparison uses Blender Cycles.Source: §3.4 · p. 7

4.1 EXPERIMENT PROTOCOL

• The evaluation uses 100 Front3D images with 3D ground truth, 30 real-world HARMONY30 images, and a broader 300-scene HARMONY300 benchmark; rendering metrics include N-CLIP, PL, and LPIPS.Front3D also uses Chamfer Distance and F-scores; the user study has 16 participants ranking HARMONY against five baselines on 20 scenes.Source: §4.1 · pp. 7–8

4.3 QUALITATIVE RESULTS

• Across Figure 3 comparisons, HARMONY's semantic reasoning and local geometry refinement yield more faithful poses and coherent arrangements than baselines with geometry, orientation, or segmentation failures.The comparisons describe merged nearby objects and degraded appearance for Gen3DSR, inaccurate placements and missing decorations for 3D-ReGen, and segmentation-related robustness problems for SAM3D and CAST.Source: Figure 3 · p. 8, §4.3 · p. 9

4.2 QUANTITATIVE RESULTS

• On Front3D, HARMONY achieves the best score on every reported rendering- and geometry-quality metric; GPT-6 Astra has larger geometric errors.The authors attribute the results to VLM object planning and placement grounded in dense metric image evidence.Source: §4.2 · p. 8, Table 1 · p. 9
• On real-world HARMONY30 images, HARMONY achieves the best N-CLIP and PL scores.The authors say the results indicate better preservation of semantic content and perceptual structure, including for complex real-world inputs.Source: §4.2 · p. 8, Table 2 · p. 9
• In the user study, HARMONY ranks first in 67.0% of choices, ranks in the top two in 86.0%, and has a mean rank of 1.54.Sixteen participants ranked the method against five baselines across 20 scenes.Source: §4.2 · p. 8, §4.1 · p. 8, Table 2 · p. 9

4.4 ABLATION STUDY

• Ablations show that silhouette refinement and camera calibration matter most for perceptual similarity, while removing VLM reasoning or structured placement harms semantic fidelity or object relationships.The study separately ablates VLM reasoning, placement order and refinement, as well as depth-first furniture traversal.Source: §4.4 · p. 9, Table 3 · p. 9

4.5 FAILURE CASES

• Rare-type vocabulary mismatch leaves 25.9% of detections with low grounding confidence (<0.35), though genuine placement failures are 0.3% of objects; foreground-clipped objects can have underconstrained meshes.Figure 4 also shows an object present in the input and top-down reconstruction but occluded in the rendered view because it sits against a wall hidden from the render camera.Source: §4.5 · pp. 9–10, Figure 4 · p. 10

A OVERVIEW

• HARMONY reconstructs compositional 3D scenes from monocular indoor images through hierarchical reasoning and geometry-grounded refinement.The framework proceeds from floorplan and camera recovery through object placement and lighting estimation, with local verifiers intended to catch and repair errors before they propagate.Source: §5 · p. 10, A OVERVIEW · pp. 14–15

5 CONCLUSION AND FUTURE WORK

• Future work can extend HARMONY to multi-view images and infer object articulations.The overview identifies foundation models in Table 4; its caption says prompted VLM and image-editor models can be served locally or through hosted APIs without changing the pipeline.Source: §5 · p. 10, Table 4 · p. 15

A OVERVIEW

• The pipeline places wall-mounted objects, furniture, ceiling objects, and decorations in stages, then estimates lighting.Objects are rebuilt independently through common roles, with object-level and geometric verifiers gating stages.Source: A OVERVIEW · pp. 14–15

B.1 FOUNDATION MODELS

• HARMONY composes off-the-shelf models without training or fine-tuning; one VLM handles reasoning, one image-editing model handles appearance synthesis, and an image-to-3D model generates meshes.The prompted VLM and image editor can be served locally or through hosted APIs without changing the pipeline.Source: §B.1 · p. 15

B.2 CAMERA CALIBRATION PARAMETERS

• Camera calibration holds the row-band fraction at ρ = 0.2, the normal-alignment threshold at τ = 0.7, and extension percentiles at pfc = 98 and pwall = 97.The procedure estimates Manhattan axes and bounding planes from the VGGT point cloud, then solves for a transform that pins a room corner to its canonical counterpart.Source: §B.2 · p. 15, §B.4 · p. 16

B.3 IMAGE-TO-3D SETTINGS

• Hunyuan3D-2 meshes use octree resolution 384, 25 shape-diffusion steps, and an 80,000-face target.The paper reports these settings reduce fragmented thin structures relative to fast-preview defaults of 128 / 5 / 40,000.Source: §B.3 · p. 15

B.4 HARDWARE AND RUNTIME

• A scene takes 20–90 minutes on an NVIDIA L40 GPU, averaging 3.4 VLM calls per placed object.Camera solving estimates a Manhattan frame, refines six bounding planes, and computes a rigid-body similarity transform that pins the deepest floor/ceiling corner to the canonical wall corner.Source: §B.4 · pp. 15–16

B.5 RENDERING

• HARMONY uses pyrender with flat ambient light and baked textures before relighting with Blender Cycles.The exported meshes are renderable with any renderer.Source: §B.5 · p. 16

B.6 BASELINE SETUP

• Baselines receive the same monocular image and are rendered from their recovered cameras; mesh-only methods use HARMONY’s flat-ambient pyrender setup.VIGA and 3D-RE-GEN require documented non-default changes, while SAM3D receives HARMONY segmentation masks and CAST uses the best available unofficial implementation.Source: §B.6 · p. 16

C.1 GEOMETRY PREPROCESSING

• For planar wall objects, HARMONY rectifies the segmented crop before meshing, while other objects are meshed from the raw crop.Layout recovery uses VLM-inferred room dimensions, walls, openings, and camera pose, then refines the camera against an empty-room render; canonicalization bakes a VLM-selected front-face yaw into each mesh.Source: §C.1 · p. 17, §C.2 · p. 18

C.2 WALL-MOUNTED OBJECT PLACEMENT

• Wall-mounted object positions come from casting each mask centroid ray onto room wall planes, with VLM tie-breaking near corner seams.Size is estimated from mask extent, focal length, and camera-space depth, with foreshortening correction and category-based aspect-ratio clamping.Source: §C.2 · p. 19

VLM Detection Prompt

• Wall-object detections are filtered for containment and duplicates, then VLM-verified to remove fragments and false positives.The pipeline rejects candidates before their masks are written to disk.Source: VLM Detection Prompt · p. 19, Figure 9 · p. 20

C.3 FURNITURE PLACEMENT

• Furniture placement uses VLM-assigned wall affinity, facing, functional groups, support relations, and dependency-aware order, then refines meshes against reference silhouettes and predicted depth.Post-placement group refinement can align chair orientations toward a shared table, unify member scales, and swap visually distinct members to match the reference arrangement.Source: Detection Prompt · p. 21

Detection Prompt

• The furniture layout prompt orders deepest-corner and wall-affine objects before nearer or centre objects, while placing dependent objects after their anchors.It also asks the VLM to verify objects and assign relations such as on-top-of, in-front-of, facing-toward, and behind.Source: Detection Prompt · p. 21

C.4 CEILING OBJECT PLACEMENT

• Ceiling fixtures are detected, completed, meshed, and hung from the ceiling plane; decorations are identified by comparing reference furniture crops with clean reconstructed furniture.Further checks verify the supporting furniture and decoration match; orientation correction compares four yaw renders, and matched groups can be reordered by colour and shape.Source: §C.4 · pp. 21–22

C.5 MATERIAL AND LIGHTING ESTIMATION

• The VLM assigns each generated object a dominant material and PBR factors, then estimates which light sources emit and their color and intensity.A reflective lighting pass compares the current lit preview with the reference and adjusts global or individual light settings until it converges.Source: §C.5 · pp. 23–24

D HARMONY300 DATASET

• HARMONY300 contains 300 single-image indoor scenes across three difficulty levels, each paired with a reconstructed scene and physically lit render.The dataset extends the curated real-image set described in the main paper.Source: D HARMONY300 DATASET · p. 24

D.1 SPLITS

• The easy split contains 100 synthetic 3D-FRONT scenes with ground truth, while the medium and complicated splits each contain 100 real photos.The complicated split spans varied room types and cluttered, multi-object layouts.Source: §D.1 · p. 24

D.2 PER-SCENE CONTENTS

• Each scene includes a reference photo, render, reconstructed scene, calibrated camera, segmentation colormap, and object-level relation annotations.The annotations include object size, world position, wall affinity, and relations such as facing, in-front-of, on-top-of, and grouped-with.Source: §D.2 · p. 25

D.3 SOURCING AND LICENSING

• The dataset draws 196 scenes from Pexels, two from Unsplash, 100 from 3D-FRONT, and two synthetic renders from 3D-FUTURE.It is released under CC-BY-NC-4.0; the 3D-FRONT and 3D-FUTURE scenes are research-only and require citation.Source: §D.3 · p. 25

E USER STUDY

• Sixteen uncompensated volunteers ranked HARMONY and five baselines on 20 scenes, assigning each reconstruction a unique perceptual-match rank from 1 to 6.The scenes comprise 10 easy Front3D cases and 10 real-life medium or hard cases; panel-to-method mappings were reshuffled for each participant and scene.Source: E USER STUDY · p. 25

• The qualitative results cover all three HARMONY300 splits and include an alternative viewpoint and multi-room layout generalization.Source: F ADDITIONAL RESULTS · p. 25
• The participant study presents an input photo alongside six candidate reconstructions, including HARMONY and five baselines.Source: Figure 13 · p. 26
• In the live study interface, candidates are anonymized as A–F and reshuffled for each participant and scene.Source: Figure 13 · p. 26

• The easy split contains scenes rendered from 3D-Front.Source: §F.1 · p. 26
• The examples span diverse layouts, furniture styles, wall colours, and lighting conditions.Source: §F.1 · p. 26

F.2 DIVERSE ROOM TYPES

• Reconstructions in the medium and complicated splits span real photos of bedrooms, living rooms, offices, kitchens, and bathrooms, including cluttered layouts.Source: §F.2 · p. 26, Figure 15 · p. 28

F.3 CANONICAL VIEWS

• Canonical-view renders expose the full room layout and make furniture arrangement and geometry comparable across scenes independent of photo composition.Source: §F.3 · p. 27

F.4 MULTI-ROOM LAYOUT

• Refining camera poses relative to a canonical layout lets HARMONY adapt to diverse room configurations and multi-room environments while maintaining consistent spatial relationships.Source: §F.4 · p. 27

F.5 APPLICATIONS

• The downstream demonstrations test passive physical plausibility and contact-rich interaction by an embodied agent.Source: §F.5 · p. 27

• Earthquake simulations treat furniture and decorations as dynamic rigid bodies moved by gravity and friction using VLM-estimated mass and friction.A tall bookcase topples while a heavy sofa barely shifts.Source: §F.5.1 · p. 28, Figure 18 · p. 30
• A canonical elevated render removes the ceiling and near wall to reveal the full room layout.Source: Figure 16 · p. 29
• A Unitree H1 performs a 230-frame chair-moving and sitting sequence, with contacts resolved against reconstructed geometry.The described contacts include hands on the backrest, thighs on the seat pan, and feet on the floor.Source: Figure 19 · p. 30

F.6 ROBOT INTERACTION

• A Unitree H1 relocates the office chair, sits, and reaches the workstation, while a Franka Panda grasps the chair backrest and pulls it back.Contact targets come from the reconstructed geometry rather than hand-placed proxies.Source: §F.6 · p. 30
• The scripted robot sequences test whether reconstructed surfaces support contact-rich interaction, not whether a control policy can perform the tasks.The scene is imported into Isaac Sim, and the sequences are kinematically scripted.Source: §F.6 · p. 30
• The Panda grasps the chair backrest at 0.868 m, where its 68 mm thickness fits the 80 mm jaw; 35 mm lower, it thickens to 87 mm and cannot be grasped.Lula IK solved end-effector poses without failures across 140 frames during a 0.45 m pull.Source: Figure 20 · p. 31