PaperlayerView exact revision on arXiv

Source-linked AI summary

Easy-3Dfy: Object-Level Geometric Composition for Single-Image 3D Scene Construction

Ziang Zhang, Wenyou Yi, Zehan Wang, Jiachen Xu, Xintong Han, Chunchao Guo, Zhou Zhao

arXiv:2610.32250v1cs.CVcs.AI

TL;DR

Single-image 3D scene construction must produce complete object assets and place them in a common scene coordinate system, while joint pipelines can struggle with fine geometry and asset quality. Easy-3Dfy decouples object generation from layout estimation and uses an RPF aligner to connect object-local meshes with partial scene pointmaps. It reports consistent geometric gains over SAM-3Dfy across two benchmarks and can benefit from upstream object-generation improvements.

  • Problem

    Joint scene-construction pipelines can limit fine-grained asset quality and geometric layout, despite specialized generators and spatial models offering complementary capabilities.

    Source: §1 · pp. 1–2

  • Method

    Easy-3Dfy decouples object generation from scene reconstruction and uses an RPF aligner to establish correspondences and transforms between object-local assets and partial scene point clouds.

    Source: §1 · p. 2

  • Results

    Easy-3Dfy consistently improves multiple scene-level geometric metrics over SAM-3Dfy on Scene-Maker and 3D-FUTURE200.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The decoupled workflow can benefit independently from advances in upstream object generation.

    Source: §1 · p. 2

    Geometry-centric metrics do not directly measure appearance, texture and material quality, semantic identity, scene completeness, or perceptual plausibility, and are limited for topology and fine surface structure.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Single-image 3D scene construction is a fundamental problem in 3D vision. Recent methods such as SAM-3Dfy enable end-to-end reconstruction of 3D assets and their scene layouts, but still struggle with fine-grained scene geometry and asset quality. Meanwhile, specialized 3D object generators provide detailed object geometry, while 3D foundation models offer strong and generalizable spatial understanding. We propose Easy-3Dfy, a compositional workflow that decouples object generation from scene layout estimation, allowing specialized models to fully exploit their respective strengths. An RPF-based aligner then bridges these capabilities by establishing geometric correspondence between object-local meshes and partial scene pointmaps, recovering object layout for scene composition. Without requiring complete scene-level supervision, the proposed workflow consistently outperforms SAM-3Dfy across multiple geometric metrics on Scene-Maker and 3D-FUTURE200. Our results demonstrate that explicit geometric composition can effectively combine high-quality object generation with strong spatial understanding, while enabling the workflow to benefit independently from advances in both 3D object generation and foundation models.

1 INTRODUCTION

Single-image 3D scene construction must recover complete object assets and arrange them in a common metric coordinate system, but joint scene pipelines can limit asset detail and geometric alignment. Easy-3Dfy separates object generation from scene layout estimation and uses geometric alignment to compose them. It improves multiple scene-level geometric metrics over SAM-3Dfy and can benefit from stronger upstream object generators.

Problem

  • Single-image 3D scene construction seeks complete 3D object assets arranged in a common metric coordinate system, while joint pipelines can limit asset detail and layout quality.Scene-oriented systems may produce less detailed objects, and the lack of explicit geometric alignment can yield interpenetrating or floating layouts.Source: §1 · p. 1

Results

  • Easy-3Dfy consistently improves multiple scene-level geometric metrics over SAM-3Dfy on Scene-Maker and 3D-FUTURE200.With the same object assets, it improves asset composition and scene layout; replacing object generators also shows benefits from upstream improvements.Source: §1 · p. 2

Approach

  • Easy-3Dfy decouples object generation from scene reconstruction and uses an RPF aligner to establish correspondences between object assets and partial scene point clouds.The aligner estimates the relative transform, connecting object-local geometry to the scene coordinate system.Source: §1 · p. 2

2 RELATED WORK

Related work supplies complementary object-local assets, scene geometry, and layout strategies, but these are often coupled or rely on scene-level supervision. Easy-3Dfy instead composes reusable asset and geometry priors through point-cloud alignment.

2.1 SINGLE-OBJECT IMAGE-TO-3D GENERATION

  • Image-to-3D generators provide object-local geometry, while Easy-3Dfy uses these models as upstream providers and estimates placement separately from scene geometry.It discards providers’ pose or layout outputs rather than claiming to improve their generated assets.Source: §2.1 · p. 3

2.2 MONOCULAR GEOMETRY ESTIMATION AND POINT-CLOUD ALIGNMENT

  • Monocular geometry models provide visible scene geometry, but partial visibility and depth uncertainty leave object layout undefined; Easy-3Dfy trains an RPF aligner for partial-to-complete geometry matching.Its object-level rendered observations and layout augmentations provide alignment supervision without complete calibrated scenes.Source: §2.2 · p. 3

2.3 SINGLE-IMAGE 3D SCENE GENERATION AND LAYOUT PREDICTION PARADIGMS

  • Scene-generation methods range from joint end-to-end systems to post-hoc assembly, while Easy-3Dfy learns local-to-scene composition in the point-cloud domain.Unlike Scene-Maker, which still relies on scene-level supervision for layout, Easy-3Dfy separates asset generation from geometric alignment.Source: §2.3 · p. 3

3 METHOD

Easy-3Dfy represents scene construction as modular composition: it creates object-local assets, predicts point correspondences to partial scene geometry, and estimates transforms to assemble the meshes. Its aligner is trained from object-level data rather than complete scenes.

3.1 PROBLEM FORMULATION

  • For each object, Easy-3Dfy predicts a pixel-aligned scene pointmap and normals, selects the visible region, and estimates a uniform-scale similarity transform for its complete local mesh.Explicit interfaces for object meshes, scene pointmaps, and transforms allow the composition modules to be replaced independently.Source: §3.1 · pp. 3–4

3.2 CONSTRUCTING OBJECT ASSETS FROM A SINGLE IMAGE

  • Object assets are built by masking and cropping the object, completing occluded regions with an image editor, then generating a complete object-local mesh from the completed image.Mask estimation is outside the work’s scope; benchmark masks are shared inputs for consistent comparison.Source: §3.2 · p. 4

3.3 LAYOUT ESTIMATION AND SCENE ASSEMBLY

  • RPF predicts anchor-coordinate correspondences for partial-query points using XYZ and surface normals, then Umeyama fitting recovers the local-to-scene transform applied to each complete mesh.The aligner excludes RGB and texture, using geometry-only registration to reduce sensitivity to appearance variation.Source: §3.3 · pp. 4–5

3.4 TRAINING THE RPF-BASED ALIGNER

  • RPF training pairs combine mesh-sampled anchors with rendered RGB–depth views, monocular partial queries, pixel-matched correspondence targets, and independent geometric augmentations.Random occlusions simulate incomplete inference geometry; independent rotations, scales, and translations expose the model to coordinate-frame and pose changes.Source: §3.4 · p. 5, Figure 3 · p. 5
  • Object-level registration training requires neither complete scene-level data nor explicit inter-object relationships, simplifying training-pair synthesis and enabling efficient dataset scaling.Source: §3.4 · pp. 5–6

3.5 IMPLEMENTATION SUMMARY

  • The implementation uses interchangeable monocular geometry and object-asset providers, while mesh providers supply only object-local assets and not pose or layout predictions.MogeV2 supplies partial geometry for RPF training, and the appendix reports detailed implementation and inference settings.Source: §3.5 · p. 6

4 EXPERIMENTS

Experiments compare Easy-3Dfy with prior scene-construction methods on Scene-Maker and 3D-FUTURE200, then ablate alignment design, scene geometry and completion models, and mesh providers. Easy-3Dfy improves geometric metrics across datasets, while ablations show that some component effects depend on the metric or dataset.

4.1 EXPERIMENTAL SETUP

  • Evaluation uses 1,000 Scene-Maker scenes and 200 3D-FUTURE test scenes, with dataset-provided masks and scene- and object-level CD and F-score metrics.Scene-Maker spans broader open-set categories, while 3D-FUTURE200 primarily contains indoor furniture scenes.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • Across both datasets, Easy-3Dfy beats MIDI and SceneGen on all reported metrics and, with SAM-3Dfy meshes fixed, improves Scene-CD, Scene-FScore, and BBox IoU.The shared meshes give identical Object-CD and Object-FScore, isolating layout prediction in the comparison with SAM-3Dfy.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 ABLATIONS

  • Removing surface-normal conditioning degrades all three layout metrics, while omitting point-cloud cleaning has only a small effect, mainly a slight Scene-FScore decrease.The authors attribute the limited cleaning effect to RPF's training on augmented data and tolerance of moderate noise.Source: §4.3 · p. 7, Table 3 · p. 8
  • Completion-model effects vary by dataset and metric: GPT-Image-2 slightly improves 3D-FUTURE200 scene-level metrics but does not consistently improve object-level metrics or Scene-Maker results.The comparison uses Meshy-T2 and six-channel XYZ–normal RPF input.Source: §4.3 · pp. 7–8, Table 2 · p. 8
  • More advanced mesh providers offer finer surface geometry and more accurate depth extent, providing stronger correspondence cues for subsequent alignment.Figure 4 pairs full-scene views, which show scale, extent, and placement, with close-ups of the same asset regions.Source: Figure 4 · p. 7, §4.3 · p. 8

5 ANALYSIS AND METRIC INTERPRETATION

The analysis finds that direct object alignment improves spatial relationships, attainable layout quality depends on asset geometry matching the observation, and geometry-focused metrics omit important aspects of scene quality.

5.1 WHAT DOES DIRECT ALIGNMENT IMPROVE?

  • Easy-3Dfy improves spatial consistency over SAM-3Dfy’s layout, while MIDI and SceneGen often distort relative orientation, ordering, and spacing in open-set scenes.Small or severely occluded objects, contaminated masks, and unreliable pointmaps can leave the transform underconstrained and produce inaccurate poses.Source: §5.1 · pp. 8–9

5.2 GEOMETRIC COMPATIBILITY ENHANCES ATTAINABLE LAYOUT QUALITY

  • Alignment works across tested mesh providers, but layout quality remains sensitive to dimensions and proportions that match the partial observation.Table 4 fixes the pointmap, mask, RPF model, and inference setting; dataset ground-truth meshes provide an upper-bound reference.Source: §5.2 · p. 9, Table 4 · p. 9

5.3 WHAT GEOMETRY-CENTRIC LAYOUT METRICS OMIT

  • Geometry-centric metrics do not directly measure appearance, texture and material quality, semantic identity, scene completeness, or perceptual plausibility, and have limited sensitivity to topology and fine surface structure.A better layout score can coexist with blurred textures or local details, while a visually detailed asset can remain geometrically incompatible with the scene.Source: §5.3 · p. 9

6 CONCLUSION

Easy-3Dfy composes replaceable object assets, scene geometry, and metric layout alignment, using an RPF aligner trained on scalable object-level data to recover local-to-scene transforms. Experiments report consistent layout gains under a shared-mask protocol and applicability across mesh providers, while highlighting limits of geometry-centric metrics.

6 CONCLUSION

  • Easy-3Dfy formulates single-image scene construction as object-level geometric composition and uses an RPF aligner trained on scalable object-level data to recover local-to-scene transforms.Experiments report consistent layout gains under the shared-mask protocol and applicability across mesh providers; the conclusion also motivates more complete evaluation protocols.Source: §6 · p. 9

AI USE STATEMENT

The authors say human researchers developed the research ideas, problem formulation, and experimental design; AI-assisted programming supported code development, and language models polished the manuscript.

AI USE STATEMENT

  • Human researchers developed the research ideas, problem formulation, and experimental design; AI tools supported coding and manuscript language polishing, with authors reviewing and verifying the final work.The authors state they take full responsibility for the final work.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The authors report no identifiable ethical, security, copyright, or privacy concerns in the datasets and visual materials used, and state that resources were used for research under applicable licenses and terms.

ETHICS STATEMENT

  • The authors report no identifiable ethical, security, copyright, or privacy concerns in the datasets and visual materials, which they used for research under applicable licenses and terms.Source: ETHICS STATEMENT · p. 10

A.1 ADDITIONAL IMPLEMENTATION DETAILS

The completion prompt combines a cropped scene region, an aligned target mask, and the full scene image to guide object restoration. It specifies structural completion, preservation of intrinsic attributes, and isolation under neutral lighting.

A.1 ADDITIONAL IMPLEMENTATION DETAILS

  • The prompt uses the original crop, crop-aligned target mask, and complete scene image as references for extracting and restoring the target.The mask marks target regions in white and non-target regions in black; the full image supplies global context and a reference for the target’s appearance.Source: §A.1 · p. 13
  • The completion instructions remove occluders and reconstruct the target’s complete structure while preserving its material, texture, color, and shape.They also require a centered object on a pure black background with complex environmental lighting removed.Source: §A.1 · p. 13
  • The prompt forbids foreground occluders, broken or truncated structures, implausible perspective distortion, blurred textures, and extraneous objects or overlays.It also excludes complex backgrounds and environmental light spots.Source: §A.1 · p. 13

A.2 RPF TRAINING-PAIR CONSTRUCTION AND INFERENCE DETAILS

These supplementary sections describe RPF training and transform fitting, evaluation alignment, ablations, visualizations, runtime, and failure cases. They report that removing RGB improves layout metrics and identify degraded partial geometry as a boundary for reliable layout estimation.

A.2 RPF TRAINING-PAIR CONSTRUCTION AND INFERENCE DETAILS

  • RPF training uses about 21,000 × 40 asset instances with 20 rendered views each, random occlusion and geometric augmentation, and 50-step training and inference.Training uses batch size 128 on two nodes with eight NVIDIA A100 GPUs per node.Source: §A.2 · p. 13
  • The aligner fits a uniform-scale transform from predicted correspondences using arithmetic centroids and unweighted cross-covariance, then enforces a proper rotation.If the scale denominator is at most 10^-12, the scale is set to 1; post-fit errors use Euclidean residuals.Source: §A.2 · pp. 13–14

A.4 COMPLETE EXPERIMENTS RESULTS

  • Tables 5 and 6 cover alignment-design and scene-geometry/object-completion ablations; Table 7 compares mesh providers, and Table 8 tests RGB conditioning.Table 7 includes GT as an upper-bound reference under the current layout workflow; Tables 5 and 6 provide complete two-dataset versions of ablations summarized in the main text.Source: Table 5 · p. 14, Table 6 · p. 14, §A.4 · p. 14

A.3 EVALUATION-TIME ALIGNMENT AND METRIC PROTOCOL

  • Evaluation applies one global bbox-Chamfer alignment per scene, while object CD and F-score use independent object-local normalization and rigid registration.For failed scenes or objects, CD is omitted from its aggregation, whereas F-score and bounding-box IoU receive zero and remain in the average.Source: §A.3 · p. 14

A.4 COMPLETE EXPERIMENTS RESULTS

  • Removing RGB improves all reported layout metrics on both datasets under the evaluated setting.The RGB-conditioning comparison fixes FLUX-2-pro completion and Meshy-T2 assets; its 9-channel checkpoint is retrained with RGB features.Source: §A.4 · p. 14, Table 8 · p. 15

A.5 ADDITIONAL VISUALIZATION RESULTS

  • Additional visualizations show 3D-FUTURE results and failure-mode comparisons under geometrically challenging conditions.Figure A1 provides additional 3D-FUTURE visualizations, while Figure A2 compares failures under challenging geometry.Source: Figure A1 · p. 15, §A.5 · p. 15, Figure A2 · p. 16

A.6 INFERENCE EFFICIENCY

  • Inference time depends mainly on module runtimes and parallelism, while batched parallel mesh registration by the RPF aligner takes a few seconds.The passage gives examples ranging from under 10 seconds for Meshy-T2 geometry generation to about one minute for Hunyuan3D-V3.1 textured assets.Source: §A.6 · p. 16

A.7 REPRESENTATIVE FAILURE CASES

  • Severely degraded or insufficient object-specific partial geometry can leave layout estimation underconstrained or cause failure.Tiny objects, severe occlusions, motion or defocus blur, and unreliable depth maps are cited as sources of information loss; Easy-3Dfy can show rotation instability when constraints are severely underdetermined.Source: §A.7 · p. 16

Related papers

Loading 2610.32250v1…