PaperlayerView exact revision on arXiv

Source-linked AI summary

ObjectSplat: Improving Mesh Fidelity and Interactivity for 3D Scenes via Object-Level Mesh Splatting

Minhas Kamal, Hiranya Garbha Kumar, Mahedi Kamal, Balakrishnan Prabhakaran

arXiv:2608.30423v1cs.CVcs.AIcs.LG

TL;DR

Holistic splatting lacks object-level structure and can contaminate unobserved regions, limiting editing, interaction, and reconstruction quality. ObjectSplat decomposes posed frames into objects and background before independently reconstructing them with mesh splatting, then composes the components. The approach improves mesh fidelity and novel-view synthesis while enabling object-wise scene manipulation, although current cross-view consistency and evaluation limitations constrain completion-based claims.

  • Problem

    Holistic splatting lacks object-level structure for downstream editing and can fit contaminated geometry in regions never directly observed.

  • Method

    ObjectSplat segments each frame into objects and background, inpaints the background, reconstructs components independently with mesh splatting, and composes them.

  • Results

    Object-level decomposition improves mesh fidelity and novel-view synthesis over vanilla mesh splatting while producing an editable, object-centric scene.

  • Takeaways & Limitations

    The resulting separable, mesh-exportable representation supports direct downstream editing and interactive object movement, removal, and rearrangement.

  • Takeaways & Limitations

    Cross-view inconsistency in segmentation, association, completion, and inpainting, together with incomplete ground-truth data, limits current evaluation and completion claims.

Abstract

from arXiv · show

Splatting-based algorithms reconstruct photorealistic, real-time-renderable, and mesh-exportable 3D scenes from regular images, but they represent a scene as a single monolithic field. Therefore, the reconstruction has no object-level structure, leaving it infeasible for downstream editing or interaction. Moreover, regions that are never directly observed in the input scans are contaminated by the surrounding texture and left uncorrected, capping both mesh fidelity and novel-view synthesis. We propose a decompose-before-reconstruct approach: we segment the instances out of every frame, consider the remaining as background and inpaint it, reconstruct each instance and the background independently with mesh splatting, and compose them into a single scene. Our method significantly improves mesh fidelity (over a 5\% gain in F-score) and novel-view synthesis, while supporting object-wise modifiability and interactivity. The code will be made publicly available.

1. Introduction

Existing splatting methods reconstruct scenes as unstructured holistic fields, limiting object-level interaction and leaving unobserved regions vulnerable to contaminated geometry. ObjectSplat decomposes scenes before reconstruction and reports improved fidelity, novel-view synthesis, and editable object-centric representations.

  • Motivation: Holistic splatting represents each scene as one unstructured field, making direct object selection, movement, duplication, or removal infeasible.Moving an object can expose unobserved surfaces and leave corrupted geometry or holes.
  • Motivation: Unobserved surfaces in holistic reconstruction receive no corrective gradient, so surrounding textures can produce stray, miscolored primitives and degraded geometry.Masking other components lets optimization fit only the object itself.
  • Interactivity: The method supports object manipulation, including scaling, appearance editing, duplication, and rigid transformation, with inpainted backgrounds preventing holes after relocation.These capabilities are demonstrated qualitatively rather than established as quantitative evaluation outcomes.
  • Approach: ObjectSplat decomposes posed frames into objects and background, reconstructs components independently with mesh splatting, and composes them into one scene.The method follows a decompose-before-reconstruct principle and uses opacity-based pruning of extraneous geometry.
  • Results: Per-object decomposition improves mesh fidelity and novel-view synthesis over vanilla mesh splatting while producing an editable, object-centric scene.Per-object optimization converges in under half the iterations required by holistic reconstruction.
  • Limitations: Current decomposition-based reconstruction is constrained by cross-view consistency limits in segmentation, association, completion, and inpainting, plus incomplete evaluation datasets and metrics.The authors identify these issues as bottlenecks for future progress.

2. Related Works

Prior work separately addresses mesh fidelity, object semantics, inpainting, amodal completion, or sparse-view reconstruction, but generally retains a holistic scene representation. ObjectSplat closes this combined gap by decomposing observations first and independently reconstructing object and background mesh splats.

  • Mesh and object structure: Mesh-splatting methods produce explicit, engine-ready meshes, but existing radiance-field approaches generally reconstruct scenes as single monolithic fields without object structure.This separates mesh exportability from object-level editability.
  • Object semantics: Post-reconstruction semantic methods attach object identities to primitives, but shared primitives crossing boundaries can make editing one object perturb neighboring objects.Some approaches also expose background regions that must be filled at edit time.
  • Completion: Inpainting and amodal-completion methods address revealed or occluded regions, but current amodal completions can be inconsistent across views and degrade per-object optimization.The paper therefore treats object completion as promising but not yet reliable.
  • Sparse-view reconstruction: Sparse-view methods improve few-shot reconstruction, yet most remain holistic and appearance-oriented rather than combining sparse-view robustness with per-object, mesh-accurate decomposition.The paper identifies this combination as the regime where its largest gains occur.
  • Positioning: No prior method in the comparison jointly satisfies object decomposition, mesh accuracy, photorealistic appearance, completed background, and sparse-view support.ObjectSplat addresses the gap by decomposing multi-view observations before reconstructing independent components.

3. Methodology

ObjectSplat partitions posed image pixels into independently reconstructed objects and background, then composes them in a shared world frame. Masked-object training and randomized backgrounds remove unsupported geometry while preserving editability and enabling completion for interactive relocation.

  • Image-Space Partition: The method reconstructs each object and the background independently from a partition of image pixels, then composes all components into one scene.Every component uses the shared cameras and world frame, so their union covers the scene without leaving decomposition-induced holes.
  • Object-Level Interactivity: Each self-contained object remains selectable, movable, or removable without disturbing neighboring components.Object separability is retained after composition because components are fitted independently in the shared world frame.
  • Mask Construction: Per-frame mutually exclusive masks come from annotated geometry or video segmentation and tracking, preserving visibility and object identity across frames.Annotated geometry resolves occlusion by depth testing, while video methods propagate identities through the capture without requiring ground-truth geometry.
  • Background Completion: Background images use the per-frame complement of object masks, and unseen regions are completed for interactive relocation when objects are moved.The background is trained from pixels not claimed by objects; image inpainting fills regions that remain unobserved behind removed objects.
  • Completion Scope: Amodal completion is avoided for headline metrics because inconsistent cross-view completions can collapse optimization, while incomplete ground truth can penalize plausible filled geometry.A deterministic hole-filling proxy is used only for tractable flat supporting surfaces and interactive completeness analysis.
  • Independent Reconstruction: Per-object optimization reaches its best checkpoint in fewer than half the iterations required by holistic reconstruction.The same mesh-splatting optimizer and protocol are used for every component, but isolated objects present less geometric and appearance variation.
  • Background Randomization: Randomizing the renderer background forces stray primitives in masked regions to be removed instead of merely recolored.With saturated opacities and nonzero coverage, matching changing backgrounds requires driving foreground weight toward zero, which prunes unsupported geometry.

4. Experiments

Experiments on ScanNet++ evaluate object decomposition for sparse-view reconstruction, showing stronger geometric fidelity, modest appearance gains, faster per-object convergence, and direct interactive editing.

  • Evaluation setup: ScanNet++ evaluation uses per-frame instance masks linked across views and ground-truth meshes, with held-out perspectives testing novel-view synthesis.The primary evaluation selects ten scenes and compares against vanilla mesh splatting using PSNR, SSIM, LPIPS, and F-score.
  • Results: Object decomposition improves F-score on every tested scene at both 30k iterations and best-checkpoint settings, while PSNR and SSIM remain comparable.LPIPS also improves on most scenes, indicating substantial geometric gains alongside more modest appearance changes.
  • Results: Background randomization removes contaminated underside geometry and textures, identifying induced pruning as a primary source of the geometric gain.An ablation on the circular table shows that removing background randomization reintroduces contamination.
  • Results: Under the conventional dense split, the method improves F1 scores by 2% while visual fidelity remains comparable.The appendix also reports operation without ground-truth annotations on MipNeRF-360.
  • Analysis: Per-object reconstruction converges in fewer than half the iterations required by full-scene training, helping explain its novel-view synthesis gains.Holistic trimming needs enough viewpoints to expose inconsistent primitives, whereas background randomization can prune them from a single view.
  • Interactive reconstruction: The composed scene supports moving, removing, and rearranging objects, with exported meshes usable in a standard game engine.Diffusion inpainting fills regions revealed behind relocated objects for qualitative interaction, but is excluded from quantitative results because it lacks cross-view consistency.

5. Limitations and Future Work

The method’s main limitations are cross-view consistency, inadequate ground-truth geometry for evaluating completion, and inherited constraints of mesh-splatting representations.

  • Cross-view consistency: Independent reconstruction requires consistent object delineation across frames, making segmentation, cross-frame association, and occlusion handling a major bottleneck.The paper identifies cross-view consistency as the largest limitation of the decomposition pipeline.
  • Completion evaluation: Incomplete ground-truth meshes penalize plausible completed surfaces as false positives, preventing geometric metrics from confirming completion benefits.Completion therefore remains qualitatively useful for hole-free interactive scenes but is excluded from reported geometric results.
  • Dataset limitations: ScanNet++ is the only identified dataset combining consistent per-frame instance masks tied to 3D objects with ground-truth meshes, constraining evaluation and scene diversity.Fair completion evaluation additionally requires geometry-complete ground truth, which the paper says is unavailable for this setting.
  • Representation limits: The recovered meshes are not watertight, bake in capture lighting, and are spikily triangulated, limiting physics simulation, relighting, close-up fidelity, and contact geometry.The reconstructions also assume static scenes and recovered camera poses; per-object optimization adds cost, partly offset by parallel training.

6. Conclusion

The paper concludes that independently optimizing objects and background with mesh splatting improves sparse-view reconstruction while producing separable, editable scenes, though annotation and completion evaluation remain constrained.

  • 6. Conclusion: The approach independently optimizes objects and background over a partition of image space, then composes them into a mesh-splatting scene.This implements the paper’s decompose-before-reconstruct principle.
  • 6. Conclusion: Background-randomized training exploits primitive non-transparency to prune stray geometry and drive gains in mesh fidelity and novel-view synthesis, especially with few input views.The conclusion identifies this pruning mechanism as central to the reported improvements.
  • 6. Conclusion: The decomposition yields a separable, mesh-exportable scene directly usable for downstream applications.The scene organization supports object-wise use beyond holistic reconstruction.
  • 6. Conclusion: Completion benefits are difficult to measure because recovered surfaces are scored against ground-truth meshes that are themselves incomplete.The paper distinguishes difficulty of measurement from absence of completion benefit.
  • 6. Conclusion: Evaluation uses ten ScanNet++ scenes selected for relatively complete ground-truth meshes and complete 2D masks generated by backprojection.These criteria support quantitative geometric evaluation and mask generation.
  • 6. Conclusion: Backprojected masks inherit missing surfaces and scanning artifacts from reconstructed meshes, imposing an upper bound that pristine 3D annotations could resolve.The cited failure cases include occluded regions, thin structures, transparent or specular surfaces, and dark non-Lambertian materials.

C. Per-Component Training Convergence

Training dynamics vary with component complexity: smaller objects converge faster than large structures or dense backgrounds, favoring per-component optimization and individual checkpoint selection.

  • C. Per-Component Training Convergence: Smaller object instances require significantly fewer optimization iterations because they have lower geometric and textural complexity than large structures or dense backgrounds.The resulting disparity makes per-component training advantageous for selecting the best checkpoint separately for each object.

D.1. Network Architecture

The completion and inpainting modules use a shared latent-diffusion design, while mesh-splatting training evaluates object and whole-scene convergence across 30k iterations.

  • The unified latent diffusion architecture supports both background inpainting and amodal completion for occluded scene regions.
  • Training and validation graphs track PSNR, SSIM, LPIPS, and a weighted L1 combination throughout 30k optimization iterations.
  • The diffusion model conditions denoising on incomplete RGB inputs and masks, with frozen text and VAE encoders.Inference uses a variant of classifier-free guidance.
  • The models begin from pretrained Stable Diffusion v1.5 and update only the diffusion U-Net during retraining.Synthetic Hypersim scenes provide varied indoor geometry and lighting for training.

D.2. Performance and Optimization Limitations

Independent 2D completion generates inconsistent content across views, creating contradictory supervision that degrades splatting-based mesh optimization.

  • Independent background inpainting produces different textures for the same physical region across views.Examples include differently colored grid patterns and flat gray surfaces.
  • Frame-to-frame generative variations create conflicting gradients that degrade 3D mesh optimization.The problem arises because splatting relies on photometric consistency across overlapping views.
  • Independent amodal completion yields inconsistent predictions across frames, including missing legs, texture noise, and geometric warping.

E. MipNeRF-360 Results

MipNeRF-360 demonstrates that the method can operate without ground-truth annotations, though automatic mask inconsistency slightly reduces visual fidelity and prevents geometric evaluation.

  • MipNeRF-360 uses automatically generated video-segmentation masks rather than ground-truth masks.
  • Automatic segmentation causes a marginal loss of visual fidelity relative to ground-truth-mask settings.
  • MipNeRF-360 lacks a ground-truth mesh, so the effect of mask inconsistency on geometric fidelity cannot be evaluated quantitatively.

F. Surface Completion

Surface completion repairs occluded planar support surfaces for interactive editing, but its measured geometric effect is marginal and current benchmarks limit quantitative evaluation.

  • A completed background mask is intended to support downstream interactive editing.
  • The completion pipeline uses OpenCV Navier-Stokes inpainting over projected planar surfaces on four selected ScanNet++ scenes.
  • Surface completion marginally decreases 3D F-score while leaving PSNR, SSIM, and LPIPS virtually unchanged.Foreground objects remain in their original positions during evaluation, reoccluding the completed regions.
  • Deterministic surface completion repairs occlusion holes and broken edges on planar supporting surfaces.The OpenCV method fills regions where foreground objects previously stood.
  • Quantitative evaluation of completion requires geometry-complete ground truth, which is unavailable at ScanNet++ scale and complexity.

G. Dense-View Experiments

Dense-view experiments show that decomposition does not improve reconstruction over holistic mesh splatting when viewpoints are highly redundant. In this regime, noisy object boundaries can outweigh decomposition’s benefits.

  • Dense-view results are reported on ScanNet++ using SSIM, PSNR, LPIPS, and F-score under a 7:1 train-to-test split.
  • Our method’s scene reconstruction quality is on par with vanilla mesh splatting in dense-view experiments.
  • Our method performs marginally below holistic mesh splatting in dense-view experiments.
  • With redundant viewpoints, vanilla mesh splatting can prune erroneous geometry through dense multi-view optimization, reducing decomposition’s relative improvement.
  • Noisy, incomplete object boundaries across many dense-view frames can dominate the trade-off and reduce decomposition performance.
Loading 2608.30423v1…