Source-linked AI summary
SAMV-DUSt3R: Instance-Centric 3D Scene Decoupling from Sparse Multi-Views
Zhao, Langxu, Gu, Zuan, Zhang, Yingdan, Zhao, Pengfei, Gao, Tianhan
TL;DR
SAMV-DUSt3R addresses prompt-guided 3D object decoupling from sparse, unordered, uncalibrated views. It injects SAM2-derived masks into MV-DUSt3R reconstruction and automates reference-view selection with Spatial RankGNN. Spatial RankGNN achieves 73.5% ranking accuracy, while the method is reported to efficiently disentangle target objects within scenes.
Problem
Existing MV-DUSt3R reconstruction relies on a single reference view, while object-centric reconstruction requires target masks and focused reconstruction.
Method
SAMV-DUSt3R combines MV-DUSt3R with SAM2 masks, Cross Flow Mask Block, Masked Confidence Loss, and Spatial RankGNN for target-focused reconstruction and reference-view selection.
Results
73.5% Spatial RankGNN ranking accuracy was achieved under the stated rank-tolerance criterion.
Takeaways & Limitations
SAMV-DUSt3R efficiently disentangles prompt-specified 3D target objects within scenes from sparse RGB views.
Abstract
from arXiv · showhide
With the rising demand to decouple objects from 3D scenes, we propose SAMV-DUSt3R, an end-to-end model that injects SAM2 2D masks into MV-DUSt3R reconstruction. A Cross Flow Mask Block uses these masks to steer the network toward the target instance, jointly improving shape accuracy and achieving object-level disentanglement without multi-stage pipelines. To ensure reconstruction stability, a lightweight Spatial RankGNN selects the optimal reference view with a selection accuracy of 73.5\%. Extensive experiments demonstrate that our method boosts average reconstruction precision by 11\% across various metrics compared to state-of-the-art baselines. These results reveal a strong instance-disentanglement capability and clear benefits for driving, robotics, AR/VR, and heritage digitisation.
I. INTRODUCTION
SAMV-DUSt3R targets prompt-specified 3D object decoupling from unordered, uncalibrated sparse views. It combines MV-DUSt3R reconstruction with mask guidance and automated reference-view selection to address single-reference limitations.
- SAMV-DUSt3R decouples prompt-specified 3D objects from sparse, unordered, uncalibrated RGB views without camera parameters or image calibration.
- The approach is designed for object-level disentanglement and downstream applications including driving, robotics, AR/VR, and heritage digitisation.
- The model combines MV-DUSt3R with SAM2 masks so multi-view reconstruction can focus on the target object.
- Spatial RankGNN ranks view spatial information to select a reference view, addressing MV-DUSt3R’s reliance on a single manually selected reference view.
III. METHOD
SAMV-DUSt3R uses Spatial RankGNN to automate reference-view selection and Cross Flow Mask Block to inject 2D mask cues into MV-DUSt3R sequence features. The resulting pipeline produces point maps, confidence maps, and segmentation masks for target-focused reconstruction.
- Architecture Design: SAMV-DUSt3R processes unordered views with MV-DUSt3R, Spatial RankGNN, SAM2 masks, and Cross Flow Mask Block to reconstruct target objects.
- Architecture Design: Spatial RankGNN models image positions as graph nodes, uses global RankMessagePassing interactions, and outputs normalized image-ranking scores.
- Architecture Design: Cross Flow Mask Block converts 2D mask features into token-aligned keys and values, while MV-DUSt3R sequence tokens act as queries.
- Architecture Design: The module uses cross-attention, residual connections, layer normalization, and feedforward refinement to guide sequence representations toward object regions.
B. Loss function
The training procedure combines Spatial RankGNN ranking supervision with target-sensitive confidence learning. Masked Confidence Loss emphasizes target pixels and regularizes confidence within masked regions to support object-focused reconstruction.
- Spatial RankGNN training uses 3,600 SfM scenes containing 4, 12, or 24 views distributed at a 5:8:23 ratio.Each scene is represented by a covisibility graph whose nodes are input views and whose edge weights count co-visible points.
- 73.5% ranking accuracy was achieved by Spatial RankGNN under the stated rank-tolerance criterion.The module was trained for 5 hours on 8 × NVIDIA A800 GPUs.
- Masked Confidence Loss addresses low-confidence target pixels by jointly considering confidence-weighted 3D regression and target-region confidence distributions.The method identifies a confidence basin in which decoupling-worthy object pixels have unreliable 3D position regression.
- Foreground weighting amplifies masked-pixel regression errors while retaining baseline weighting for background regions.
- A uniformity regularizer encourages homogeneous high confidence within target masks and uses λ = 0.01.The regularizer minimizes intra-mask confidence variance while preserving the confidence-raising term.
A. Datasets
The method uses SfM-derived multi-view scenes to train Spatial RankGNN and object-centric datasets to train the mask-guided reconstruction components. DTU is used to evaluate 3D object disentanglement.
- Spatial RankGNN training uses 3,600 scenes with 4, 12, or 24 input views distributed in a 5:8:23 ratio.The scenes are represented as covisibility graphs with views as nodes and co-visible points as edge weights.
- Co3Dv2 and WildRGBD provide RGB images, depth maps, camera parameters, and ground-truth masks for Cross Flow Mask Block training.
- The object-centric DTU dataset is used to evaluate SAMV-DUSt3R’s disentanglement capability.
1) Metrics:
The evaluation measures local surface detail and normal precision, as well as global reconstruction fidelity and coverage.
- ND, DAc, and CD assess local surface detail and normal precision, while Acc and Comp measure global reconstruction fidelity and coverage.
2) Details:
The DTU evaluation samples views uniformly, resizes them consistently, and centers cropping on object masks while using perceptual and image-quality metrics.
- DTU scenes provide 49 views, sampled uniformly with a fixed step size for even input-view distribution.Each view is resized to 224×224.
- Mask-centered cropping maximizes retention of object regions before quantitative evaluation.
- PSNR, SSIM, and LPIPS quantify disparity between rendered views and corresponding ground-truth images.
2) Details:
The evaluation initializes 3D Gaussians from masked pointmaps and computes object-focused novel-view metrics, with an object-centric ablation protocol for comparing model variants.
- Masked pointmaps from Co3Dv2 and WildRGBD initialize 3D Gaussian positions, while novel views use cameras outside the input views.
- Pixels outside the object mask are zeroed in ground-truth and rendered images to compute masked metrics Pmsk, Smsk, and Lmsk.
- Because MV-DUSt3R lacks object-centric training and requires masks, the study trains MV-DUSt3Robject and SAMV-DUSt3R with Masked Confidence Loss, then removes Spatial RankGNN for SAMV-DUSt3Runrank.
2) Spatial RankGNN:
Spatial RankGNN operates on encoder image embeddings to rank reference views, and its integration is reported to improve reconstruction accuracy through automatic selection.
- Spatial RankGNN significantly enhances reconstruction accuracy by automatically selecting the most suitable reference view.It operates solely on encoder-generated image embeddings and outputs ranking scores without modifying those embeddings.
- Figure 4 visualizes masked-region confidence distributions, target disentanglement under the same scene, and a single-channel confidence heatmap.
- SAMV-DUSt3R combines automated reference-view selection with mask-guided reconstruction to disentangle prompt-specified targets from sparse RGB views.