PaperlayerView exact revision on arXiv

Source-linked AI summary

CADSplat: Sparse-View 3D Gaussian Splatting Aided by CAD Models for Robust, Photorealistic Digital-Twin Reconstruction

Kristof Overdulve, Lode Jorissen, Nick Michiels

arXiv:2609.18473v1cs.CVcs.GR

TL;DR

Sparse, wide-baseline images make accurate object reconstruction difficult, with learned geometry prone to noise and hallucination. CADSplat retrieves and deforms a CAD model while jointly refining camera poses and surface-anchored 3DGS; it outperforms the reported baselines and degrades gracefully to three views.

  • Problem

    Sparse-view reconstruction is underdetermined, so learned representations can develop noisy geometry and hallucinate structure that fails to generalize to novel views.

    Source: §1 · p. 2

  • Method

    CADSplat matches object silhouettes to CAD-library renders, anchors Gaussians to the retrieved mesh, and jointly optimizes camera registration, deformation, and 3DGS parameters.

    Source: §1 · p. 2, §3.4 · p. 7

  • Results

    CADSplat outperforms unconstrained, few-shot, and mesh-texturing baselines, with quality degrading gracefully down to three views.

    Source: §7 · p. 21

  • Takeaways & Limitations

    The constrained deformable representation accounts for much of the rendering-quality gain, while CAD geometry matters most under sparse or self-occluded views.

    Source: §7 · p. 21

    The method requires a good initial estimate of relative camera calibration for silhouette matching and pose optimization to work well.

    Source: §6 · p. 20

Abstract

from arXiv · show

We present CADSplat, a framework that reconstructs photorealistic, geometrically accurate digital twins from sparse ($<15$ views), wide-baseline posed images of an object by regularizing 3D Gaussian Splatting (3DGS) with an explicit CAD shape prior. Using such a prior requires finding a CAD model whose shape resembles the object depicted in the images and determining the pose of each camera relative to the object. We obtain both by matching segmented object silhouettes against silhouettes rendered from a CAD library and keeping the camera-to-object poses of the best-matching model. We then anchor 3D Gaussian primitives to the surface of the retrieved model and jointly optimize the 3DGS parameters, the camera-to-object registration, and a non-rigid deformation field to account for shape differences between the physical object and the CAD model. Across two real-world datasets, CADSplat outperforms unconstrained, few-shot, and mesh-texturing baselines and degrades gracefully to as few as 3 views. Our experiments show that most of the gain in rendering quality comes from how the splats are constrained---a fixed set of splats tied to a surface and moved by a single smooth deformation field---rather than from the CAD shape itself. The CAD model adds shape knowledge where views are scarcest, in the sparsest captures and on strongly self-occluded objects, and it places every camera in the object's own frame. This enables applications beyond novel-view synthesis, such as markerless augmented reality registration, per-image object pose estimation, physical simulations, and the transfer of part labels from the design to the reconstruction.

1 Introduction

Sparse-view reconstruction can yield noisy geometry and hallucinated structure, while CAD models provide clean geometry but lack the appearance needed for photorealism. CADSplat combines a retrieved CAD prior with jointly refined camera poses, deformation, and 3DGS to reconstruct digital twins and support object-centered applications.

1 Introduction

  • Sparse-view image reconstruction is underdetermined, producing noisy geometry and hallucinated structure that may fail to generalize to novel views.CAD models offer clean, interpretable geometry but often lack textures, material properties, and subtle imperfections needed for photorealism.Source: §1 · pp. 1–2
  • The method combines a topologically accurate CAD model with image-based appearance to reconstruct a photorealistic, geometrically accurate digital twin.The CAD prior can come from manufacturer models or repositories such as ShapeNet [5].Source: §1 · p. 2
  • CADSplat matches segmented silhouettes to CAD-library renders, then jointly refines camera-to-object poses, non-rigid deformation, and 3DGS parameters.The deformation accommodates differences between the retrieved model and the physical object.Source: §1 · p. 2, Figure 1 · p. 2
  • The object-centered reconstruction supports markerless augmented-reality registration, per-image pose estimation, physics simulation, CAD part-label transfer, and interpretable inspection of design deviations.Part labels transfer because each splat is sampled from a CAD-mesh face.Source: §1 · pp. 2–3
  • CADSplat achieves state-of-the-art rendering quality from 9 to 13 views and degrades gracefully to 3 views, using a fixed dense splat set anchored to a deformable shape prior.Its contributions also include two appearance regularizers for generalization in the sparse-view regime.Source: §1 · p. 3

2 Related Work

Prior work addresses sparse-view synthesis, mesh texturing and anchoring, pose recovery, CAD retrieval, and joint pose-and-reconstruction problems through different priors and capture assumptions. CADSplat distinguishes itself by retrieving a CAD model for a calibrated multi-view capture and refining its registration jointly with a surface-anchored 3DGS representation.

2.1 Sparse-View Synthesis

  • Sparse-view synthesis methods use geometric, semantic, generative, or learned priors, while CADSplat imposes an explicit prior from the object’s CAD model.Generative priors can complete ambiguous regions plausibly but not necessarily faithfully, and may transfer poorly to reflective or textureless industrial objects.Source: §2.1 · p. 3

2.2 Classical Mesh Texturing

  • Classical MVS texturing assigns image textures to a fixed mesh but is fragile under sparse overlap and pose error and lacks view-dependent appearance.These limitations can produce fragmented, inconsistent textures.Source: §2.2 · p. 3

2.3 Mesh Anchoring

  • Prior mesh-anchored Gaussian methods use meshes extracted from dense captures or supplied in known alignment; CADSplat retrieves and registers its CAD model for sparse-view regularization.Its deformation field repurposes architectures used for dynamic motion to align canonical CAD space with static physical observations.Source: §2.3 · pp. 3–4

2.4 Pose Estimation and CAD-Based Tracking

  • CADSplat initializes object poses by aggregating silhouette matches across views into a registration consistent with relative calibration, avoiding a learned per-pixel coordinate predictor.The resulting poses are coarse and require joint refinement.Source: §2.4 · p. 4

2.5 CAD Model Retrieval and Alignment

  • Unlike single-image CAD retrieval methods trained on scan annotations or synthetic data, CADSplat uses silhouette proposals and existing relative calibration to register a whole capture.It then refines that capture-level registration together with the reconstruction.Source: §2.5 · p. 4

2.6 Joint Novel-View Synthesis and Pose Estimation

  • Compared with NeRS, CADSplat retrieves an actual CAD model for a stronger geometric prior and camera frame; compared with CADNeRF, it uses surface-aligned 3DGS and a shared smooth deformation.CADSplat also differs from pose-free methods that assume temporally ordered video with small inter-frame distances, and from CADSim’s vehicle-specific priors.Source: §2.6 · pp. 4–5

3 Methodology

CADSplat reconstructs a photorealistic asset and camera-to-object poses from sparsely captured images with known relative calibration. It retrieves and deforms a CAD mesh, anchors Gaussians to its surface, and jointly optimizes pose, geometry, and appearance.

3 Methodology

  • The pipeline retrieves a CAD mesh and coarse camera registration from silhouette matches, anchors Gaussians to the mesh, and jointly refines pose, deformation, and appearance.Optimization first aligns rendered alpha with masks, then fits view-dependent appearance to masked photographs.Source: §3 · p. 5

3.1 Preliminaries: 3D Gaussian Splatting

  • 3DGS represents each primitive with a center, three scales, a rotation quaternion, opacity, and view-dependent color encoded by spherical-harmonic coefficients.Rendering projects the primitives into screen space using splatting and alpha blending.Source: §3.1 · p. 5

3.2 Silhouette-Based CAD Matching

Silhouette matching retrieves a CAD model and initializes camera-to-object poses by combining per-image shape matches with relative camera calibration. Consensus registration addresses ambiguity in individual silhouette matches before joint refinement.

3.2 Silhouette-Based CAD Matching

  • Observed masks and CAD-rendered silhouettes are cropped to tight bounding boxes and resized to fixed square canvases, enabling comparisons independent of image position and object scale.Candidate models are rendered from a dense viewpoint grid, with masks extracted from posed input images using SAM3.Source: §3.2 · p. 5, Figure 2 · p. 6
  • For each observed mask, the method retains up to 50 viewpoint hypotheses with IoU of at least 0.7, then combines them with relative calibration using RANSAC over Sim(3).The selected CAD model and transformation maximize aggregate per-view IoU support.Source: §3.2 · pp. 5–6
  • Consensus registration places cameras coherently around the object, unlike independent closest-silhouette matches, and remains unaffected by object material properties.A single discriminative viewpoint can anchor the trajectory even when other views are ambiguous.Source: §3.2 · p. 6

3.3 Camera Pose Optimization

CADSplat refines coarse silhouette-based camera-to-object registration jointly with reconstruction. A global correction handles accurate relative calibration, while per-camera corrections address individually unreliable poses.

3.3 Camera Pose Optimization

  • The silhouette-based registration is only coarse, so camera pose corrections are learned jointly with reconstruction.When relative calibration is accurate, one global correction is shared across cameras; when poses are unreliable per view, each camera receives its own correction.Source: §3.3 · pp. 6–7
  • Consensus registration uses one similarity transform for the calibrated trajectory, producing consistent initial poses that joint optimization refines.Independent silhouette matches can scatter cameras into different, sometimes mirrored, orientations; relative calibration serves as a RANSAC consistency filter.Source: Figure 3 · p. 7
  • The global correction scales camera centers to absorb the unknown scale between the calibrated trajectory and the CAD canonical frame.It also applies a shared rotation and translation; the alternative for unreliable per-view poses is an independent correction per camera.Source: §3.3 · p. 7

3.4 Geometric Initialization

CADSplat initializes its Gaussian primitives by sampling the retrieved CAD mesh rather than the SfM sparse point cloud. Mesh normals orient the splats as surface-tangent disks, and the method omits densification and pruning.

3.4 Geometric Initialization

  • The initial Gaussian primitives are sampled directly from the retrieved CAD mesh instead of the SfM sparse point cloud.The method draws area-weighted uniform samples from the mesh surface to reach a target primitive count.Source: §3.4 · p. 7
  • Each sampled point’s mesh normal sets its Gaussian rotation, initializing the primitive as a disk tangent to the surface.Normals come from the intersected mesh face, and the rotation aligns the Gaussian’s local z-axis with that normal.Source: §3.4 · p. 7
  • Unlike standard 3DGS, the initialization uses no densification or pruning strategies such as cloning or splitting.Source: §3.4 · p. 7

3.5 Deformable Mesh-to-Reality Alignment

A coordinate-based deformation network adjusts the retrieved CAD geometry to match the physical object. Its shared canonical-space offsets provide a smooth shape correction across viewpoints.

3.5 Deformable Mesh-to-Reality Alignment

  • A coordinate-based deformation network addresses shape discrepancies between the retrieved CAD mesh and the physical object, including library mismatch and manufacturing tolerances or errors.The network is an 8-layer MLP with width 256 and a middle-layer skip connection.Source: §3.5 · p. 7
  • The network maps a canonical splat position to a position offset, using sinusoidal positional encoding as input.Its architecture uses a skip connection at Layer 4 and a single linear head to predict the offset.Source: §3.5 · p. 7, Figure 4 · p. 8
  • Because offsets are defined in canonical space, one smooth deformation corrects the mesh shape across viewpoints and helps avoid overfitting sparse training views.The output head is initialized so the offset is approximately zero at the start of training.Source: §3.5 · p. 8

3.6 Optimization Pipeline

CADSplat separates geometric warmup from photometric finetuning, then regularizes appearance to address sparse-view ambiguity. Splats remain fixed in canonical placement while pose and a shared deformation field absorb geometric corrections.

3.6 Optimization Pipeline

  • The pipeline separates pose and deformation learning from appearance fitting, while freezing splat means and normal-aligned rotations so the deformation field absorbs geometric corrections.The geometric warmup has pose-only and pose-plus-deformation sub-phases, followed by photometric finetuning.Source: §3.6 · p. 8

3.6.1 Stage I: Geometric Warmup via Mask Alignment

  • During the first 3,000 iterations, only pose is optimized; from 3,000 to 6,000, deformation is enabled to wrap the CAD silhouette to the observed mask.Stage I aligns rendered alpha with object masks using an L1 loss.Source: §3.6.1 · p. 8

3.6.2 Stage II: Photometric Finetuning

  • After 6,000 Stage-I iterations, photometric fitting optimizes color, scale, and opacity while continuing pose and deformation updates.The masked RGB loss combines L1 and D-SSIM; random backgrounds help isolate the object and prevent memorization of environmental artifacts.Source: §3.6.2 · p. 8

3.6.3 Sparse-View Appearance Regularizers

  • Stage II uses CAD-neighbor SH smoothness and stochastic splat dropout to regularize appearance under sparse views.Each splat’s K=20 nearest neighbors are fixed in canonical space, making the smoothness term invariant to small deformations; dropout affects appearance only, with half the splats dropped initially and the rate reduced to zero over 20k iterations.Source: §3.6.3 · pp. 8–9

4 Experiments and Evaluation

CADSplat is evaluated on sparse, object-centric captures using protocols designed to address calibration mismatch and compare object-level with scene-level methods. It achieves the best six-scene average across all metrics, while qualitative and sparsity evaluations examine reconstruction quality under limited views.

4.1 Implementation Details

  • The implementation initializes 600,000 surface-anchored splats and trains for 30,000 iterations, with pose, deformation, and 3DGS parameters jointly optimized in the final phase.The splat count remains fixed without densification or pruning, and dropout anneals to zero over Stage II's first 20,000 iterations.Source: §4.1 · p. 9, Table 1 · p. 10

4.2 The SCO-CAD Dataset

  • SCO-CAD contains six real objects paired with approximate public CAD models, deliberately creating geometric deviations that test deformation robustness.Training poses come from sparse-image HLoc calibration, while dense COLMAP calibration is reserved for evaluation.Source: §4.2 · p. 10

4.3 Evaluation Protocol and Metrics

  • The evaluation refines each held-out camera for 400 steps with the reconstruction frozen, separating pose-alignment effects from reconstruction quality.Eroded-mask metrics assess reconstruction inside the object mask, while fgPSNR scores the intersection of rendered and ground-truth foreground masks; scene-level baselines also require masking for object-to-object comparison.Source: §4.3 · p. 11

4.5 Quantitative Results

  • CADSplat achieves the best six-scene average across all metrics with test-time pose refinement.The ranking without refinement matches on eight of ten metrics; mSSIM and mFLIP shift by no more than 0.001.Source: §4.5 · p. 12, Table 2 · p. 14, Table 3 · p. 14
  • Vanilla 3DGS trails CADSplat across reported metrics, with the largest mPSNR gaps on voluminous, occluded objects and the smallest on flat scissors.The mPSNR gap is +2.7 to +3.1 dB for car1, carjack, and senseo, versus +0.9 dB for scissors; masking objects and randomizing background colors prevents floaters and narrows the baseline gap.Source: §4.5 · p. 12
  • MVS-Texturing is the weakest baseline, averaging PSNR 11.77, mPSNR 14.64, and fgPSNR 6.83 across six scenes.Its patchwork textures, holes, and missing parts worsen when pose or CAD shape is inaccurate; test-time refinement diverges from the object.Source: §4.5 · p. 12
  • InstantSplat trails CADSplat on every reported metric, while FSGS is competitive on pixel metrics but trails on perceptual metrics.InstantSplat reports mPSNR 27.9 vs. 29.0, mLPIPS 0.045 vs. 0.031, and fgPSNR 19.6 vs. 20.4; FSGS reports mLPIPS 0.046 vs. 0.031 and mFLIP 0.057 vs. 0.048.Source: §4.5 · p. 12, Table 2 · p. 14

4.5.1 Robustness to View Sparsity

  • Fig. 7 compares sparse-view reconstructions from 9–13 images: CADSplat preserves sharp edges and accurate textures, unlike the baselines' patchwork, smoothing, or coarse geometry.The sparsity evaluation selects training views by farthest-point sampling over camera centers and evaluates against the full held-out trajectory.Source: Figure 7 · p. 13, Table 4 · p. 14

3 Views 5 Views 8 Views 10 Views

CADSplat retains stable geometry as views are removed, with the CAD prior most useful under severe scarcity and occlusion. Ablations show contributions from deformation, pose and mask alignment, and appearance regularizers; the method also improves camera-to-object registration and performs strongly on NeRS datasets.

3 Views 5 Views 8 Views 10 Views

  • From all views to 3, silhouette IoU falls from 0.942 to 0.929, while mLPIPS grows 2.7× (0.031 → 0.083) and fgPSNR drops 3.3 dB (20.36 →17.01).At 3 views, the reported metrics remain usable: PSNR 20.7, mPSNR 25.2, and IoU 0.93; reduced-view tests reuse calibration and silhouette registration from the full capture.Source: §3 · pp. 14–15

4.5.2 Initialization: CAD Mesh vs. a Generic Sphere

  • At 3 views, CAD initialization beats a sphere by +1.4 dB PSNR, +1.1 dB mPSNR, +1.0 dB fgPSNR, and 4.9 silhouette-IoU points (0.929 vs. 0.879).At the full view budget, sphere initialization is slightly ahead on PSNR (23.54 vs. 23.24), mPSNR (29.40 vs. 28.95), and IoU (0.946 vs. 0.942); sphere-initialized CADSplat also exceeds vanilla 3DGS by +2.8 dB mPSNR, +2.6 dB fgPSNR, and +0.03 IoU.Source: §4.5.2 · p. 15, Table 5 · p. 16

4.5.3 Ablation of Design Decisions

  • Deformation alone reaches PSNR 22.92, mPSNR 28.38, and IoU 0.941, while pose optimization alone reaches PSNR 22.67; without mask alignment, reconstruction collapses to PSNR 18.88 and IoU 0.784.Disabling deformation exposes CAD-to-object mismatch as ghosting; pose optimization cannot recover non-rigid shape discrepancies, and without geometric warmup the geometry does not recover.Source: §4.5.3 · pp. 15–16, Table 6 · p. 16
  • SH-neighbor smoothness improves PSNR from 22.93 to 23.13 and mLPIPS from 0.039 to 0.032, while splat dropout raises PSNR from 23.13 to 23.24 and fgPSNR from 20.18 to 20.36.Neighbor smoothness removes high-frequency color changes in unobserved views; dropout supplies optimization signals to splats occluded during training but visible or differently sorted in held-out views.Source: §4.5.3 · p. 16

4.5.4 Camera-to-Object Registration Accuracy

  • Joint pose optimization reduces median camera-to-object errors from 6.2° rotation and 18.6% position error to 2.9° and 6.1%, respectively.It reduces both errors on every scene except senseo’s rotation, which changes from 3.1° to 3.3° after an already accurate consensus registration.Source: §4.5.4 · pp. 16–17

4.6 NeRS MVMC

  • On NeRS MVMC, CADSplat achieves the best PSNR and SSIM among the comparison methods, improves on NeRS across all three metrics, and trails CADSim only on LPIPS.The comparison uses the dataset’s object-crop metrics and NeRS’s strongest published setting with fixed optimized cameras.Source: Table 7 · p. 18, §4.6 · p. 18

4.7 NeRS Misc

  • On NeRS Misc, CADSplat produces clean, artifact-free reconstructions even when the closest ShapeNetCore shape is only a rough geometric approximation.Examples include using a bottle for a fire hydrant and a table for a PS5 controller, with deformation bridging the remaining category-level gap.Source: §4.7 · p. 18

5 Applications

CADSplat’s CAD alignment and surface-anchored Gaussians connect photorealistic appearance to the model’s structure and motion properties. The examples demonstrate physics-based interaction and editable exploded views.

5 Applications

  • A rigid-body simulation uses deformed CAD meshes as collision geometry while surface-anchored Gaussians follow the resulting poses, enabling photorealistic hammer–skateboard interaction.Contact is resolved from the true object geometry rather than a bounding proxy.Source: §5 · pp. 18–19, Figure 12 · p. 20
  • Because Gaussians are bound to CAD parts, translating the skateboard’s wheels off their axles leaves the deck fixed and produces a photorealistic exploded view.The CAD model supplies part structure, while the reconstruction supplies appearance.Source: §5 · pp. 18–19, Figure 13 · p. 20

6 Limitations

CADSplat depends on usable camera calibration and benefits from a geometrically similar CAD model. Its appearance can fail at extreme viewpoints, and its deformed mesh is not designed for measurement.

6 Limitations

  • The method requires a good initial relative-calibration estimate for silhouette matching and for camera-pose optimization to converge well.The estimate may come from manual pre-calibration or sparse-view SfM.Source: §6 · p. 20
  • The deformation field is optimized for reconstruction, not measurement, and the underlying deformed mesh is not itself optimized to be a plausible surface.Improving the surface toward metrology-grade measurement is left for future work.Source: §6 · p. 20
  • Extreme-viewpoint extrapolation can yield unusable appearance because per-splat opacities and view-dependent spherical harmonics remain partly overfit to training viewpoints.The regularizers reduce but do not eliminate overfitting; broader out-of-distribution viewpoint generalization requires priors on object materials as well as geometry.Source: §6 · p. 20, Figure 14 · p. 20

7 Conclusion

CADSplat reconstructs digital twins from sparse wide-baseline images by anchoring 3D Gaussians to a retrieved CAD model and jointly refining registration, deformation, and appearance. It outperforms the evaluated baselines, with CAD geometry most useful when image views are sparse or self-occlusion is strong.

7 Conclusion

  • Anchoring 3D Gaussians to a retrieved CAD model and jointly refining registration, a non-rigid deformation field, and appearance reconstructs a photorealistic, geometrically clean digital twin.Camera registration uses silhouette consensus.Source: §7 · pp. 20–21
  • CADSplat outperforms unconstrained, few-shot, and mesh-texturing baselines, with rendering quality degrading gracefully down to three views.The constrained deformable representation accounts for much of the rendering-quality improvement over unconstrained 3DGS.Source: §7 · p. 21
  • Much of the rendering-quality improvement over unconstrained 3DGS comes from the constrained deformable representation, rather than the CAD geometry alone.The representation fixes the splat set to a surface and moves it with a single smooth deformation field.Source: §7 · p. 21
  • The CAD geometry prior matters most when image evidence is ambiguous, including extreme view sparsity or significant self-occlusion.Source: §7 · p. 21
  • Camera-to-object poses from the CAD prior enable markerless augmented-reality registration, per-image pose estimation, physically based interaction, and part-aware editing.Source: §7 · p. 21
  • Future work targets metrology-grade surface quality through improved deformation and extension to extreme-view synthesis.Source: §7 · p. 21
Loading 2609.18473v1…