RGBD-to-3D Object Mesh Refinement via Depth Matching and Symmetry Propagation

1 Introduction

• Visible-surface errors are most apparent near depth discontinuities and self-occlusions, where plausible reconstructions can disagree with input geometry.Because much of the shape is unobserved and visible surfaces are ambiguous, small visible errors can grow into larger full-mesh distortions.Source: §1 · p. 1
• Existing test-time refinement can require costly per-instance optimization or rely on silhouettes that provide little guidance for fine surface corrections.Prior symmetry methods impose symmetry globally or learn it end-to-end rather than transferring reliable visible corrections into unseen regions.Source: §1 · p. 2
• The framework refines any RGB-to-3D reconstructor without retraining by matching visible vertices to depth points, mirroring corrections across a symmetry plane, and propagating them with a smoothness solver.Each stage is a closed-form solve; experiments report consistent gains across five backbones and GSO and OmniObject3D, including with monocular pseudo-depth.Source: §1 · p. 2

Single-view 3D object reconstruction

• Single-view reconstruction methods use generative priors, diffusion, feed-forward triplanes and transformers, or generated novel views, but input-view accuracy can still lag.RGB-D methods use depth but typically complete partial point clouds rather than produce full watertight meshes.Source: Single-view 3D object reconstruction · p. 2

Test-time refinement for 3D

• Test-time refinement uses instance-specific cues, but prior approaches include costly optimization and silhouette-based guidance with limited fine-correction signal.The paper contrasts its closed-form stages and metric-depth anchoring with iterative per-instance refinement.Source: Test-time refinement for 3D · p. 3

Symmetry for 3D

• Prior symmetry methods enforce symmetry on predicted shapes or learn it end-to-end; this method mirrors depth-guided visible-side displacements onto the occluded side.Other single-image approaches use reflection planes to create a virtual view or constrain depth, or fuse flipped predictions with perceptual objectives.Source: Symmetry for 3D · p. 3

3.1 Problem Definition

• The method estimates per-vertex displacements from an initial mesh, a metric depth map, camera intrinsics, projection, and camera pose, then applies them to update the mesh.Its three stages are visible-surface depth guidance, symmetry-guided transfer, and full-mesh smoothness propagation.Source: §3.1 · p. 3

3.2 Depth-guided Vertex Displacement

• Visible mesh vertices are matched to back-projected depth points in a shared camera frame, producing reliable displacements on the observed surface.The mesh is rasterized to identify visible vertices; assignment is one-to-one but partial, and high-cost matches are discarded.Source: §3.2 · p. 3, Camera-frame transformation · p. 4, Depth guidance for visible vertices · p. 4

3.3 Symmetry-guided Vertex Displacement

• A dominant bilateral symmetry plane is selected from candidate planes using rasterized signed-distance maps, and matched visible-side corrections are assigned to opposite-side vertices for reflection.The reflection stage discards high-cost matches and never overwrites a target with a depth-driven displacement.Source: Bilateral symmetry plane detection · p. 5, Displacement transfer by bipartite… · p. 6

3.4 Mesh Refinement with Vertex Displacement

• Depth and reflected corrections serve as Dirichlet handles; a cotangent-Laplacian solve propagates their displacements to free vertices while enforcing smoothness.Degenerate-triangle weights are clamped, the linear system is stabilized, and free boundary vertices are softly anchored; visible-only propagation is a separate variant.Source: Motivation · p. 6, Displacement propagation via Dirichlet… · p. 6

4.1 Datasets and Evaluation

• Evaluation uses 1,030 GSO scans and 1,038 OmniObject3D shapes, with one of 24 rendered viewpoints as input and geometry and image metrics assessed against GT.Meshes are independently aligned to GT for geometric evaluation; the refinement output is not realigned or rescaled.Source: Datasets · p. 7, Protocol · p. 7, Metrics · p. 7

4.2 Implementation Details

• The experiments use fixed matching thresholds and symmetry-search settings without per-dataset tuning; on GSO, refinement averages 7.5 seconds per instance, with propagation the bottleneck.Matching accounts for 34.5% of runtime and cot-Laplacian propagation for 61.9%.Source: Refinement · pp. 7–8, Matching · p. 8

4.3 Model-agnostic Single-view Mesh Refinement

• CD decreases and FS rises at both thresholds for all five backbones on both datasets; rendered-image SSIM, PSNR, and LPIPS also improve everywhere.The gains are largest for weaker initial meshes and at the stricter 0.05 threshold.Source: Table 1 · p. 8, Table 2 · p. 8, §4.3 · p. 9

4.4 Comparison with Single-view RGBD Methods

• Depth-guided refinement improves Affostruction on both datasets, while the symmetry stage adds no gain; refined RGB-to-3D backbones also surpass the completion methods.The comparison gives all methods the same RGB-D observation, including GT depth and camera pose.Source: §4.4 · pp. 9–10, Table 3 · p. 9

4.5 Comparison with Test-time Refinement Methods

• On REFINE’s six-instance demo set, pseudo-depth improves the 10k-vertex mesh but leaves both 1.5k-vertex meshes below their initial versions; the method is an order of magnitude faster.REFINE improves its own coarse template but mask-only supervision axis-stretches both InstantMesh meshes.Source: Table 4 · p. 9, Comparison with REFINE · p. 10
• On nine Pix3D instances, MeTTA scores better on input-view PSNR and LPIPS, while the refinement improves the initial mesh with GT depth and stays closer with pseudo-depth.Against GT-mesh renders at novel views, MeTTA falls below the initial mesh at every view.Source: Table 5 · p. 9, Comparison with MeTTA · p. 10

4.6 Ablation Study

• Depth guidance alone gives the largest CD reduction on both datasets, with smaller but consistent gains from pseudo-depth rescaled to the GT depth range.This pattern holds across all five backbones.Source: Depth guidance · p. 11
• Adding symmetry lowers CD for every backbone, with the largest gains on symmetric objects and possible slight harm on asymmetric ones.Mirrored corrections can mismatch the true geometry of asymmetric objects.Source: Symmetry guidance · p. 11
• At the 0.1 matching threshold, propagation further lowers CD and raises strict FS(0.02), while results are stable from 0.1 to 0.2.Tighter matching weakens correction, much looser matching admits wrong correspondences, and changing the symmetry threshold barely affects results.Source: Propagation and matching thresholds · pp. 11–12, Table 8 · p. 12

4.7 Analysis

• Estimated depth or pose still improves InstantMesh/GSO when the other input is reliable, but combining estimated pose with GT-scaled pseudo-depth falls below the initial mesh.No-GT pseudo-depth fitting performs on par with GT-range scaling when pose is GT.Source: Robustness to camera pose and depth · pp. 12–13, Table 9 · p. 12
• On YCB-V, refinement improves symmetric objects and partially visible objects, while asymmetric objects change little and the mostly visible set yields smaller average gains.The evaluation uses sensor depth, annotated camera poses, and guarded ICP.Source: Table 10 · p. 12, Real-world generalization · p. 13

4.8 Qualitative Results

• Across categories, source-view meshes show cleaner silhouettes and flatter planes, while thin or dense structures can distort and imperfect alignment biases occluded regions.The qualitative figures show the input view in the center and novel viewpoints on the sides.Source: §4.8 · pp. 13–14, Figure 2 · p. 13, Figure 3 · p. 14

5 Conclusion

• The framework refines any RGB-to-3D reconstructor without retraining by matching depth, mirroring corrections through symmetry, and propagating them with a smoothness solver.Each stage is closed form.Source: §5 · p. 14
• Across five backbones and two datasets, the method yields consistent gains that persist with pseudo-depth, with larger symmetry gains on symmetric objects and favorable accuracy and runtime against prior refinement.Source: §5 · p. 14
• Future work includes learned completion for severe occlusions and improved symmetry-plane confidence.Source: §5 · p. 14

Camera pose and intrinsics

• On GSO, CD improvement declines from −7.19 to −2.04 at ±10° polar rotation and is −2.00 at +20% focal inflation; refinement remains beneficial at every tested perturbation.Symmetry-guided refinement is best at every level.Source: Camera pose and intrinsics · p. 19, Table 11 · p. 19

Depth perturbation

• On GSO, depth noise up to 5%, blur, and up to 20% missing pixels change CD by at most 0.002, but a 5% global scale error drops results below the initial mesh.A 2% scale error costs 0.002–0.004 CD; affine fitting removes the scale error at a cost of 0.0003 CD.Source: Depth perturbation · p. 19, Table 12 · p. 19

Depth-to-vertex density

• With the sparsest depth, matching covers only 8% of visible vertices yet CD still drops by 25%; across tested densities, CD stays within 0.0203–0.0212.Sparse depth yields fewer handles, not wrong displacements.Source: Depth-to-vertex density · p. 20, Table 13 · p. 20

Symmetry plane detection

• The plane-detection score has a single clear minimum at the correct plane as candidate planes are swept, supporting stable selection.Figure 4 visualizes the signed-distance maps for candidate planes.Source: Figure 4 · p. 20, Symmetry plane detection · p. 21

Symmetry-plane source

• Reflect3D’s learned plane hurts refinement on valid GSO cases; the camera-frame search returns the left/right plane at about 0.27 s per object without an extra network.For objects with several symmetries, Reflect3D’s top-1 plane may be a top or front plane.Source: Symmetry-plane source · p. 21, Table 14 · p. 21

Failure analysis

• Despite mean gains on symmetric objects and at most +0.0008 CD on asymmetric ones, 16–33% and 21–48% respectively degrade, and the plane-score margin does not predict degradation.Even an oracle per-object choice gains only 0.0007 CD.Source: Failure analysis · p. 21

• The 24 rendered views use 12 azimuths spaced 15° apart, from −82.5° to 82.5°.Source: Rendering and views · p. 21
• The auction solver uses ε = 10−3 of the cost scale and up to 12 ε-scaling rounds in both stages.Source: Depth- and symmetry-guided stages · p. 21
• Table 15 reports the per-object effect of the symmetry stage on GSO.Source: Table 15 · p. 22
• Plane detection scores 1,260 candidates across 180 azimuths and seven elevations, rasterizing two 512 × 512 maps per candidate.Source: Depth- and symmetry-guided stages · p. 22
• Propagation sets boundary-anchor weight γ = 10−3 and Tikhonov weight τ = 10−6 of the mean diagonal of Lff, with Taubin parameters λ = 0.5 and ν = −0.5.Source: Propagation stage · p. 22

D Evaluation Details

• The protocol searches rotation coarse-to-fine without ICP, evaluates SSIM and AlexNet LPIPS, and Table 16 compares GT renders at input and novel views on Pix3D.The search scores a 243 grid on one-quarter of sampled points, then a 483 grid within the 64 best coarse cells.Source: D Evaluation Details · p. 22, Table 16 · p. 23
• REFINE uses 400 iterations per instance, while MeTTA takes about 30 minutes on an RTX 3090; the pseudo-depth setup fits UniDepth-V2 to the initial mesh render.For Table 9, no-GT pseudo-depth is fit over overlapping pixels, and FoundationPose uses default settings.Source: Test-time refinement baselines · p. 22, Robustness to depth and pose · p. 23
• With GT depth, refinement improves the initial mesh at the input view and matches or improves it at novel views except for CLIP; pseudo-depth matches only at input and falls below at novel views.Against GT CAD renders, MeTTA falls below the initial mesh at every view, despite scoring better against the photo at the input view.Source: Novel-view evaluation against MeTTA · p. 22, Table 16 · p. 23
• For RGB-D baselines, the symmetry stage brings no gain; depth-only refinement is recommended when the backbone already conditions its occluded side on depth.Source: RGB-D baselines · p. 23
• On YCB-V, 12 of 21 objects and 54% of instances improve, while rigid alignment residual and initial CD 1.6× that on GSO limit the gain.Source: YCB-V · p. 23