Source-linked AI summary
View-Guided Point Cloud Completion
Xuancheng Zhang, Yutong Feng, Siqi Li, Changqing Zou, Hai Wan, Xibin Zhao, Yandong Guo, Yue Gao
TL;DR
Point-cloud completion is uncertain when scans are incomplete and single-modality input cannot resolve missing structure. ViPC uses an extra single-view image and staged cross-modality and cross-level fusion; it significantly improves completion quality over existing single-modality methods on a new large-scale dataset.
Problem
Large incompleteness makes missing points uncertain, while sparse, unstructured point clouds make it difficult to distinguish inherent sparsity from missing data.
Source: §1 · p. 1Method
ViPC uses a single-view image for global structural guidance, transfers image information into point-cloud space, and differentially refines coarse and fine parts.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
An extra single-view image provides crucial global structural prior information for completing a partial point cloud.
Source: §1 · p. 2Rendered textures do not reflect real-world illumination, producing poor-quality transferred point clouds and unsatisfactory real-world completion; training on real-world data may help.
Source: §5.3 · p. 8
Abstract
from arXiv · showhide
This paper presents a view-guided solution for the task of point cloud completion. Unlike most existing methods directly inferring the missing points using shape priors, we address this task by introducing ViPC (view-guided point cloud completion) that takes the missing crucial global structure information from an extra single-view image. By leveraging a framework that sequentially performs effective cross-modality and cross-level fusions, our method achieves significantly superior results over typical existing solutions on a new large-scale dataset we collect for the view-guided point cloud completion task.
1. Introduction
ViPC uses an extra single-view image to supply global structure for point-cloud completion, combining cross-modality and cross-level fusion. On ShapeNet-ViPC, it significantly outperforms existing state-of-the-art approaches.
1. Introduction
- Point-cloud completion is difficult when large portions are missing because single-modality data leaves uncertainty about missing points and whether blank space reflects sparsity or incompleteness.Source: §1 · p. 1
- Its three-stage framework first addresses cross-modality fusion, then cross-level fusion between local point-cloud details and image-derived global structure.It reconstructs a coarse point cloud from the image and transfers completion-relevant information into the point-cloud domain.Source: §1 · p. 2
- On ShapeNet-ViPC, ViPC significantly outperforms existing state-of-the-art approaches.The dataset contains 38,328 objects from 13 categories, with 24 sets of ground-truth data per object.Source: §1 · p. 2
- ViPC uses an extra single-view image to provide the crucial global structural prior for completing a partial point cloud.Source: §1 · pp. 1–2
- The Dynamic Offset Predictor differentially refines points, applying light refinement to higher-quality points and heavy refinement to lower-quality points.Source: §1 · p. 2
- ShapeNet-ViPC is a large-scale completion dataset built on ShapeNet to simulate occlusion-related defects and support future benchmarking.Source: §1 · p. 2
2. Related Works
Related work groups point-cloud completion methods into geometry-based, alignment-based, and learning-based approaches. Each offers different strengths and limitations, while auxiliary data has rarely been used to supplement point-cloud input.
Geometry-based Methods
- Geometry-based methods infer missing shape from geometric assumptions or regularities, but rely on hand-crafted priors and apply only to models with small incompleteness.Source: §2 · p. 2
Alignment-based Methods
- Alignment-based methods retrieve and align database models or primitives; they handle varied models and incompleteness but incur high inference and database costs and are noise-sensitive.Source: §2 · p. 2
Learning-based Methods
- Learning-based methods map incomplete to complete shapes, but mono-modal inputs make large-scale incompleteness difficult and auxiliary data has rarely supplemented point-cloud input.These methods may perform well only on particular object categories or shapes with small-scale incompleteness.Source: §2 · p. 2
3. Methods
ViPC uses a three-stage pipeline to fuse a partial point cloud with a single-view image, transferring image information into 3D before filtering and refining points. It preserves input-derived detail during refinement while using image-derived structure to complete the shape.
3.1. Overview
- The three-stage pipeline transfers image information into a point-cloud representation, filters a coarse cloud into Fine and Coarse parts, then refines the Coarse part while constraining the Fine part.The framework addresses cross-modality fusion first and cross-level fusion across the later stages.Source: §3.1 · p. 3
3.2. Modality Transfer
- Modality Transfer reconstructs a coarse point cloud from the single-view image, aligns it with the partial input in 3D, and preserves image feature maps for later guidance.The image encoder maps the view to a latent vector, and the decoder outputs point coordinates.Source: §3.2 · p. 3
3.3. Part Filter
- Part Filter merges the partial and reconstructed clouds, uses farthest-point sampling for a coarse representation, and separates input-near Fine points from the remaining Coarse points.Fine points are selected using proximity to the input cloud under an adaptive density-based threshold.Source: §3.3 · pp. 3–4
3.4. Part Refinement
- Part Refinement combines point coordinates with four guidance-feature types and predicts offsets for upsampling, using a protective mask to limit Fine-part movement.Global 3D features come from PointNet applied to the partial and reconstructed clouds; image-map features and a 2D grid provide additional guidance.Source: §3.4 · p. 4
3.5. Loss Function
- Training combines symmetric Chamfer Distance, which encourages proximity and coverage, with Earth Mover’s Distance using trade-off weights α and β.The default weights are α = 1 and β = 0.0001.Source: §3.5 · p. 5
4. Experimental Settings
ShapeNet-ViPC simulates incomplete point-cloud acquisition across categories and viewpoints, with both noisy and noise-free inputs. Experiments use a subset of the dataset and evaluate completion with Chamfer Distance and F-Score.
4.1. ShapeNet-ViPC
- ShapeNet-ViPC contains 38,328 objects across 13 categories, with two incomplete-cloud scenarios generated at 24 viewpoints and 2,048-point complete ground truth.The scenarios use point clouds with or without device noise; each set also includes a view-aligned image.Source: §4.1 · p. 5, Figure 5 · p. 5
- Experiments use 31,650 objects from eight categories, split 80% for training and 20% for testing.These objects yield 759,600 experimental sets.Source: §4.1 · p. 5
4.2. Implementation details and evaluation metrics
- Completion is evaluated with Chamfer Distance and F-Score, where lower CD and higher F-Score indicate better quality.The input image is 224 × 224, the partial cloud contains 2,048 points, and the completed output contains 2,048 points.Source: §4.2 · p. 5
5. Experimental Results and Analysis
ViPC is evaluated against point-cloud completion baselines, through analyses of image-view contributions and component ablations, and in two settings that expose limitations.
5.1. Comparisons
- ViPC consistently outperforms the compared baselines across eight categories on both CD and F-Score; its advantages are greater for airplane, watercraft, lamp, and car.The comparison uses AtlasNet, FoldingNet, PCN, and TopNet; the tables report results on 2,048-point clouds.Source: §5.1 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
- Qualitatively, ViPC shows better results than baselines across eight categories while preserving input details and producing more reasonable completions of missing structures.The method uses points from the Fine part as a constraint while inferring a local distribution for the Coarse part.Source: §5.1 · p. 6, Figure 6 · p. 7
5.2. Experimental Analysis
- The ablation study compares outputs from Modality Transfer, Part Filter, and Part Refinement, and evaluates differential refinement by comparing Pglobal with Pcomplete.Pglobal disables the constraint that Fine-part points move only within a small sphere.Source: §5.2 · p. 6, Table 3 · p. 8
- Different reference views yield different completion quality, and views that provide more information about the missing part produce better results.The analysis evaluates 24 rendered views for each of 400 selected partial point clouds and reports average CD beneath representative views.Source: §5.2 · p. 7, Figure 7 · p. 7
- The complete ViPC framework outperforms PSG by a large margin, although PSG produces better results than the Modality Transfer network alone.The authors suggest that using a more effective network such as PSG for Modality Transfer may improve the solution.Source: Table 4 · p. 8, §5.2 · p. 8
- Among the tested Part Refinement features, contributions rank xyz > fpartial ≈ frecon > fpixel > fgrid.The comparison removes each feature from the fused representation in turn.Source: Table 5 · p. 8, §5.2 · p. 8
5.3. Limitations
- For non-calibrated devices, accurate alignment is difficult because reconstructed point clouds are sparse and partial inputs are incomplete.The paper reports trying ICP and ICP-MCC and identifies learning-based registration as future work.Source: §5.3 · p. 8
- In real-world scenes, training on rendered images whose textures do not reflect real illumination leads to poor Modality Transfer reconstructions and unsatisfactory completion.The authors suggest that training on real-world data may address this issue.Source: §5.3 · p. 8
6. Conclusion
ViPC uses an extra single-view image to supply missing global structure for point cloud completion. Its Dynamic Offset Predictor refines a coarse point cloud, and comparisons on a new large-scale dataset show significant quality improvement over existing single-modality methods.
6. Conclusion
- ViPC uses global structure from an extra single-view image to complete a partial point cloud.Source: §6 · p. 8
- The Dynamic Offset Predictor differentially refines points in a coarse point cloud.Source: §6 · p. 8
- ViPC demonstrates significant quality improvement over existing single-modality methods on a new large-scale dataset.Source: §6 · p. 8