Source-linked AI summary
SplAttN: Bridging 2D and 3D with Gaussian Soft Splatting and Attention for Point Cloud Completion
Zhaoyang Li, Zhichao You, Tianrui Li
TL;DR
Point cloud completion lacks clear evidence explaining how multimodal connections improve learning, and hard projection can collapse visual support into sparse image-plane features. SplAttN uses Differentiable Gaussian Splatting as continuous density estimation to connect geometric and visual streams, achieving state-of-the-art benchmark performance and robust visual dependence in KITTI counter-factual tests.
Problem
The theoretical basis of multimodal point cloud completion is under-explored, while hard projection can cause Cross-Modal Entropy Collapse that limits learnable 2D–3D connection.
Method
SplAttN replaces hard projection with Differentiable Gaussian Splatting to create a dense continuous visual representation and enable geometric querying of visual priors.
Results
SplAttN achieves state-of-the-art performance on PCN and ShapeNet-55/34, while KITTI counter-factual evaluation shows robust dependence on visual cues unlike baseline methods.
Takeaways & Limitations
The counter-factual results support SplAttN as establishing an effective cross-modal connection rather than decoupling generation from observation.
Takeaways & Limitations
The formulation assumes a known projection π : R3 → Ω ⊂ R2 for analyzing cross-modal gradient flow.
Abstract
from arXiv · showhide
Although multi-modal learning has advanced point cloud completion, the theoretical mechanisms remain unclear. Recent works attribute success to the connection between modalities, yet we identify that standard hard projection severs this connection: projecting a sparse point cloud onto the image plane yields an extremely sparse support, which hinders visual prior propagation, a failure mode we term Cross-Modal Entropy Collapse. To address this practical limitation, we propose SplAttN, which replaces hard projection with Differentiable Gaussian Splatting to produce a dense, continuous image-plane representation. By reformulating projection as continuous density estimation, SplAttN avoids collapsed sparse support, facilitates gradient flow, and improves cross-modal connection learnability. Extensive experiments show that SplAttN achieves state-of-the-art performance on PCN and ShapeNet-55/34. Crucially, we utilize the real-world KITTI benchmark as a stress test for multi-modal reliance. Counter-factual evaluation reveals that while baselines degenerate into unimodal template retrievers insensitive to visual removal, SplAttN maintains a robust dependency on visual cues, validating that our method establishes an effective cross-modal connection. Code is available at https://github.com/zay002/SplAttN.
1. Introduction
SplAttN addresses the under-explored theoretical basis of multimodal point cloud completion by replacing sparse hard projection with differentiable Gaussian splatting. It evaluates whether this design creates genuine cross-modal reliance rather than unimodal template retrieval.
- Multimodal completion uses 2D images as semantic priors, but the statistical advantages of combining modalities remain under-explored.
- Hard projection can induce Cross-Modal Entropy Collapse by mapping continuous 3D geometry onto sparse 2D support.The resulting feature divergence impedes gradient flow and limits learning of the connection between visual and geometric spaces.
- SplAttN replaces hard projection with Differentiable Gaussian Splatting, producing a dense continuous visual representation for geometric querying.The method treats projection as probabilistic density estimation and maps vertices to spatially coherent visual density.
- The architecture combines a GS-Bridge and Local Encoder with a Global-Local Decoder for global semantics, topology-aware details, and hierarchical refinement.The decoder predicts a sparse skeleton before progressively upsampling while injecting local features through cross-attention.
- SplAttN achieves state-of-the-art performance on PCN and ShapeNet-55/34, while KITTI counter-factual tests show robust visual dependency.Baselines show negligible sensitivity to visual removal and degenerate into unimodal template retrievers.
2. Related Works
Prior completion methods range from structure-based and transformer architectures to multimodal fusion, generative models, differentiable splatting, and visual backbones. SplAttN is positioned against hard-projection fusion by using continuous feature-density representations to connect 2D and 3D information.
- Structure-based methods use folding, tree decoders, grids, iterative refinement, aggregation, point paths, and keypoint alignment to improve completion.
- Transformer architectures recast completion as set-to-set translation and explore coarse-to-fine generation, discriminative nodes, attention, cross-resolution modeling, and state-space models.Single-modal methods remain challenged by semantic ambiguity under severe occlusion.
- Multimodal fusion introduces 2D semantic priors through view guidance, vision-language models, and fusion modules to resolve geometric ambiguity.
- SVDFormer and GeoFormer use deterministic hard projection, which the paper associates with feature sparsity, impaired gradient flow, and unimodal template reliance.
- Diffusion models achieve high fidelity but incur expensive iterative denoising, limiting real-time applicability.
- Differentiable splatting propagates gradients from pixels to geometry and can transform discrete point signals into continuous differentiable feature manifolds.SplAttN repurposes Gaussian-splatting concepts for feature-density estimation across modalities.
- Visual backbones have evolved from CNNs to Transformers, MAE, and TinyViT, while soft splatting supports transferring 2D semantic priors to irregular 3D point features.
3. Method
SplAttN models point-cloud completion as a differentiable mapping between sparse geometry and dense visual features. Its Gaussian Splatting Bridge expands projection into continuous density estimation, while hybrid geometric encoding and global-local decoding combine local detail with global structure.
- Problem formulation: SplAttN maps partial point clouds and RGB priors to completed point clouds while explicitly distinguishing points, projected coordinates, and continuous visual queries.The formulation treats cross-modal alignment as a differentiability-critical design choice.
- Theoretical analysis: Hard projection represents visual queries with Dirac-delta support, causing zero-almost-everywhere gradients and preventing geometric updates from visual supervision.Its zero-measure support produces Cross-Modal Entropy Collapse.
- Theoretical analysis: Differentiable Gaussian Splatting replaces hard projection with a continuous Gaussian density field whose effective support extends within 3σ of projected points.The resulting non-degenerate probability field has non-vanishing gradients and restores a learnable cross-modal connection.
- Gaussian Splatting Bridge: The Gaussian Splatting Bridge converts discrete visual features into a continuous density field and aggregates projected primitives at arbitrary sub-pixel queries using normalized soft weights.Each primitive contributes a feature, concretely instantiated in the CCM implementation as a three-channel pseudo-color from normalized 3D coordinates.
- Hybrid geometric tokenization: Hybrid geometric tokenization combines EdgeConv-based local surface primitives with Transformer self-attention for global topology and long-range dependencies.The resulting Fgeo encodes fine-grained geometry and global shape semantics.
- Gaussian Splatting Bridge: Spatial kernels smooth quantization and misalignment noise while preserving gradients, and inverse-depth weighting prioritizes foreground geometry through a differentiable Z-buffer.The Gaussian tail keeps gradients non-vanishing for slightly misaligned points.
- Attention and decoding: Active Attention uses geometric features as Queries and the visual manifold as Keys and Values to retrieve relevant visual context for specific 3D parts.The Global-Local Decoder then hierarchically densifies the coarse skeleton, using uncertainty-aware and similarity-based branches to combine global guidance with local refinement.
4. Experiment
SplAttN is evaluated on PCN, ShapeNet-55/34, and KITTI, combining benchmark performance, ablations, qualitative comparisons, and a counterfactual test of visual dependence. It achieves strong reconstruction results while showing substantially greater sensitivity to visual cues than baselines under KITTI distribution shift.
- KITTI stress test: KITTI evaluates whether models retain cross-modal reliance when a PCN-trained model is applied to real-world car instances without fine-tuning.
- Benchmark evaluation: SplAttN achieves state-of-the-art PCN performance with an average CD of 6.36.
- Benchmark evaluation: On ShapeNet-55, SplAttN obtains the highest F1-Score of 0.520 and surpasses SVDFormer with an average CD of 0.77.
- Benchmark evaluation: On ShapeNet-34/21, SplAttN achieves the best F1-Scores of 0.533 and lowest average CD on seen and unseen splits, at 0.65 and 1.22.
- KITTI stress test: Removing visual input changes SVDFormer performance by only +0.4%, whereas SplAttN’s consistency drops by −26.1%.GeoFormer instead gains +20.9% without images, while SplAttN reaches average CMIT 200.5.
- Ablation study: Ablations show that differentiable splatting, explicit 3D coordinates, hybrid attention, and a 5M TinyViT encoder improve the PCN configuration.Differentiable splatting reaches CD 6.36; CCM reaches 6.41 versus Depth at 6.43, while the 21M encoder degrades performance to 6.42.
5. Conclusion
The paper concludes that differentiable density estimation addresses Cross-Modal Entropy Collapse and bridges sparse geometric and visual representations. PCN and ShapeNet-55/34 results are accompanied by KITTI counterfactual evidence of stronger cross-modal reliance than baselines.
- Differentiable density estimation expands 2D information support and bridges the gradient gap in sparse geometric completion.
- SplAttN achieves state-of-the-art performance on PCN and ShapeNet-55/34 while establishing a robust cross-modal connection on KITTI.
- Future work targets unsupervised domain adaptation, lightweight backbones, and more efficient inter-modal fusion.
Impact Statement
The work aims to advance 3D point cloud completion and multi-modal learning. Its stated potential societal applications include autonomous driving perception and robotic manipulation.
- The paper targets advances in 3D point cloud completion and multi-modal learning.
- Potential applications include autonomous driving perception systems and robotic manipulation, with possible safety and efficiency benefits.
- The authors identify no specific ethical issues requiring emphasis.
A. Additional Qualitative Results on ShapeNet-55
Qualitative ShapeNet-55 results span easy, median, and hard difficulty levels. Across these settings, SplAttN is reported to preserve fine-grained structures and recover missing regions more faithfully than baselines.
- Qualitative comparisons: Across Easy, Median, and Hard splits, SplAttN preserves fine-grained geometry and recovers missing regions more accurately than baselines.
- Qualitative comparisons: Baseline methods such as SVDFormer often oversmooth shapes or lose local details, whereas SplAttN faithfully recovers blurred details.
- Evaluation metrics: The qualitative evaluation reports Average CD and F-Score@1% across the difficulty levels.
C. Additional Theoretical Analysis
The analysis frames SplAttN's projection as differentiable density estimation that expands spatial support and supports cross-modal information flow. It also introduces evaluation measures and notes a Sim-to-Real limitation of template-based plausibility metrics.
- Hard projection produces isolated, zero-measure support, so off-support visual queries receive no smooth response for gradient-based cross-modal alignment.
- Gaussian splatting replaces the Dirac measure with a smooth, depth-aware kernel mixture whose normalized weights aggregate projected features.The contribution weight is nonnegative, and normalization occurs over the image-plane domain.
- SplAttN expands per-channel support through Gaussian kernels, increasing the joint information density passed to the visual backbone.
- CMIT multiplies channel-aware entropy by spatial coverage to proxy total information yield, and SplAttN achieves orders-of-magnitude higher CMIT than baselines on KITTI.
- MMD is unreliable for Sim-to-Real KITTI evaluation because matching noisy, sparse LiDAR scans to synthetic holistic templates introduces domain bias and high computational cost.The protocol also normalizes KITTI pose to isolate reconstruction capability from arbitrary pose variation.
- SCS uses a pretrained DGCNN confidence score to assess whether completed point clouds preserve recognizable semantic structure.Higher SCS indicates higher-fidelity semantic features recognizable by the classifier.
F. Additional Entropy Collapse Analysis
Additional PCN visualizations compare hard and soft-splatted feature representations. They show that SplAttN maintains denser feature coverage and a broader value distribution across samples.
- SplAttN maintains high feature coverage across PCN samples, mitigating entropy collapse relative to Hard Depth and Hard CCM.
- Figure 12 compares dense feature maps from the proposed method with sparse baseline representations.
- Figure 13 uses histograms to show that SplAttN produces a broader feature-value distribution than the baselines.
G. Additional KITTI Qualitative Results
Additional KITTI visualizations provide qualitative comparisons on sparse, noisy real-world scans. Their reconstruction differences are consistent with rankings from the Semantic Consistency Score.
- KITTI reconstructions visually follow the performance rankings produced by the Semantic Consistency Score.
- KITTI is a challenging stress test because its scans are substantially sparser and noisier than synthetic benchmark data.
H. Additional KITTI Robustness Analysis
The robustness analysis examines projection under KITTI's sim-to-real shift. Hard projection leaves little active support, whereas Gaussian splatting substantially increases coverage while preserving sparse geometric structure.
- Figure 15 compares intermediate feature representations to assess information retention under domain shift.
- KITTI LiDAR scans are extremely sparse, and deterministic hard projection can reduce active pixel coverage below 10%, including 5.3% in the Front View.
- SplAttN increases valid feature coverage by approximately 4.3×, reaching 25.7% in the Top View through continuous density estimation.
- Figure 16 visualizes point-cloud projection and feature-map coverage for KITTI robustness analysis.