PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-PCQA: Multi-Modal Learning for No-reference Point Cloud Quality Assessment

Zicheng Zhang, Wei Sun, Xiongkuo Min, Quan Zhou, Jun He, Qiyuan Wang, Guangtao Zhai

arXiv:2209.00244v2cs.CVeess.IV

TL;DR

PCQA supports quality-aware simplification and compression, but no-reference methods generally rely on either 3D point-cloud features or 2D projections. MM-PCQA combines sub-model geometry and rendered-image features with symmetric cross-modal attention, and reports the best performance among compared methods across three databases.

  • Problem

    No-reference PCQA matters because pristine reference point clouds are not always available, while existing methods generally use either point-cloud or projected-image features rather than both.

    Source: §1 · p. 1

  • Method

    MM-PCQA encodes point-cloud sub-models and rendered image projections separately, then uses symmetric cross-modal attention to combine their quality-aware features.

    Source: §1 · p. 2, §3.4 · p. 4

  • Results

    Across all three databases, MM-PCQA achieves the best performance; on WPC it records 0.84 vs. 0.70 (PQA-net) in SRCC.

    Source: Performance Discussion · p. 6, §5 · p. 7

  • Takeaways & Limitations

    The results support combining modalities that capture different distortion cues, with point clouds revealing geometry damage and images reflecting texture distortions.

    Source: §1 · p. 2

Abstract

from arXiv · show

The visual quality of point clouds has been greatly emphasized since the ever-increasing 3D vision applications are expected to provide cost-effective and high-quality experiences for users. Looking back on the development of point cloud quality assessment (PCQA) methods, the visual quality is usually evaluated by utilizing single-modal information, i.e., either extracted from the 2D projections or 3D point cloud. The 2D projections contain rich texture and semantic information but are highly dependent on viewpoints, while the 3D point clouds are more sensitive to geometry distortions and invariant to viewpoints. Therefore, to leverage the advantages of both point cloud and projected image modalities, we propose a novel no-reference point cloud quality assessment (NR-PCQA) metric in a multi-modal fashion. In specific, we split the point clouds into sub-models to represent local geometry distortions such as point shift and down-sampling. Then we render the point clouds into 2D image projections for texture feature extraction. To achieve the goals, the sub-models and projected images are encoded with point-based and image-based neural networks. Finally, symmetric cross-modal attention is employed to fuse multi-modal quality-aware information. Experimental results show that our approach outperforms all compared state-of-the-art methods and is far ahead of previous NR-PCQA methods, which highlights the effectiveness of the proposed method. The code is available at https://github.com/zzc-1998/MM-PCQA.

1 Introduction

Point cloud quality assessment (PCQA) matters for guiding simplification and compression while protecting users’ quality of experience, but it remains less studied than other point-cloud tasks. Existing no-reference methods use either 3D point-cloud or 2D projection features, motivating MM-PCQA’s joint use of both modalities.

1 Introduction

  • PCQA predicts point-cloud visual quality and can guide simplification and compression without negatively affecting users’ quality of experience.The paper identifies metaverse and VR/AR as application settings.Source: §1 · p. 1
  • Existing NR-PCQA methods use either 3D point-cloud features or 2D projected-image features, and the paper says they fail to use both jointly.Point clouds better reveal structural damage and geometry down-sampling, while images are more sensitive to texture distortions and provide semantic information more easily.Source: §1 · pp. 1–2
  • MM-PCQA jointly uses point-cloud and projected-image features, with sub-models for local geometry, rendered views for texture, and symmetric cross-modal attention to combine quality-aware features.The method encodes the two modalities separately and decodes their fused features into quality values.Source: §1 · p. 2
  • MM-PCQA achieves the best performance among compared state-of-the-art methods, including full-reference methods, and its ablations support contributions from both modalities, patch-up, and cross-modal attention.The introduction reports that the method is far ahead of previous NR-PCQA methods.Source: §1 · p. 2

2 Related Work

Prior PCQA work includes both reference-based geometric metrics and no-reference methods built on point-cloud or projected-image features. Related multimodal 3D-vision work chiefly combines point clouds and images for detection, rather than PCQA.

2.1 Quality Assessment for Point Cloud

  • Full-reference PCQA progressed from point-level metrics such as p2point [Mekuria et al., 2016a] and p2plane to structural-feature metrics including PCQM, GraphSIM [Yang et al., 2020b], and PointSSIM.PSNR-yuv was introduced for colored point clouds because point-level differences struggle to reflect complex distortions.Source: §2.1 · p. 2
  • No-reference PCQA spans patch-wise handcrafted features, multi-view projections, statistical geometry and color features, video sequences, sparse CNNs, and image-domain adaptation, but these methods use single-modal information.The passage names methods by Chetouani et al., PQA-net [Liu et al., 2021b], Zhang et al. [Zhang et al., 2022c], Fan et al. [Fan et al., 2022b], Liu et al. [Liu et al., 2022c], and Yang et al. [Yang et al., 2022].Source: §2.1 · p. 2

2.2 Multi-modal Learning for Point Cloud

  • Prior point-cloud–image multimodal work mainly targets 3D detection, using mechanisms such as bird’s-eye-view region proposals, point-wise fusion, and gated fusion.Other cited work transfers camera-view features through auto-calibrated projection or learns multimodal representations self-supervised for downstream tasks.Source: §2.2 · p. 2

3 Proposed Method

MM-PCQA combines local 3D geometry features with rendered 2D image features, fuses them through symmetric cross-modality attention, and regresses a quality score. Its feature extraction preserves local point patterns, while its loss combines value fitting with quality ranking.

3 Proposed Method

  • The pipeline encodes segmented point-cloud sub-models and rendered projections, applies symmetric cross-modality attention, then concatenates features for quality regression.Source: §3 · p. 3

3.1 Preliminaries

  • The 3D modality normalizes geometry coordinates and excludes color, while rendered images are generated from the colored point cloud.Source: §3.1 · p. 3

3.2 Point Cloud Feature Extraction

  • Farthest-point anchors and KNN neighborhoods form sub-models, avoiding down-sampling that can damage geometry patterns and retaining local smoothness and roughness.Source: §3.2 · pp. 3–4

3.3 Image Feature Extraction

  • The image branch renders N_I views of distorted colored clouds from a fixed-distance circular path, then encodes and average-fuses projection features.Source: §3.3 · p. 4

3.4 Symmetric Cross-Modality Attention

  • Symmetric cross-modality attention lets each modality guide the other's attention learning; final quality features concatenate both intra-modal and guided features.Source: §3.4 · pp. 4–5

3.5 Quality Regression & Loss Function

  • A fully connected regressor predicts quality scores; training combines MSE for label-value fit with rank loss to distinguish clouds with close quality labels.Source: §3.5 · p. 5

4 Experiments

Experiments use three subjective PCQA databases with distinct reference-cloud counts and distortion settings. The reported setup standardizes training, testing, and model inputs across evaluations.

4.1 Databases

  • SJTU-PCQA contains 378 distorted clouds from 9 references, WPC contains 740 from 20 references, and WPC2.0 contains 400 from 16 references.The databases use different distortion settings: SJTU-PCQA has seven types at six strengths, WPC covers four types, and WPC2.0 uses 25 V-PCC settings.Source: §4.1 · p. 5

4.2 Implementation Details

  • The model uses 6 sub-models of 2,048 points each and 4 image projections, with PointNet++ and ResNet50 as the respective encoders.Projected images are cropped to 224×224×3 patches; ResNet50 is initialized from ImageNet pretraining.Source: §4.2 · p. 5
  • Evaluation uses 9-, 5-, and 4-fold cross-validation for SJTU-PCQA, WPC, and WPC2.0, with no reference-cloud content overlap between training and testing.Baselines are retrained using the same database splits, and average performance is reported.Source: §4.2 · p. 5

4.3 Competitors and Evaluation Criteria

The evaluation compares MM-PCQA with full-reference and no-reference methods using four standard correlation and error criteria. MM-PCQA leads the compared methods on all three databases and shows a smaller reported performance drop under more complex distortions.

4.3 Competitors and Evaluation Criteria

  • The evaluation includes 14 methods—8 FR-PCQA and 6 NR-PCQA—and reports SRCC, KRCC, PLCC, and RMSE against subjective MOS ratings.A five-parameter logistic function maps predicted scores to subjective ratings.Source: §4.3 · pp. 5–6

Performance Discussion

  • MM-PCQA leads NR-PCQA methods in SRCC on all three databases: 0.91 vs. 0.86 (ResSCNN) on SJTU-PCQA, 0.84 vs. 0.70 (PQA-net) on WPC, and 0.80 vs. 0.75 (ResSCNN) on WPC2.0.Table 1 compares state-of-the-art approaches on the three databases.Source: Performance Discussion · p. 6, Table 1 · p. 6
  • From SJTU-PCQA to WPC, MM-PCQA's SRCC and PLCC drop by 0.07 and 0.08, while most top-performing methods other than 3D-NSS drop by over 0.1 on both measures.The text attributes the more challenging WPC and WPC2.0 results to their more complex distortion settings.Source: Performance Discussion · p. 6
  • BRISQUE, NIQE, and IL-NIQE perform poorest among the compared methods; the text notes they were designed for natural-scene images rather than point-cloud projections.The authors interpret the overall experimental performance as validating multi-modal learning for point-cloud visual quality assessment.Source: Performance Discussion · p. 6

4.4 Contributions of the Modalities

The ablation compares single-modality and multi-modal models, then tests the symmetric cross-modality attention module. Both modalities contribute, and adding SCMA further boosts performance.

4.4 Contributions of the Modalities

  • The model using both point-cloud and image features outperforms either single-modal model.The passage interprets this as evidence that both feature types contribute to the final results.Source: §4.4 · p. 6
  • The point-cloud-only and image-only variants each perform worse than the multi-modal model.The ablation is presented as a test of the authors' hypothesis that multi-modal learning improves quality representation.Source: §4.4 · p. 6
  • Adding SCMA further boosts performance over the multi-modal model without that module.The authors take this as validation of cross-modality attention's effectiveness.Source: §4.4 · p. 6

4.5 Ablation for the Patch-up Strategy

The patch-up strategy is compared with farthest point sampling (FPS) using an equal point count. The reported results favor patch-up for point-cloud-only features, with local-pattern preservation offered as the rationale.

4.5 Ablation for the Patch-up Strategy

  • The FPS comparison samples 12,288 points per cloud, matching the 6 sub-models of 2,048 points in the default setup.The passage explains that sampled points may not preserve local patterns important for quality assessment.Source: §4.5 · pp. 6–7
  • With only point-cloud features, patch-up greatly improves performance over FPS.Table 3 compares the FPS and patch-up strategies for point-cloud-only and point-cloud-plus-image features.Source: §4.5 · p. 7, Table 3 · p. 7
  • For clouds with more points, sampled points may fail to preserve the object's main shape unless the sampling count is greatly increased.Source: §4.5 · p. 7

4.6 Cross-database Evaluation

Cross-database evaluation finds that MM-PCQA transfers learned representations from WPC better than the most competitive NR-PCQA methods, including a WPC-trained model that surpasses competitors trained directly on WPC2.0.

4.6 Cross-database Evaluation

  • WPC→WPC2.0 MM-PCQA is even superior to all competitors directly trained on WPC2.0.The evaluation trains mainly on WPC (740) and validates on SJTU-PCQA (378) and WPC2.0 (400).Source: §4.6 · p. 7
  • MM-PCQA better generalizes learned feature representation from WPC than the most competitive NR-PCQA methods.WPC is the largest database in this evaluation, with 740 samples.Source: §4.6 · p. 7
  • The cross-database evaluation tests generalization by training mainly on WPC and validating on SJTU-PCQA and WPC2.0.The databases contain 740, 378, and 400 samples, respectively.Source: §4.6 · p. 7

4.7 Number of 2D Projections and 3D sub-models

The branch study varies the numbers of 3D sub-models and 2D projections; six sub-models and four projections yield the best performance on major aspects, while increasing counts may introduce redundancy.

4.7 Number of 2D Projections and 3D sub-models

  • 6 sub-models and 4 projections yield the best performance on major aspects for the point cloud and image branches, respectively.The evaluation varies the number of sub-models and projections on SJTU-PCQA, WPC, and WPC2.0.Source: §4.7 · p. 7, Table 5 · p. 7
  • Increasing the numbers of sub-models and projections may make extracted features redundant, resulting in a performance drop.Source: §4.7 · p. 7
  • The branch comparison varies sub-model and projection counts across SJTU-PCQA, WPC, and WPC2.0.Table 5 presents these performance results.Source: Table 5 · p. 7

4.8 Different Feature Encoders

Among the tested feature-encoder combinations, PointNet++ with ResNet50 performs best, and the evaluated backbones remain competitive with other NR-PCQA methods.

4.8 Different Feature Encoders

  • PointNet++ and ResNet50 together are superior to the other encoder combinations tested.Source: §4.8 · p. 7
  • The evaluated image backbones are VGG16, MobileNetV2, and ResNet50, while the point-cloud backbones are PointNet++ and DGCNN.Source: §4.8 · p. 7
  • The different backbone combinations remain competitive compared with other NR-PCQA methods.The authors present this comparison as further confirmation of the framework's effectiveness.Source: §4.8 · p. 7

5 Conclusion

MM-PCQA combines local point-cloud geometry and projected-image texture features using PointNet++, ResNet50, and symmetric cross-modal attention; it achieves state-of-the-art results on three databases.

5 Conclusion

  • MM-PCQA reaches a new state-of-the-art on the SJTU-PCQA, WPC, and WPC2.0 databases.Source: §5 · p. 7
  • The method patches point clouds to preserve local geometry, uses projected images for texture distortions, and fuses features with symmetric cross-modal attention.PointNet++ and ResNet50 serve as the point-cloud and image feature encoders.Source: §5 · p. 7
  • Extensive ablation studies further demonstrate the potency of the proposed multi-modal learning framework.Source: §5 · p. 7
Loading 2209.00244v2…