PaperlayerView exact revision on arXiv

Source-linked AI summary

MVTN: Multi-View Transformation Network for 3D Shape Recognition

Abdullah Hamdi, Silvio Giancola, Bernard Ghanem

arXiv:2011.13244v3cs.CVcs.LG

TL;DR

Multi-view recognition commonly uses viewpoints fixed across all shapes, leaving viewpoint selection insufficiently adaptive. MVTN predicts shape-specific viewpoints through differentiable rendering and trains jointly with the recognition network, improving classification, retrieval, and robustness results across several benchmarks.

  • Problem

    Existing multi-view methods typically use heuristically selected viewpoints fixed for all shapes, limiting viewpoint adaptivity in 3D recognition.

  • Method

    MVTN encodes each shape to regress bounded viewpoints and optimizes them jointly with a multi-view network through differentiable mesh or point-cloud rendering.

  • Results

    MVTN improves 3D classification and retrieval, reaching state-of-the-art results including up to 6% classification improvement, 92.9% mAP on ModelNet40 retrieval, and 82.9 mAP on ShapeNet Core55.

  • Takeaways & Limitations

    Learned viewpoints provide a single end-to-end multi-view pipeline for mesh and point-cloud recognition, with reported robustness to rotation and occlusion.

Abstract

from arXiv · show

Multi-view projection methods have demonstrated their ability to reach state-of-the-art performance on 3D shape recognition. Those methods learn different ways to aggregate information from multiple views. However, the camera view-points for those views tend to be heuristically set and fixed for all shapes. To circumvent the lack of dynamism of current multi-view methods, we propose to learn those view-points. In particular, we introduce the Multi-View Transformation Network (MVTN) that regresses optimal view-points for 3D shape recognition, building upon advances in differentiable rendering. As a result, MVTN can be trained end-to-end along with any multi-view network for 3D shape classification. We integrate MVTN in a novel adaptive multi-view pipeline that can render either 3D meshes or point clouds. MVTN exhibits clear performance gains in the tasks of 3D shape classification and 3D shape retrieval without the need for extra training supervision. In these tasks, MVTN achieves state-of-the-art performance on ModelNet40, ShapeNet Core55, and the most recent and realistic ScanObjectNN dataset (up to 6% improvement). Interestingly, we also show that MVTN can provide network robustness against rotation and occlusion in the 3D domain. The code is available at https://github.com/ajhamdi/MVTN .

1. Introduction

Multi-view methods achieve strong 3D recognition by rendering several views, but their camera viewpoints are typically chosen heuristically and fixed across shapes. MVTN learns shape-specific viewpoints differentiably and jointly with the task network.

  • Motivation: Multi-view methods bridge 3D tasks and 2D convolutional learning by rendering several images per shape.They can also leverage large image datasets for pre-training when labeled 3D data are scarce.
  • Motivation: Current methods commonly rely on random or predefined viewpoint-selection heuristics.The passage identifies viewpoint selection as largely unexplored.
  • Contribution: MVTN predicts task-specific viewpoints with a differentiable module trained jointly with a multi-view network, without extra supervision.The approach is designed for both mesh and point-cloud classification and retrieval.
  • Contribution: Combining MVTN with multi-view approaches yields state-of-the-art 3D classification and shape-retrieval results on standard benchmarks.The supplied passage truncates the benchmark list after ModelNet4.

2. Related Work

Prior work spans direct point-cloud, mesh, voxel, and multi-view representations, while MVTN builds on differentiable rendering to make viewpoint prediction learnable within multi-view recognition.

  • Multi-view recognition: Earlier multi-view approaches aggregate features from fixed rendered views using pooling, graph convolutions, or rotation-aware operations.These methods generally rely on fixed rendered datasets.
  • Differentiable rendering: Differentiable-rendering research makes gradients flow from rendered images to rendering parameters.Pytorch3D provides faster, modular mesh and point-cloud rendering with customizable shaders.
  • MVTN context: MVTN uses differentiable rendering to train viewpoint prediction jointly with multi-view networks for mesh and point-cloud recognition and retrieval.This integration addresses viewpoint learning within an end-to-end pipeline.

3. Methodology

MVTN replaces dataset-wide fixed scene parameters with shape-conditioned viewpoints predicted by a lightweight network and optimized end-to-end through differentiable rendering. The pipeline supports mesh and point-cloud inputs for classification and retrieval.

  • Overview: A conventional multi-view recognizer renders M images from fixed viewpoints, processes each with a shared backbone, and aggregates view features.The aggregated representation supports downstream classification or retrieval.
  • Canonical views: Fixed camera viewpoints are commonly arranged in circular or spherical configurations based on dataset model alignment.These configurations use constant scene parameters for the entire dataset.
  • Differentiable renderer: A differentiable renderer maps a mesh or point cloud and scene parameters to rendered images while permitting gradients to reach those parameters.Mesh rendering uses rasterization and shading, while point-cloud rendering uses alpha blending.
  • MVTN: MVTN encodes each 3D shape with a point encoder and regresses bounded, shape-specific viewpoints for the task network.A scaled hyperbolic tangent keeps predicted parameters within permissible bounds.
  • Point-cloud rendering: Point-cloud renderings can expose information hidden from a camera viewpoint, such as wheels on an occluded side of a car.This additional visible content can be useful for recognition.
  • Training and tasks: The classifier and MVTN are optimized jointly with the same task loss in an end-to-end pipeline.For point clouds, the pipeline substitutes a differentiable point-cloud renderer while retaining the same framework.

4. Experiments

MVTN is evaluated across classification and retrieval benchmarks using mesh and point-cloud rendering, multiple view configurations, and established baselines. It achieves strong classification and retrieval results, including state-of-the-art performance on challenging ScanObjectNN and ShapeNet Core55 settings.

  • Datasets and tasks: MVTN is evaluated on ModelNet40, ShapeNet Core55, and ScanObjectNN, covering classification and shape retrieval tasks.ScanObjectNN includes object-only, background, and perturbed variants; ShapeNet Core55 is used for retrieval.
  • Baselines: MVTN compares against voxel, point-cloud, and multi-view baselines, including MVCNN, RotationNet, GVCNN, ViewGCN, PointNet, DGCNN, and KPConv.The baselines use different representations and convolution or feature-aggregation operators.
  • Rendering: MVTN uses differentiable mesh and point-cloud renderers, producing 224×224 rendered images for the multi-view pipeline.ModelNet40 uses the differentiable mesh renderer, with random lighting and object colors during training for augmentation.
  • View-point configurations: MVTN predicts view-points either directly or as bounded offsets from circular or spherical initial configurations.The offset formulation concatenates point features with the initial parameters before predicting view-point offsets.

5. Results

MVTN achieves strong classification, retrieval, rotation-robustness, and occlusion-robustness results across several 3D benchmarks. It also provides qualitative retrieval examples and competitive performance against established methods.

  • 3D Shape Retrieval: Figure 5 contrasts test-set query objects with MVTN's top-five training-set retrievals, framing negative retrieved objects.The figure provides qualitative evidence for the retrieval behavior rather than an additional aggregate metric.
  • 3D Shape Classification: Up to 6% improvement on ScanObjectNN classification establishes MVTN's state-of-the-art performance on the realistic benchmark.MVTN also reports 93.8% test accuracy on ModelNet40.
  • 3D Shape Classification: MVTN achieves 93.8% test accuracy on ModelNet40 compared with previous methods.The comparison uses the MVTN rendering process, including a reported fair-comparison setting against ViewGCN.
  • 3D Shape Retrieval: MVTN achieves state-of-the-art retrieval performance with 92.9% mAP on ModelNet40 and 82.9 mAP on ShapeNet Core55.The ShapeNet result improves the state of the art while using only 12 views.
  • Rotation Robustness: 91.2% test accuracy under Y-axis rotation perturbations demonstrates state-of-the-art rotation robustness for the MVTN-circular variant.The perturbations are applied only at test time, while MVTN is trained without targeting rotation invariance.
  • Occlusion Robustness: 13% higher test accuracy than PointNet when half the object is occluded demonstrates strong occlusion robustness.The evaluation averages accuracy over six cropping directions and uses test-time cropping ratios from 0% to 100%.

6. Analysis and Insights

Ablations show that learned views consistently improve performance across view counts and integrate across multi-view networks. MVTN adds negligible computational overhead, while deeper backbones and more complex point encoders do not improve accuracy.

  • Number of Views: A consistent 2% improvement over fixed and random configurations is observed across a variety of view counts.The comparison evaluates MVCNN trained with fixed, random, and learned view configurations.
  • Backbones and Point Encoders: Using deeper backbone CNNs or a more complex point encoder does not increase ModelNet40 test accuracy.This finding comes from the reported ablation study of MVTN components.
  • Number of Views: Learned MVTN-spherical views achieve consistently superior performance across different numbers of views.The study compares fixed circular or spherical views, learned MVTN views, and random views using repeated experiments.
  • Multi-View Integration: MVTN can be integrated with different multi-view networks by comparing fixed views with learned views at 12 views.The integration study reports overall ModelNet40 classification accuracies across multi-view networks.
  • Other Factors: The study also examines lighting, camera distance, object color, view transferability, and MVTN variants as factors affecting performance.Additional details are provided in the appendix.
  • Efficiency: MVTN contributes negligibly to the multi-view pipeline's time and memory requirements.The analysis measures GFLOPs and single-sample forward-pass time alongside memory requirements.

7. Conclusions and Future Work

MVTN replaces fixed dataset-aligned views with learned viewpoint regression in a differentiable, supervision-free pipeline. The authors identify extensions to segmentation and to learning additional scene parameters as future work.

  • Conclusions: MVTN learns viewpoint transformations for multi-view networks in a fully differentiable pipeline without extra training supervision.The conclusion describes MVTN as addressing the fixed-view limitation of current multi-view methods.
  • Future Work: Future work includes extending MVTN to shape and scene segmentation and learning scene parameters such as lighting and textures.These parameters would complement the learned camera viewpoints.

A. Detailed Experimental Setup

The experimental setup uses mesh or point-cloud rendering with configurable circular, spherical, and random view arrangements. Training includes randomized color and lighting for ModelNet40, while ShapeNet uses point-cloud rendering to manage high-poly meshes.

  • ModelNet40: ModelNet40 uses mesh renderings with randomized object color and light direction during training, fixed during testing.The setup shows eight circular views for each shape.
  • ShapeNet Core55: ShapeNet Core55 uses point-cloud renderings that expose content hidden from the camera viewpoint, such as occluded-side car wheels.The setup shows 12 spherical views for each shape.
  • Rendering: Point-cloud rendering is used for ShapeNet because high-poly meshes hinder training and simplification can corrupt their main visual clues.It also allows all shapes to be processed with equal memory requirements.
  • MVTN Details: The MVTN regression network is a 4-layer MLP whose input includes view angles and point features, with output offsets for the views.The network has around 9K parameters when M = 12.
  • Training Augmentation: ModelNet40 training uses randomized color and lighting augmentation while testing keeps those rendering factors fixed for stable performance.The figure presents eight circular views per 3D shape.
  • View Configurations: MVTN supports circular, spherical, and random configurations for varying numbers of views.Its regression architectures output azimuth and elevation angles for the learned views, using b = 40 in most experiments.

B.1. Classification and Retrieval Benchmarks

MVTN improves prior state-of-the-art classification performance under the same ModelNet40 setup, while implementation differences reflect the differentiable renderer used.

  • Classification benchmarks: MVTN improves ViewGCN’s previous state-of-the-art ModelNet40 classification result under the same evaluation setup.The reported MVTN implementations use Pytorch3D’s differentiable renderer, whereas other multi-view results may use pre-rendered OpenGL views.
  • Benchmark coverage: Reported results are presented across ModelNet40, ScanObjectNN, and ShapeNet Core55 classification and retrieval benchmarks.Each table reports four runs as max (avg ± std).
  • Evaluation conditions: The renderer choice may explain differences between the authors’ implementations and results reported in the original papers.The paper attributes this possibility to differences between Pytorch3D differentiable rendering and non-differentiable OpenGL rendering.

B.2. Rotation Robustness

The study evaluates classification under random Y-axis rotation perturbations and finds that fewer views increase sensitivity to rotation.

  • Rotation setup: The experiment injects random test-time rotations around the Y-axis, the gravity axis, into an aligned dataset.This follows the robustness-testing setup of prior work.
  • Rotation findings: Using fewer views makes circular MVCNN more sensitive to rotation perturbations.The comparison considers M = 6 and M = 12 views.
  • Rotation findings: Figure 13 compares circular MVCNN with circular-MVTN at 6 and 12 views as maximum Y-rotation perturbation varies.The evaluation measures classification accuracy on ModelNet40.

B.3. Occlusion Robustness

MVTN is evaluated for occlusion robustness by cropping 3D objects along canonical directions, with comparisons against point-cloud and multi-view methods.

  • Occlusion robustness: 13% higher test accuracy than PointNet is reported when half of the object is occluded.The comparison uses ModelNet40 point clouds and averages accuracy across six cropping directions.
  • Evaluation setup: The occlusion experiment crops objects from 0% to 75% along the ±X, ±Y, and ±Z directions at test time.PointNet, DGCNN, and MVTN are trained on ModelNet40 point clouds.
  • Comparisons: Figure 14 plots test accuracy against occlusion ratio for PointNet, DGCNN, and MVTN.MVTN1 uses MVCNN, while MVTN2 uses View-GCN as the multi-view network.
  • Benchmark context: The benchmark tables include classification results for ModelNet40, ScanObjectNN, and ShapeNet Core55, plus retrieval results on ModelNet40 and ShapeNet Core55.Table 10 specifically covers ScanObjectNN variants with background, object-only, and hardest settings.

C.1. Ablation Study

The ablations examine MVTN variants, rendering conditions, computational cost, transferability, and whether predicted views vary by instance rather than collapsing per class.

  • MVTN variants: MVTN-spherical achieves consistently superior performance across view counts in the ModelNet40 variant study.The experiments repeat each setup four times and report average accuracies with confidence intervals.
  • Ablation results: 91.0%, 91.6%, and 91.9% are obtained for the evaluated settings, while point counts of 512, 1024, and 2048 yield 91.2%, 91.6%, and 91.6%.These values are reported in the ablation results.
  • Distance ablation: Learning camera distance generally performs worse or fails to improve performance, so the main experiments fix distance at 2.2 units.The fixed distance is described as a middle ground providing best accuracy.
  • Efficiency: MVTN contributes negligibly to the multi-view pipeline’s time and memory requirements.The analysis compares FLOPs, MACs, forward-pass time, and parameter counts across modules.
  • Rendering conditions: Randomizing light direction reduces overfitting for larger view counts and improves generalization.This effect is studied for MVTN-circular.
  • Transferability: Transferred MVTN views reach 93.1% ViewGCN accuracy, a 0.7% boost over the original ViewGCN’s 92.4%.The MVTN module is frozen while ViewGCN is trained on the learned fixed views.
  • Predicted views: Per-instance views reach 91.0% on ModelNet40 versus 90.6% for per-class averaged views and 89% for fixed views.The result supports variation across instances rather than a single view configuration per class.
  • Predicted views: MVTN-circular and MVTN-spherical adjust common circular or spherical camera configurations using offsets tailored to each object.Qualitative examples show shifts intended to better represent discriminative object details.
Loading 2011.13244v3…