Source-linked AI summary
Dominant Set Clustering and Pooling for Multi-View 3D Object Recognition
Chu Wang, Marcello Pelillo, Kaleem Siddiqi
TL;DR
The paper addresses how to aggregate multi-view features for 3D object recognition without discarding potentially informative views or relying on pairwise formulations. It introduces recurrent dominant-set clustering and within-cluster pooling in a pretrained CNN, achieving 93.8% test-set recognition accuracy on ModelNet40 while a faster non-end-to-end strategy reaches 93.3%.
Problem
Existing multi-view strategies may discard informative views through winner-take-all pooling or use a pairwise formulation for recognition.
Method
The method clusters views using a similarity graph and dominant sets, pools features within each cluster, and recurrently refines the pooled vectors in a pretrained CNN.
Results
93.8% full test set accuracy is achieved on ModelNet40, while fast non-end-to-end training achieves 93.3% with greatly reduced training cost.
Takeaways & Limitations
Recurrent cluster-pooling provides state-of-the-art multi-view recognition, and sacrificing end-to-end training can greatly speed computation with negligible accuracy loss.
Takeaways & Limitations
Within-cluster pooling is fixed throughout recurrence as either max or average pooling, and the dataset split is assumed from the cited description.
Abstract
from arXiv · showhide
View based strategies for 3D object recognition have proven to be very successful. The state-of-the-art methods now achieve over 90% correct category level recognition performance on appearance images. We improve upon these methods by introducing a view clustering and pooling layer based on dominant sets. The key idea is to pool information from views which are similar and thus belong to the same cluster. The pooled feature vectors are then fed as inputs to the same layer, in a recurrent fashion. This recurrent clustering and pooling module, when inserted in an off-the-shelf pretrained CNN, boosts performance for multi-view 3D object recognition, achieving a new state of the art test set recognition accuracy of 93.8% on the ModelNet 40 database. We also explore a fast approximate learning strategy for our cluster-pooling CNN, which, while sacrificing end-to-end learning, greatly improves its training efficiency with only a slight reduction of recognition accuracy to 93.3%. Our implementation is available at https://github.com/fate3439/dscnn.
1 Introduction
View-based 3D recognition uses projected multi-view CNN features and generally outperforms volumetric approaches on popular 3D model databases. The paper introduces recurrent dominant-set clustering and within-cluster pooling to aggregate similar views while refining their features.
- 1 Introduction: View-based methods classify 3D objects from hand-designed or CNN features extracted from multiple 2D renderings.These approaches reduce recognition to classification over features aggregated across views.
- 1 Introduction: View-based approaches outperform volumetric methods on popular 3D object model databases.Volumetric methods use features from meshes, voxelized grids, or point clouds, including 3D convolutional networks on occupancy grids.
- 1 Introduction: MVCNN fuses multi-view relu vectors with full-stride channel-wise max pooling before fully connected category prediction.The pairwise decomposition method instead uses separate CNNs for view-pair selection and pairwise label prediction.
- 1 Introduction: The proposed layer builds a view-similarity graph, finds dominant sets, and pools features only within each similar-view cluster.Dominant sets provide high within-cluster similarity and between-cluster dissimilarity.
- 1 Introduction: Recurrent clustering feeds changed pooled vectors back into the same layer until clusters stabilize, then applies full-stride pooling for unified prediction.The unified representation is subsequently processed by fully connected layers.
2 Recurrent Clustering and Pooling Layer
The layer builds a complete view-similarity graph, extracts dominant-set clusters, pools features within clusters, and recurrently refines them until cluster assignments stabilize. It supports back propagation through the repeated clustering-and-pooling operations and then forwards the pooled representation to subsequent network layers.
- View Similarity Graph: The layer constructs a complete weighted undirected graph whose nodes are rendered views and whose edge weights encode pairwise view similarity.The affinity matrix is nonnegative, symmetric, and has zero diagonal because the graph has no self-loops.
- View Similarity Graph: View similarity is computed from CNN relu feature vectors, with larger inner products indicating more similar views.The relu features provide low-dimensional appearance signatures for the rendered views.
- Dominant Set Clustering: Dominant-set clustering partitions views into internally coherent groups with high intra-cluster similarity and inter-cluster dissimilarity.Replicator dynamics provides an extraction algorithm with run time complexity O(V^2).
- Clustering, Pooling and Recurrence: After clustering, the method pools information only within each cluster and feeds the resulting relu vectors back into the layer for another recurrence.Within-cluster pooling is fixed to either max or average pooling throughout the recurrence, while the paper describes alternating max and average pooling during recurrences.
- Clustering, Pooling and Recurrence: For a mug with 12 initial views, the first stage produces three clusters for side, front-or-rear, and oblique views, after which green and blue views combine and no further clusters form.The figure illustrates the clustering and pooling recurrences for this example.
- Back propagation: The recurrent layer has no learnable parameters, so its gradients are computed by reversing pooling and clustering and mapping them back to the preceding recurrence.During back propagation, gradients are accumulated for the prior recurrence and only nodes belonging to the relevant cluster receive non-zero values.
3 Experiments
Experiments evaluate the recurrent clustering-and-pooling CNN on ModelNet40, including training variants, feature modalities, view counts, and comparisons with existing methods.
- Training and Testing Procedure: The experiments use a pretrained VGG-M CNN fine-tuned on the ModelNet40 training set after inserting customized layers after relu6.ModelNet40 contains 12,311 CAD models across 40 categories, with 9,843 training and 2,468 test models.
- Training and Testing Procedure: The evaluation renders 12 views per object from cameras placed every 30 degrees at 30-degree elevation, and also considers surface normals and depth.Surface normals are computed from mesh vertices and interpolated across faces; normalized depth values are rendered directly.
- Training and Testing Procedure: Fast training computes pretrained relu7 features before recurrent clustering and pooling, using a universal clustering scheme shared across training and testing objects.The scheme averages affinity matrices across training objects and records recurrence-wise clusterings for end-to-end training.
- Additional Feature Types: 93.3% test accuracy is achieved by combining appearance, depth, and surface normals without CNN training.The additional modalities significantly improve recognition accuracy under the fast training scheme.
- Effects from number of Views: Recognition accuracy improves steadily with more views for the proposed method, while MVCNN drops when increasing from 6 to 12 views.The comparison attributes the MVCNN drop to a potential drawback of its single full-stride max-pooling strategy.
- Comparison with the present State-of-the-art: 93.8% full-test-set accuracy and 92.8% subset accuracy are obtained on ModelNet40, while fast training reaches 93.3% and 92.1%, respectively.Fast training greatly reduces training cost while retaining state-of-the-art recognition accuracy.
- Comparison with the present State-of-the-art: A voxel-based ensemble achieves 95.54% accuracy, whereas one VRN reaches 91.33% and the proposed method reaches 92.2% using only RGB features.The ensemble uses five 45-layer Voxception-ResNet models and one Inception-like architecture; fine-tuning the proposed model takes 20 hours on a Tesla K40.
4 Conclusion
The paper concludes that recurrent clustering and pooling improves multi-view feature aggregation for 3D object recognition, while fast training substantially reduces computation with only a negligible accuracy loss.
- 4 Conclusion: The recurrent clustering and pooling layer is intended to provide more discriminative multi-view aggregation for 3D object recognition.The conclusion positions the layer as applicable to real-world 3D vision systems where recognition performance and computational cost both matter.
- 4 Conclusion: State-of-the-art object category recognition is achieved on ModelNet40 by inserting the recurrent layer into a standard pretrained network.The conclusion reports this result as evidence from experiments on the ModelNet40 benchmark.
- 4 Conclusion: Fast training greatly speeds computation at the cost of sacrificing end-to-end training, with a negligible loss in multi-view recognition accuracy.This trade-off is presented as useful when both performance and computational cost must be considered.