PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning Representation using Autoencoder for 3D Shape Retrieval

Zhuotun Zhu, Xinggang Wang, Song Bai, Cong Yao, Xiang Bai

arXiv:1409.7164v1cs.CV

TL;DR

The paper tackles the lack of effective deep representations for complex 3D shapes and limited training data. It projects shapes into 2D depth images, learns features with an autoencoder, and combines the resulting global representation with local descriptors. The combined method achieves state-of-the-art 3D shape retrieval performance.

  • Problem

    3D shape recognition lacks widely applied deep representations because shapes have complex 3D structure and limited feature-learning data.

  • Method

    The method projects 3D shapes into multiple depth images, learns an autoencoder code, and combines its global representation with BoF-SIFT local descriptors.

  • Results

    The autoencoder outperforms other global descriptors, while combining it with local descriptors achieves state-of-the-art 3D shape retrieval performance.

  • Takeaways & Limitations

    Global autoencoder features and local descriptor features are complementary for 3D shape retrieval.

Abstract

from arXiv · show

We study the problem of how to build a deep learning representation for 3D shape. Deep learning has shown to be very effective in variety of visual applications, such as image classification and object detection. However, it has not been successfully applied to 3D shape recognition. This is because 3D shape has complex structure in 3D space and there are limited number of 3D shapes for feature learning. To address these problems, we project 3D shapes into 2D space and use autoencoder for feature learning on the 2D images. High accuracy 3D shape retrieval performance is obtained by aggregating the features learned on 2D images. In addition, we show the proposed deep learning feature is complementary to conventional local image descriptors. By combing the global deep learning representation and the local descriptor representation, our method can obtain the state-of-the-art performance on 3D shape retrieval benchmarks.

I. INTRODUCTION

The paper addresses the difficulty of representing complex 3D shapes for retrieval by learning global features from projected depth images. It combines this representation with local descriptors to improve retrieval performance.

  • 3D shape retrieval requires efficient representations because 3D models have complex structures that are difficult to handle directly.
  • Deep learning has been effective for image recognition, but 3D shape recognition has mainly relied on hand-crafted descriptors.
  • The method projects each 3D shape into multiple 2D depth images and learns an autoencoder representation from those projections.
  • The learned autoencoder representation is global, while local descriptors such as SIFT capture complementary information.
  • Combining global deep learning and local descriptor representations achieves state-of-the-art 3D shape retrieval performance.

III. DEEP LEARNING REPRESENTATION USING AUTOENCODER

The representation pipeline normalizes each 3D model, generates depth-buffer projections from multiple viewpoints, and trains an autoencoder on them. The resulting code provides a low-dimensional representation for 3D shape retrieval.

  • Each 3D model is pose-normalized to account for translation and scale differences before projection.
  • The projections train an autoencoder whose code serves as a low-dimensional representation of the depth images.
  • The learned code is used to perform 3D shape retrieval.
  • A 3D shape is represented by a set of depth-buffer images collected from different viewing angles.
  • The projection views vary in azimuth and elevation, with azimuth describing rotation in the x-y plane and elevation describing angles above or below it.

B. Deep Belief Network

The deep belief network is built by stacking restricted Boltzmann machines, whose restricted connections support probabilistic reconstruction and efficient learning. Its pretrained parameters initialize the autoencoder.

  • A DBN stacks RBMs so each preceding hidden layer becomes the visible layer for the next, improving the variational lower bound on data log probability.
  • RBMs connect visible and hidden units but contain no within-layer connections, making hidden activations conditionally independent given visible activations and conversely.
  • The network assigns probabilities to visible-hidden configurations through an energy function, a partition function, and marginalization over hidden vectors.
  • Learning increases training-image probability by lowering its energy relative to other configurations, using stochastic steepest ascent based on expectation differences.
  • Contrastive sampling alternates hidden and visible updates, then uses a confabulation to approximate model expectations for weight adjustment.
  • The fast learning procedure approximates the log-probability derivative but works well experimentally.

C. Fine-tuning the Autoencoder

After DBN pretraining, the autoencoder is globally fine-tuned with real-valued activities and backpropagation to improve reconstruction of depth images. The resulting code layer serves as the image representation for 3D retrieval.

  • Fine-tuning replaces stochastic binary activities with real-valued probabilities and adjusts weights and biases through backpropagation.
  • The procedure minimizes squared reconstruction error to obtain a deep reconstruction model for 2D depth-image inputs.
  • Pretraining uses stacked RBMs, after which the unfolded autoencoder is fine-tuned for better reconstruction performance.
  • The code layer is used as an efficient representation of each input image for 3D retrieval.
  • Fine-tuning produces better reconstruction than pretraining alone, supporting the code layer as a representation of the 2D image.

D. Set-to-Set Distance

Each 3D model is represented by a set of code vectors obtained from its projected depth images. Retrieval compares two models through a set-to-set distance between their code sets.

  • The code set C(S) contains one coding vector for each of a model’s Np projection images.
  • Each coding vector has Nc dimensions, with cij denoting the value of its j-th dimension.
  • Two models are compared by forming their respective code sets from projection-specific coding vectors.
  • A Hausdorff-distance variant defines the distance between two models, using a vector distance such as a p-norm or algebraic distance.
  • The resulting distances support retrieval by ranking candidate models according to their distance from the query.

IV. BAG OF FEATURES REPRESENTATION

The paper represents each 3D model with a Bag-of-Features histogram built from SIFT descriptors, complementing the autoencoder’s global representation with local visual information.

  • BoF-SIFT serves as the local descriptor paired with the autoencoder’s global descriptor for 3D shape retrieval.
  • Unlike IM-SIFT, BoF-SIFT places SIFT descriptors from all depth images into one bag without rotation normalization.
  • A 1500-word K-means vocabulary and vector quantization convert each model’s SIFT set into an L1-normalized histogram.
  • The paper evaluates the complementarity between autoencoder and BoF-SIFT representations.

V. EXPERIMENTS

The experiments evaluate the proposed 3D shape retrieval method on two standard datasets and compare it with state-of-the-art approaches.

  • Experiments test the method on two widely used standard 3D-shape datasets.
  • The evaluation compares the proposed method with state-of-the-art approaches for 3D shape retrieval.
  • The algorithm was implemented in MATLAB and run on a laptop with an Intel Core i5-3210M CPU and 4GB memory.

A. Princeton Shape Benchmark(PSB)

The Princeton Shape Benchmark contains 1,814 models across 182 classes, with an equally partitioned training and testing setup and varied class sizes.

  • 1,814 models form the Princeton Shape Benchmark, whose base classification is equally partitioned into training and testing sets.
  • The training set contains 90 classes and 907 models, while the comparison set contains 92 classes and an equal number of models.
  • Class sizes vary substantially, ranging from 4 to 50 models per class.
  • Example PSB images span varied categories, including animals and buildings.

B. Engineering Shape Benchmark(ESB)

The Engineering Shape Benchmark evaluates the method on mechanical CAD parts, using projected depth images as model inputs and an autoencoder-based distance for retrieval.

  • 867 CAD models across 45 classes comprise the Engineering Shape Benchmark, with class sizes ranging from 4 to 58.
  • Each model contributes 64 projections arranged as 8 × 8, producing grayscale images with pixel values in [0, 1].
  • RBM layers use normalized depth-image pixels initially and previous-layer activation probabilities at higher levels, with stochastic hidden states.
  • DBN-trained weights initialize a symmetric autoencoder that is fine-tuned using backpropagation.
  • The trained autoencoder maps each depth image to a low-dimensional code intended to support discriminative description and reconstruction.
  • The ESB encoder uses the layer structure 5184 (72 × 72)-2000-500-100-20.
  • The reported distance between models is based on a distance function between their feature vectors, with p = 2 specified in the displayed formula.

D. Evaluation Methods

The evaluation uses standard benchmark protocols to compare global descriptors for 3D shape retrieval. The autoencoder is reported as more efficient than other global-feature-based methods.

  • The Princeton Shape Benchmark provides open-source evaluation code for comparing 3D shape retrieval algorithms.
  • Nearest Neighbor measures the percentage of closest matches sharing the query class, with higher scores indicating better performance.
  • First-Tier and Second-Tier measure the percentage of a query’s class appearing among the top M matches, using M = C −1 and M = 2(C −1), respectively.
  • Tables I and II compare the global-feature-based autoencoder with other global descriptors on two standard datasets.
  • The autoencoder is reported as more efficient than other global-feature-based methods for 3D shape retrieval.

F. Complementary Property

The method combines the autoencoder’s global information with BoF-SIFT’s local details. This hybrid representation improves retrieval results and achieves state-of-the-art performance in the reported experiments.

  • The hybrid method linearly combines global autoencoder and local BoF-SIFT descriptors with empirically equal weights.The autoencoder reconstructs global information, while BoF-SIFT captures local details.
  • The hybrid method is compared with PANORAMA, CM-BoF, and CM-BoF+GSMD, which also capture global and local shape information.
  • More than 7 percent improvements are reported for both First-Tier and Second-Tier retrieval results when autoencoder features are combined with BoF-SIFT.
  • The experiments report that linearly combining the autoencoder with a local descriptor achieves state-of-the-art 3D shape retrieval performance.
Loading 1409.7164v1…