PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Deep Geometry Learning: From a Representation Perspective

Yun-Peng Xiao, Yu-Kun Lai, Fang-Lue Zhang, Chunpeng Li, Lin Gao

arXiv:2002.07995v2cs.GR

TL;DR

Deep learning for 3D shapes faces a representation choice because different applications rely on different shape representations, with no single representation working well for all. This survey reviews methods and datasets across representations, finding that suitable representation choice depends on the task.

  • Problem

    3D shape applications use varied representations, and no unique representation works well for all applications.

    Source: §1 · p. 1

  • Method

    The survey reviews deep learning methods and datasets for 3D shape representations and discusses their advantages and disadvantages across applications.

    Source: §1 · p. 2, §8 · p. 12

  • Results

    The survey concludes that different representations are usually suitable for different applications, so representation choice matters for specific tasks.

    Source: §10 · p. 14

  • Takeaways & Limitations

    The survey summarizes a tendency toward geometry-learning methods that demand less computation and memory while representing more detailed and structured shapes.

    Source: §10 · p. 14

    The GRASS binary-tree organization creates ambiguity and is not suitable for large-scale datasets.

    Source: §6 · p. 10

Abstract

from arXiv · show

Researchers have now achieved great success on dealing with 2D images using deep learning. In recent years, 3D computer vision and Geometry Deep Learning gain more and more attention. Many advanced techniques for 3D shapes have been proposed for different applications. Unlike 2D images, which can be uniformly represented by regular grids of pixels, 3D shapes have various representations, such as depth and multi-view images, voxel-based representation, point-based representation, mesh-based representation, implicit surface representation, etc. However, the performance for different applications largely depends on the representation used, and there is no unique representation that works well for all applications. Therefore, in this survey, we review recent development in deep learning for 3D geometry from a representation perspective, summarizing the advantages and disadvantages of different representations in different applications. We also present existing datasets in these representations and further discuss future research directions.

1 Introduction

The survey addresses how to represent complex, irregular 3D shapes for deep learning, reviewing representation-specific methods and their advantages and disadvantages across applications.

1 Introduction

  • Global representations can impair matching on noisy or incomplete shapes and burden computation or encourage overfitting through high dimensionality.These drawbacks motivate more concise, discriminative, and informative shape representations.Source: §1 · p. 1
  • The survey reviews deep learning methods for 3D representations and considers their advantages and disadvantages across application scenarios.It also identifies datasets and discusses future research directions.Source: §1 · pp. 1–2
  • Image-based representations exploit regular 2D structure and existing image-processing models, but inevitably lose some geometric features.They include RGB or RGB-D images from different viewpoints.Source: §1 · p. 2
  • Voxel representations extend pixels to regular 3D grids, but resolution increases sharply raise memory and computational costs.This restricts voxel-based shape representations to low resolutions.Source: §1 · p. 2
  • Point clouds encode surfaces as sampled 3D coordinates, while meshes encode vertices and edges and can represent higher-quality shapes with less memory and computation.Point clouds lack order and connectivity; meshes are also suitable for deformation.Source: §1 · p. 2
  • Implicit fields describe spatial relationships between points and surfaces, offering infinite-resolution shape descriptions with reasonable memory use, but often require costly conversion to meshes.They can represent shapes with changing topology but do not directly reflect geometric features.Source: §1 · p. 2

2 Image-based methods

Image-based methods use RGB or depth information and multiple viewpoints to extract cues for 3D shape analysis and reconstruction. Multi-view images have supported strong recognition performance relative to other 3D representations.

2 Image-based methods

  • RGB and depth images support 3D shape inference, with depth adding geometric cues and multiple views providing complementary information.A plausible 3D shape can be inferred from images taken from different perspectives.Source: §2 · p. 3
  • Deep-learning methods use RGB and depth images for tasks including depth-map reconstruction and 3D object detection.Socher et al. processed RGB and depth images separately before merging features; Gupta et al. [37] encoded depth using disparity, height, and angle.Source: §2 · p. 3
  • Multi-view image methods perform better in 3D shape recognition than methods based on other 3D representations.MVCNN processes views separately, pools their features, and sends the merged feature through the remaining CNN; multi-resolution MVCNN was proposed to increase classification accuracy.Source: §2 · p. 3

3 Voxel-based representations

Voxel methods represent 3D shapes in volumetric grids and extend neural networks to that structure, while octrees adaptively encode grids to reduce computation at higher resolutions.

3.1 Dense Voxel Representation

  • Dense voxel methods extend deep belief networks and 3D CNNs to volumetric inputs for shape analysis, recognition, and generation.3D ShapeNets predicts categories and shapes; VoxNet offers a simple, trainable volumetric CNN and uses rotated instances with pooled predictions for orientation invariance.Source: §3.1 · pp. 3–4
  • Dense voxel networks face overfitting, orientation, data sparsity, and low-resolution challenges, and naive 2D-network extensions may not outperform image-based CNNs.Proposed responses include auxiliary subvolume classification, elongated kernels, orientation augmentation, and simultaneous class-and-orientation prediction.Source: §3.1 · p. 4

3.2 Sparse Voxel Representation (Octree)

  • Voxel computation grows exponentially with resolution, and sparse occupancy at higher resolutions can cause unnecessary computation.The cited methods range from 20^3 to 64^3 grids; FPNN was designed to address data sparsity.Source: §3.2 · pp. 4–5
  • Octrees encode voxel grids sparsely and adaptively, enabling high-resolution generation by focusing prediction on regions near the object surface.HSP generates octree grids coarse-to-fine and can reach 256^3 resolution; OGN also targets high-resolution volumetric representations.Source: §3.2 · p. 5
  • Octree-based analysis methods adapt the structure and define specialized operations, while Adaptive O-CNN uses octants to organize planar surface patches.OctNet and O-CNN introduce structures intended for deep networks; Adaptive O-CNN defines a patch-guided adaptive octree.Source: §3.2 · p. 5

4 Surface-based representations

Surface-based methods process point clouds or meshes, each with distinct structure and processing challenges. The survey covers point-cloud networks and mesh approaches in graph and manifold domains, as well as mesh generation.

4.1 Point-based Representation

  • PointNet directly processes unordered point clouds using max pooling, combined global and local features, and joint alignment for transformation invariance.PointNet++ adds a hierarchy to capture local features at multiple scales that PointNet cannot handle.Source: §4.1 · pp. 5–6
  • Point-cloud networks address irregularity through operators such as PointCNN's X-transformation and DGCNN's EdgeConv on dynamically constructed neighborhood graphs.DGCNN updates nearest neighbors in feature space across layers.Source: §4.1 · p. 6
  • Raw point clouds are easy to obtain with 3D scanners but can be unsuitable for shape analysis because of noise and data sparsity.The survey notes upsampling for fine-grained point clouds and registration to align points from multiple scans.Source: §4.1 · pp. 6–7

4.2 Mesh-based Representations

  • Mesh methods process vertex-edge graphs or discretized 2-manifolds, using learned graph or manifold operators to handle irregular surface structure.MeshCNN operates on edge features with convolution and edge-collapse pooling; manifold methods define local patches and kernels.Source: §4.2 · p. 7
  • Mesh-based methods also map surfaces to 2D domains or spherical signals for CNN processing, including geometry-image approaches and a cylinder-panorama projection.TextureNet uses a 4-RoSy field to parameterize surfaces for extracting high-resolution mesh texture information.Source: §4.2 · p. 7
  • Mesh generation methods include deforming templates, generating surfaces from patches, and autoregressively generating polygon meshes.Template deformation cannot express complex topology in some shapes; methods described include topology modification and skeleton-bridged learning.Source: §4.2 · p. 8

5 Implicit representations

Implicit representations describe shapes through learned fields, offering flexible topology and resolution, while requiring conversion to explicit geometry for some uses. The survey covers occupancy functions, signed distance functions, structured function sets, and learning without explicit 3D supervision.

Implicit surface representations

  • Implicit fields are not limited by fixed topology or resolution, motivating their use in shape analysis and generation.They can describe shapes at infinite resolution with reasonable memory, but do not directly reflect geometric features and often require conversion to explicit representations.Source: §5 · p. 8

Occupancy functions

  • Occupancy networks classify queried 3D points as inside or outside, then extract a mesh from the learned field using MISE and marching cubes [58].The network takes point position and shape observation as input and outputs occupancy probability.Source: §5 · p. 8

Signed distance functions

  • SDFs map 3D points to signed real values: positive outside, negative inside, and zero on the surface; absolute values give distance to it.DeepSDF [74] and DISNs [116] use this representation.Source: §5 · p. 8
  • DISN was the first to reconstruct shapes with flexible topology and thin structure in single-view reconstruction, a task described as difficult for other representations.The survey attributes this capability to SDF advantages.Source: §5 · p. 9

Function sets

  • DSIF combines SIF’s coarse shape-element representation with DIFs for local geometry details [30] [31].SIF represents each element with a scaled anisotropic 3D Gaussian; its parameters are learned by a CNN.Source: §5 · p. 9

Approach without 3D supervision

  • A field-probing framework learns implicit representations without explicit 3D supervision by using image silhouettes and geometry regularization [56].The silhouette loss constrains the outline, while geometry regularization constrains surface plausibility.Source: §5 · p. 9

6 Structure-based representations

Structure-based representations encode 3D shapes as parts and their relations, using linear or hierarchical organization and separating or jointly modeling structure and geometry. The survey describes primitive-based generators, graph alternatives to ambiguous binary trees, and methods that combine coarse structure with detailed geometry.

Linearly Organized

  • 3D-PRNN generates primitives sequentially, while PQ-NET encodes geometry features in its part sequence [121, 111].SDM-NET uses a two-level VAE for part geometry and structure, encoding relations such as symmetry and support [27].Source: §6 · p. 9

Hierarchically Organized

  • GRASS encodes shape structure as a binary tree whose nodes represent part geometry and adjacency or symmetry relations [51].Its architecture combines a recursive autoencoder, a GAN for root-code distributions, and an autoencoder for part geometry.Source: §6 · pp. 9–10
  • GRASS’s binary-tree organization leads to ambiguity and is described as unsuitable for large-scale datasets; StructureNet instead organizes hierarchy as graphs [51] [68].The survey also describes BSP-Net as assembling convexes in a BSP tree to depict sharp geometry features [13].Source: §6 · p. 10

Structure and Geometry

  • G2L generates coarse semantic voxel structure before refining part geometry, whereas SAGNet jointly encodes structure and geometry and performs better on tenon-mortise joints than other structure-based methods [113].G2L uses a GAN for coarse grids and a Part Refiner for local geometry details.Source: §6 · p. 10

7 Deformation-based representations

Deformation-based representations parameterize deformation information and are presented as better suited than other representations to non-rigid shapes such as articulated models. The survey covers mesh-based deformation descriptions and implicit pose-parameter approaches.

Deformation-based representations

  • Deformation-based representations parameterize deformation information and perform better than other representations for non-rigid shapes such as articulated models.The survey notes that many earlier methods focused on rigid models and paid less attention to non-rigid deformation.Source: §7 · p. 10

Mesh-based Deformation Description

  • Mesh methods generate target shapes by deforming templates and store deformation information in vertex features while preserving mesh connectivity [27] [73] [104] [109].RIMD is rotation-invariant and achieves high performance in reconstruction, deformation, and registration; ACAP constrains rotations between adjacent vertices.Source: §7 · p. 10
  • ACAP-based methods support analysis, transfer, simulation, and animation of mesh deformation, including unpaired transfer with cycle and visual-similarity consistency [95] [96].ACAP is also used as input for mesh-sequence generation by a bidirectional LSTM.Source: §7 · pp. 10–11

Implicit surface based approaches

  • NASA represents articulated deformable shapes with pose parameters recording bone transformations and compares unstructured, piecewise rigid, and piecewise deformable models [45].The survey describes this as opening another direction for representing deformable 3D shapes.Source: §7 · p. 11

8 Datasets

The survey presents datasets across RGB-D, man-made object, and non-rigid model representations and applications. It notes that larger, more detailed datasets challenge existing techniques, while appropriate dataset choice benefits model performance and generalization.

Datasets

  • Larger datasets with more detail bring new challenges for existing techniques, and choosing an appropriate dataset benefits model performance and generalization.The survey groups datasets by representation and application.Source: §8 · p. 11

RGB-D Images

  • The v1 and v2 versions collect 64 and 464 categories, respectively; KITTI provides outdoor autonomous-driving images and five categories, while Human10 samples 10 human-action sequences [85, 86, 29, 11].ScanNet includes 2.5M views in 1,513 scenes, with 3D camera poses, surface reconstructions, and semantic segmentations [17].Source: §8 · p. 12

Man-made 3D Object Datasets

  • Man-made object datasets span aligned CAD models, richly annotated ShapeNet objects, 2D-image-linked objects, voxelized rooms, hierarchical parts, and furniture scenes [112, 12, 114, 92, 23].ModelNet contains 127,915 CAD models in 662 categories; ShapeNet contains more than 3 million models in more than 4K categories [12].Source: §8 · p. 12

Non-Rigid Model Datasets

  • Non-rigid datasets include mesh models, human scans with correspondence ground truth, and motion-capture collections [8, 5, 59].AMASS contains 344 subjects, 11,265 motions, and more than 40 hours of recordings [59].Source: §8 · p. 12

9 Shape Analysis and Reconstruction

The section reviews shape-analysis and reconstruction applications, emphasizing that suitable representations vary by task.

9.1 Shape Analysis

  • Classification and retrieval use network-extracted feature vectors; retrieval learns compact latent codes and returns nearest shapes using distance metrics.Source: §9.1 · p. 12
  • Segmentation discriminates shape-part categories, commonly uses mean Intersection-over-Union, and is most often approached with point-based representations.Source: §9.1 · p. 13
  • PRS-Net [28] detects planar reflective symmetry unsupervised with symmetry-distance and regularization losses, and is reported robust to noisy or incomplete inputs and efficient versus traditional methods.It uses a 3D voxel CNN and achieves high performance at low resolution.Source: §9.1 · p. 13

9.2 Shape Reconstruction

  • Generation predicts voxel occupancy, samples or folds points, deforms mesh templates, and increasingly uses structured representations for coarse-to-fine shapes.Source: §9.2 · p. 13

10 Summary

The survey reviews deep-learning methods across 3D representations, datasets, and shape applications. It identifies a trend toward lower computation and memory demands alongside more detailed, structured geometry, and stresses task-specific representation choice.

10 Summary

  • The survey reviews deep-learning models for different 3D object representations and describes a trend toward lower computation and memory demands and more detailed, structured geometry.Source: §10 · p. 14
  • It introduces 3D datasets that provide resources and support evaluation for data-driven learning methods.Source: §10 · p. 14
  • It reviews shape-analysis and reconstruction applications and concludes that suitable 3D representations depend on the specific task.Source: §10 · p. 14

Related papers

Loading 2002.07995v2…