A survey on Deep Learning Advances on Different 3D Data Representations

• The survey categorizes 3D representations into Euclidean-structured and non-Euclidean families to provide more comprehensive coverage than prior efforts addressing one or both categories.It aims to give in-depth information about the different representations used in 3D data.Source: §1 · p. 3

• Euclidean 3D representations preserve grid structure, global parametrization, and common coordinates, making 2D deep learning extensions comparatively direct.The category includes descriptors, projections, RGB-D, volumetric, and multi-view representations.Source: §1.1 · p. 3
• Shape descriptors summarize geometric or topological characteristics as signatures, with global or local and intrinsic or extrinsic variants that can support hierarchical feature learning.Descriptors may use an object’s geometry, topology, surface, texture, or combinations of these properties.Source: §1.1.1 · pp. 3–4
• Projections convert 3D objects into 2D grids, preserving selected properties according to the projection type, including spherical and cylindrical domains.Source: §1.1.2 · p. 4
• RGB-D data combines 2D color with depth to provide inexpensive 2.5D representations for tasks including identity recognition, pose regression, and scene reconstruction.Source: §1.1.3 · p. 4
• Voxel and octree representations encode 3D occupancy on regular or hierarchical grids, but neither preserves intrinsic geometry and surface smoothness; octrees reduce computation while retaining fine detail.Source: §1.1.4 · pp. 4–5
• Multi-view representations learn from multiple 2D images to reduce noise, incompleteness, occlusion, and illumination effects, but are most suitable for rigid objects with limited deformation.Source: §1.1.5 · p. 5

• Non-Euclidean 3D data lacks global parametrization, common coordinates, and vector-space structure, so extending 2D deep learning is not straightforward.Source: §1.2 · p. 5
• Point clouds, meshes, and graphs are principal non-Euclidean structures, although point clouds and meshes can also appear Euclidean when processed locally.Source: §1.2 · p. 5, §1.2.1 · p. 5

1.2.1 3D Point clouds

• Point clouds are easy to capture but difficult to process because they lack structure and connectivity information, creating ambiguity about surface information.Source: §1.2.1 · pp. 5–6

Graphs and meshes

• Meshes encode polygonal faces, vertices, and connectivity, yet their irregular structure, noise, missing data, and resolution variation challenge deep learning and motivate completion and inpainting methods.Source: §1.2.2 · p. 6

• Extending deep learning to 3D is difficult because objects have complex geometric structure and representations exhibit substantial structural variation.Source: §2 · p. 6
• The survey therefore classifies 3D deep learning architectures into Euclidean and non-Euclidean families according to the underlying data representation.It uses this classification to compare the learning paradigms and the features they learn.Source: §2 · p. 6

RGB-D data

• RGB-D networks exploit depth effectively for segmentation and recognition, yet they infer only some 3D properties and do not learn the object's full geometry.A two-stream RGB-D segmentation model improved accuracy by 6% over previous models, while depth helped separate objects.Source: §2.1.3 · pp. 10–11

Volumetric data

• Volumetric models represent full 3D geometry with voxel tensors, but convolutional cost grows cubically with voxel resolution, limiting their practical use.LightNet addresses this constraint with a real-time volumetric CNN using multitask learning.Source: §2.1.4 · pp. 11, 13, §2.1.5 · p. 13

Multi-view data

• MVCNN with max-view pooling outperformed ShapeNet on classification and retrieval, while later unconstrained-trajectory views surpassed both MVCNN and voxel-based ShapeNet.MVCNN processes views without a fixed order; the later model weights contributions from image pairs.Source: §2.1.5 · p. 14
• Multi-view models slightly outperform volumetric representations while requiring less computational power, although view selection remains critical and intrinsic geometry is not preserved.These limitations motivate convolutions designed directly for 3D shapes.Source: §2.1.5 · p. 15

Hybrid data representations

• FusionNet combining voxel and 2D-view networks performed best among its tested modalities, while the multi-view network exceeded volumetric CNNs with less computation and no augmentation.Its performance was comparable to Brock et al.'s state-of-the-art model without that model's heavy computation.Source: §2.1.6 · p. 16

• Non-Euclidean approaches extend deep learning to geometric 3D data, including point clouds, meshes, and graphs, but irregular structure complicates convolution.Point-cloud methods learn shape geometry, mesh methods exploit vertex connectivity, and graph methods use spectral properties for intrinsic descriptors.Source: §2.2 · p. 16, §2.2.2 · p. 18

Point clouds

• Point clouds are locally Euclidean but globally unordered, making permutation-invariant learning the central processing challenge.Their global structure lacks a specific point order, despite local neighborhoods having transformation-invariant distance structure.Source: §2.2.1 · pp. 17–18
• PointNet directly aggregates point coordinates with a symmetric function, while PointNet++ recursively partitions point sets to capture finer local patterns at higher computational cost.PointNet++ improves local feature capture but produces a more complicated architecture with larger higher-level features and increased computation.Source: §2.2.1 · p. 17
• Kd-Networks impose a kd-tree on unordered points, and the depth-map approach transforms point clouds into informative 2D maps without explicitly adapting for permutation invariance.These strategies replace direct unordered processing with structured representations that support shared weights or standard image-based processing.Source: §2.2.1 · pp. 17–18

Graphs and meshes

• Graph and mesh methods divide into spectral and spatial filtering, with Geodesic CNN defining mesh patches in local polar coordinates for geodesic convolution.Spectral methods use graph-Laplacian eigen-decomposition, whereas spatial methods operate through local neighborhoods or geometric patches.Source: §2.2.2 · pp. 18, 20, 21
• Spectral filtering is basis-dependent and computationally expensive because it is non-local and requires graph-Laplacian computation, while basis alignment may require known domain correspondences.These constraints hinder transfer across domains and processing of large-scale graphs; joint diagonalization addresses basis dependency only with correspondence information.Source: §2.2.2 · p. 19

3D recognition/classification

• Multi-view methods generally outperform other approaches for recognition and classification, although max-pooling can discard non-maximal visual cues from individual views.Combining view-, geometry-, and volumetric features is presented as a way to exploit complementary input modalities.Source: §3.2.1 · pp. 23–24
• 98.46% on ModelNet10 and 97.37% on ModelNet40 are achieved by multi-view methods, outperforming volumetric methods while establishing state-of-the-art classification results.Brock et al. [15] obtain competitive volumetric results of 97.14% and 95.54%, but with a more complex architecture and substantial data-augmentation needs.Source: §3.2.1 · p. 24

3D retrieval

• Multi-view methods usually outperform other methods in retrieval accuracy, with center-loss and triplet-loss training reported to surpass the state of the art on ModelNet40 and ShapeNet [60].The cited approach trains these losses specifically for the distance measure.Source: §3.2.2 · p. 25

3D Correspondence

• Increasing synthetic noise produces increasingly erroneous correspondence assignments on FAUST, whereas changing pose or gender affects performance much less than changing shape size.Errors include wrongly labelled vertices and color-map differences from the reference mesh.Source: §3.2.3 · pp. 27–28
• Different topology causes highly erroneous SplineCNN correspondence maps on real 3DBodyTex scans, showing that connectivity variation remains a major practical challenge.The scans were down-sampled to match FAUST's 6890 vertices but retained a different connectivity pattern.Source: §3.2.3 · p. 28

4 CONCLUSION

• Multi-view representations achieve state-of-the-art classification performance and outperform volumetric methods while providing a more efficient way to learn 3D shape properties.This conclusion concerns Euclidean deep learning methods that exploit different representations of 3D geometry.Source: §4 · p. 29
• Near-perfect correspondence results can be misleading when benchmark meshes share identical topology and already have dense one-to-one vertex correspondences.On FAUST, the mapping can become a trivial vertex-order copy rather than learned topology correspondence [46].Source: §4 · p. 29