Source-linked AI summary
SurfNet: Generating 3D shape surfaces using deep residual networks
Ayan Sinha, Asim Unmesh, Qixing Huang, Karthik Ramani
TL;DR
The paper addresses the computational burden and limited surface focus of voxel-based 3D learning by directly generating shape surfaces. It builds consistent geometry images and uses residual-network extensions to generate them from images or parametric representations. The resulting models generate rigid and non-rigid surfaces, including unseen-image reconstructions, pose interpolations, and new shape surfaces.
Problem
Voxel-based 3D learning incurs high computational overhead even though most geometry information lies on surfaces.
Method
The paper creates consistent geometry images for shape categories and extends deep residual networks to generate x, y, z surface-coordinate images from images or parametric representations.
Results
The networks generate rigid and non-rigid shape surfaces, reconstruct surfaces from previously unseen images, estimate implicit viewpoints, and interpolate between poses and shapes.
Takeaways & Limitations
Geometry-image generation provides a learned representation that supports surface interpolation, novel surface generation, and reconstruction across rigid and non-rigid categories.
Takeaways & Limitations
The current approach is limited to genus-0 surfaces.
Abstract
from arXiv · showhide
3D shape models are naturally parameterized using vertices and faces, \ie, composed of polygons forming a surface. However, current 3D learning paradigms for predictive and generative tasks using convolutional neural networks focus on a voxelized representation of the object. Lifting convolution operators from the traditional 2D to 3D results in high computational overhead with little additional benefit as most of the geometry information is contained on the surface boundary. Here we study the problem of directly generating the 3D shape surface of rigid and non-rigid shapes using deep convolutional neural networks. We develop a procedure to create consistent `geometry images' representing the shape surface of a category of 3D objects. We then use this consistent representation for category-specific shape surface generation from a parametric representation or an image by developing novel extensions of deep residual networks for the task of geometry image generation. Our experiments indicate that our network learns a meaningful representation of shape surfaces allowing it to interpolate between shape orientations and poses, invent new shape surfaces and reconstruct 3D shape surfaces from previously unseen images.
1. Introduction
The paper targets direct generation of 3D shape surfaces without relying on computationally burdensome volumetric learning. It introduces consistent geometry images and residual-network extensions for generating surfaces from images or parametric inputs.
- 3D content generation is increasingly important, but applying deep learning in 3D faces substantial computational overhead from the extra dimension.
- Surface-focused representations are motivated by the observation that most 3D shape information lies on the surface rather than throughout the volume.
- Naively creating independent geometry images across a shape category fails to produce coherent 3D surfaces.
- The paper creates consistent geometry images for genus-0 shape categories by solving a large-scale correspondence problem.
- Extended deep residual networks generate x, y, z surface-coordinate geometry images with implicit pose estimation and preservation of high-frequency features.
- The resulting models generate surfaces from images or parametric representations, including unseen images, intermediate poses, and interpolations between shapes.
2. Related Work
Related work develops 3D content through primitives, patches, components, geometric models, and learned representations. Existing deep-learning approaches address recognition, completion, reconstruction, and shape generation using volumetric or vector-based representations.
- Earlier 3D content methods synthesized primitives and surface patches, while later approaches assembled shapes from components using probabilistic or deep-learned models.
- Prior work estimates wireframes, learns deformable models, and jointly analyzes images with shape collections for single-image 3D reconstruction.
- Deep-learning research introduced volumetric and vector representations for 3D shape recognition, completion, reconstruction, and generation.
3. Dataset Creation
The dataset pipeline represents surfaces as regular-grid geometry images and constructs training data for non-rigid hands and rigid cars and airplanes. It combines controlled rendering, spherical parameterization, and mesh correspondence to make category-level geometry images consistent.
- Geometry images remesh arbitrary surfaces onto a completely regular grid while encoding surface coordinates.
- Non-rigid shapes: The hand dataset uses an 18-DOF kinematic model with 1065 vertices and 2126 faces sharing connectivity across mesh instances.
- Non-rigid shapes: The hand data contains 200,000 mesh files with joint parameters, vertex coordinates, and normalized 128 × 128 depth images.
- Non-rigid shapes: Hand geometry images encode x, y, z coordinates on a 64 × 64 × 3 grid and vary with articulation.
- Rigid or man-made shapes: Rigid-shape preprocessing voxelizes ShapeNet meshes, preserves holes and sharp edges with an α-radius, and removes non-genus-0 surfaces for spherical parameterization.
- Rigid or man-made shapes: Independent parameterization creates inconsistent geometry images because different cuts produce rotations and translations for shapes in the same pose.
- Rigid or man-made shapes: The method uses D2-descriptor similarity, spectral clustering with K = 3, exemplar shapes, and blended intrinsic maps to establish dense correspondences.
- Rigid or man-made shapes: Rendered rigid-object RGB images use four elevation angles and 24 azimuth angles, paired with 64 × 64 × 3 geometry images encoding x, y, z.
4. Deep Network Architecture
The architecture extends deep residual networks to generate geometry-image channels for reconstructing or synthesizing 3D surfaces. Separate networks learn x, y, and z coordinates, using shape-aware losses to preserve sharp geometric features.
- Network design: The network extends deep residual architectures for generating geometry-image feature channels from images or parametric representations.Its building blocks combine standard convolutions with residual upsampling and downsampling blocks.
- Residual blocks: Up-residual blocks increase output size, while down-residual blocks decrease it through specialized sampling filters followed by standard residual blocks.Downsampling uses stride-2 convolution, whereas upsampling uses convolution transpose with 2× upsampling.
- Coordinate generation: Separate networks learn the x, y, and z geometry-image channels instead of a single network learning a category mean shape.The channels are concatenated into a single 64 × 64 × 3 geometry image for the hand surface.
- Training objective: A curvature-weighted shape-aware loss gives higher weight to high-curvature regions, helping preserve sharp features during reconstruction.The same loss is used for both rigid and non-rigid datasets.
- Inputs: For rigid shapes, generation uses a one-hot class label and azimuth and elevation view angles; for non-rigid hands, it uses an 18-dimensional joint-angle vector.Rigid-shape angles are represented by their sine and cosine to enforce periodicity.
- Surface synthesis: For rigid-shape generation, the residual geometry image of x, y, and z coordinates is summed with a base geometry image to produce the surface plot.The pipeline generates the surface from class and view parameters.
5. Experiments
The experiments generate non-rigid hand and rigid car and airplane surfaces from images and parametric inputs. Results show reconstruction, smooth interpolation, novel intermediate surfaces, and correspondence refinement, while also exposing failures under difficult image conditions.
- Non-rigid shapes: The network reconstructs full hand surfaces from depth images, recovering articulation close to ground truth even under occlusion.Separate networks generate x, y, and z geometry images whose combined output preserves spatial localization.
- Non-rigid shapes: Interpolated joint-angle inputs produce smooth transitions between hand poses, indicating a learned abstraction rather than memorization.This behavior appears for both 15-dimensional and 18-dimensional joint-angle interpolations.
- Rigid or man-made shapes: From single RGB images, the network estimates both viewpoint and 3D surface for diverse cars and airplanes.Reconstructions preserve sharp edges but can lack smoothness on flat regions such as car windshields.
- Rigid or man-made shapes: The method reconstructs car and airplane surfaces accurately relative to Kar et al., but produces artifacts near geometry-image boundaries and fails on some difficult images.Failures mainly occur with low contrast, poor texture, or viewpoints outside the training azimuth and elevation ranges.
- Rigid or man-made shapes: Varying one-hot encodings and azimuth angles generates realistic unseen intermediate poses and smooth morphs between vehicle shapes.Examples include transitions among convertibles, pickup trucks, sports cars, SUVs, vans, jeeps, and airplane surfaces.
- Correspondence: The network reduces noisy mesh correspondences by mapping rendered inputs to geometry images with one-to-one correspondence to a base mesh.The resulting color gradients become smoother, suggesting feedback can help rectify correspondence information.
6. Limitations and Future Work
The approach is currently restricted to genus-0 surfaces, while future work targets broader topology, improved correspondences, and simultaneous multi-category, multi-channel learning.
- Limitations: The current approach is limited to genus-0 surfaces.The authors identify removing this restriction as future work.
- Future Work: Future work includes improving correspondences and increasing training-set size with feedback or more sophisticated correspondence methods.The authors also propose learning multiple shape categories and all feature channels simultaneously without performance degradation.