Source-linked AI summary
Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction
Daeyun Shin, Charless C. Fowlkes, Derek Hoiem
TL;DR
The paper asks how shape representation and coordinate frame affect single-view 3D object prediction. It compares voxel and multi-surface decoders in viewer-centered and object-centered settings using a shared encoder-decoder framework. Surface representations favor novel categories, viewer-centered coordinates generalize better to novel objects, and object-centered coordinates favor familiar instances while relying more on category information.
Problem
Single-view 3D shape prediction lacked systematic evidence comparing volumetric versus surface representations and viewer-centered versus object-centered coordinates.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Representation and coordinate-frame choices materially affect both generalization and the information learned by the encoder.
Source: §1 · p. 2, §7 · p. 8Thin structures remain difficult to reconstruct, and surface reconstruction can lose information because predicted point data are sparse.
Source: §6 · p. 8
Abstract
from arXiv · showhide
The goal of this paper is to compare surface-based and volumetric 3D object shape representations, as well as viewer-centered and object-centered reference frames for single-view 3D shape prediction. We propose a new algorithm for predicting depth maps from multiple viewpoints, with a single depth or RGB image as input. By modifying the network and the way models are evaluated, we can directly compare the merits of voxels vs. surfaces and viewer-centered vs. object-centered for familiar vs. unfamiliar objects, as predicted from RGB or depth images. Among our findings, we show that surface-based methods outperform voxel representations for objects from novel classes and produce higher resolution outputs. We also find that using viewer-centered coordinates is advantageous for novel objects, while object-centered representations are better for more familiar objects. Interestingly, the coordinate frame significantly affects the shape representation learned, with object-centered placing more importance on implicitly recognizing the object category and viewer-centered producing shape representations with less dependence on category recognition.
1. Introduction
The paper systematically compares volumetric versus multi-surface shape representations and viewer-centered versus object-centered coordinates for single-image 3D shape prediction across familiarity conditions.
Motivation
- The study addresses the lack of systematic evaluation of shape representation and coordinate-frame choices in single-image 3D prediction.It considers depth and RGB inputs together with known objects, novel instances, and novel categories.Source: §1 · pp. 1–2, Figure 1 · p. 1
Coordinate frames
- Viewer-centered coordinates avoid canonical alignment and naturally support novel categories, while object-centered coordinates simplify prediction but require aligned models and predefined poses.Viewer-centered prediction represents shape relative to the input viewpoint; object-centered prediction uses a canonical pose.Source: §1 · pp. 1–2, Figure 2 · p. 2
Contributions
- The proposed surface method predicts silhouettes and depth maps from several viewpoints, then registers and merges them into a point cloud for surface computation.This provides a direct multi-surface alternative to volumetric prediction.Source: §1 · p. 2, Figure 2 · p. 2
- Surface-based representations outperform voxel representations for unfamiliar object categories and produce higher-resolution shapes.The comparison uses a shared encoder architecture and finds the advantage regardless of whether final evaluation is volumetric or surface-based.Source: §1 · p. 2
- Viewer-centered prediction generalizes better to novel objects, whereas object-centered prediction performs better for novel views of familiar instances.The coordinate choice also changes the encoder embedding, with object-centered models relying more on object categorization.Source: §1 · p. 2
2. Related work
Prior work spans volumetric, multi-surface, point-cloud, and multiview-generation approaches; this paper distinguishes its method by predicting multiple geometric views for complete-shape reconstruction.
Volumetric shape representations
- Volumetric methods generate voxel or octree shapes, while related evaluations use voxel IoU, mesh distance, and depth-map error.Prior systems include both template-deformation methods and encoder-decoder networks.Source: §2 · p. 2
Multi-surface representations
- Multi-surface methods have shown advantages in recognition because projections encode comparable information with fewer effective parameters than volumetric representations.Earlier classification systems use multiple projections as input, unlike this paper’s output-prediction setting.Source: §2 · pp. 2–3
Related multiview generation
- Related multiview-generation networks produce images or rotations, but their experiments do not necessarily perform 3D reconstruction or geometric analysis.The paper’s focus is complete-shape prediction from a single image rather than image synthesis alone.Source: §2 · p. 3
Proposed method
- The proposed method predicts depth images and silhouettes from fixed viewpoints around the viewing sphere, using six views for most experiments after finding similar results to twenty views.The output views are fixed relative to either the canonical object pose or the input view, depending on the coordinate frame.Source: §2 · p. 3
- Unlike approaches that directly generate complete volumetric shapes, this system converts predicted multiview surfaces into a point cloud and reconstructs the complete mesh as post-processing.It treats inconsistencies among output images as observational noise and does not use class labels during training.Source: §2 · pp. 2–3
3. Viewer-centered 3D shape completion
The method predicts complete 3D shape from one depth or RGB image in either input-view coordinates or canonical object coordinates, without category or viewpoint labels.
Coordinate frames
- Viewer-centered supervision aligns the predicted shape to the input image viewpoint, whereas object-centered supervision uses canonical model coordinates.The viewer-centered output’s zero-azimuth, zero-elevation direction corresponds to the input viewpoint.Source: §3 · p. 3
- Viewer-centered training can operate across categories without manually specified intra-category alignment, but requires separate target outputs for different viewpoint inputs.This improves flexibility while increasing training-data storage costs.Source: §3 · p. 3
Training assumptions
- Training uses only geometric or photometric input information and assumes only that gravity is known as the downward direction.No object-category label or input-viewpoint side information is supplied.Source: §3 · p. 3
Network architecture
- The network encodes input depth and silhouette into view-specific features, then uses ten output branches to synthesize silhouettes and front/back depth images.The fully connected layers are independently parameterized across branches, while the up-convolutional decoders share parameters.Source: Figure 3 · p. 4
4. Network architectures for shape prediction
The paper uses a shared encoder-decoder framework to compare multi-surface and voxel shape prediction in viewer-centered and object-centered coordinates. Multi-surface prediction generates silhouettes and depth maps from fixed viewpoints, then reconstructs a mesh, while the voxel baseline directly predicts occupancy grids.
4. Network architectures for shape prediction
- A shared encoder-decoder architecture swaps decoders and coordinate frames to compare surface-based versus voxel and viewer-centered versus object-centered prediction.The networks accept depth-plus-silhouette inputs, with an RGB variant, and can produce either coordinate-frame representation.Source: §4 · p. 4
4.1. Generating multi-surface depth and silhouettes
- Multi-surface prediction generates silhouettes and paired depth maps from evenly spaced viewpoints around the object.The system uses cameras placed at dodecahedron vertices; experiments commonly use six views because they perform similarly to twenty while reducing training and evaluation cost.Source: §4.1 · p. 4
- The multi-surface network combines silhouette logistic loss with depth-map mean squared error, using k = 0.2 to weight the losses.Depth loss is computed where the silhouette label is 1.Source: §4.1 · p. 4
4.2. Reconstructing multi-surface representations
- Predicted multiview depth images are converted into a single triangulated mesh with FSSR, which reduces prediction noise and surface-distance error.The method treats inconsistencies in neural predictions as observational noise during post-processing.Source: §4.2 · pp. 4–5
4.3. Generating voxel representations
- The voxel baseline directly predicts a 48 × 48 × 48 occupancy grid in the input camera coordinates using 3D up-convolutional layers.It is trained with logistic loss on binary voxel-occupancy labels.Source: §4.3 · p. 5, Figure 4 · p. 5
5. Experiments
The experiments train networks on synthetic data and evaluate single-view shape prediction across depth and RGB inputs, familiarity levels, representations, and coordinate frames. Evaluation combines voxel, surface, silhouette, and depth metrics, with additional qualitative assessment of reconstructed shapes.
5.1. Datasets
- Synthetic training and held-out evaluation cover novel views, novel instances from known categories, and novel categories, with depth experiments on SHREC’12 and RGB experiments tested on real PASCAL 3D+ images.The RGB setting uses 2.4M synthetic ShapeNetCore renderings and includes occlusion and background clutter at test time.Source: §5.1 · p. 5
5.2. Evaluation metrics and processes
- Voxel IoU measures overlap between filled predicted and ground-truth voxels, while surface distance averages normalized point-to-triangle distances between reconstructed and ground-truth meshes.Multi-surface experiments additionally evaluate silhouette IoU and depth error across predicted views.Source: §5.2 · pp. 5–8
- Multi-surface representations outperform voxel representations on novel classes and perform similarly on novel views and novel instances.The reconstructed multi-surface shapes also look qualitatively better because they encode higher-resolution geometry.Source: Figure 5 · p. 6
- The evaluation tables compare viewer-centered voxel and multi-surface decoders using voxel IoU, silhouette IoU, and depth error, including a per-category RGB comparison against 3D-R2N2.Table 5 reports PASCAL 3D+ voxel IoU for object-centered and viewer-centered multi-surface models alongside the voxel-based baseline.Source: Table 1 · p. 6, Table 2 · p. 7, Table 3 · p. 7
6. Discussion
The discussion finds that multi-surface representations outperform voxels for novel classes, while coordinate-frame advantages depend on object familiarity and affect learned category dependence.
Multi-surface vs. voxel shape representations
- Multi-surface representations outperform voxel-based methods for novel classes, perform similarly for novel views and instances, and produce higher-resolution shapes.Thin structures remain difficult, and surface reconstruction can lose information from sparse depth points.Source: §6 · p. 8
Viewer-centered vs. object-centered coordinates
- Viewer-centered prediction performs better for novel models and classes, whereas object-centered prediction performs better for novel views of training instances.Viewer-centered prediction also encodes pose, making it harder and sometimes penalizing otherwise accurate shapes when pose is misaligned.Source: §6 · p. 8
- Viewer-centered models produce more accurate shapes on real RGB images, but their poses can be wrong by 15–20 degrees after dataset transfer.Object-centered prediction slightly wins quantitatively in this RGB setting despite the qualitative shape advantage for viewer-centered outputs.Source: §6 · p. 8
- Object-centered models more directly reflect category recognition and may predict a visually plausible shape from the wrong category, unlike viewer-centered models.Viewer-centered errors instead tend toward oversimplification or slightly incorrect pose.Source: §6 · p. 8
Implications for object recognition
- Features from the viewer-centric shape-and-pose encoder enable an object classifier to outperform an end-to-end ResNet by 1%.The authors state that this implication for recognition requires more study.Source: §6 · p. 8
7. Conclusion
The paper compares volumetric and surface-based representations and object-centered versus viewer-centered coordinates for single-image 3D shape prediction. Surface-based methods favor novel shapes, while viewer-centered prediction generalizes better to novel objects, with further work needed on fusion, octrees, and recognition.
7. Conclusion
- Using the same encoder architecture, experiments show surface-based representations predict novel object shapes better than volumetric representations, likely because they encode detail with fewer parameters.The paper also reports that viewer-centered prediction generalizes better to novel objects than object-centered prediction.Source: §7 · p. 8
- Better alignment and fusion of predicted depth maps could further improve surface-based prediction, while octree representations require additional comparison.The paper also identifies object categorization and shape/pose prediction as needing further exploration.Source: §7 · p. 8