Source-linked AI summary
Learning a Predictable and Generative Vector Representation for Objects
Rohit Girdhar, David F. Fouhey, Mikel Rodriguez, Abhinav Gupta
TL;DR
The paper asks how to represent objects in a vector space that captures 3D structure yet remains predictable from 2D images. It introduces the TL-embedding network, which joins a voxel autoencoder with an image ConvNet in a shared embedding space. Experiments report generative reconstruction, image-to-voxel prediction, and retrieval across CAD and natural-image data.
Problem
Existing approaches separately provide discriminative image representations or generative 3D models, leaving a representation that is both 3D-generative and predictable from images underdeveloped.
Method
The TL-embedding network joins a 3D voxel autoencoder with a discriminatively trained ConvNet through a shared 64-dimensional embedding space.
Results
Experiments show the embedding supports novel CAD reconstruction, full 3D voxel prediction from images, fast natural-image CAD retrieval, and smooth, discriminative, arithmetic structure.
Takeaways & Limitations
A single learned embedding can support both 3D generation and 2D perception across CAD and natural-image tasks.
Takeaways & Limitations
The abridged version contains fewer and lower-resolution results, and the task lacks standard baselines or datasets.
Abstract
from arXiv · showhide
What is a good vector representation of an object? We believe that it should be generative in 3D, in the sense that it can produce new 3D objects; as well as be predictable from 2D, in the sense that it can be perceived from 2D images. We propose a novel architecture, called the TL-embedding network, to learn an embedding space with these properties. The network consists of two components: (a) an autoencoder that ensures the representation is generative; and (b) a convolutional network that ensures the representation is predictable. This enables tackling a number of tasks including voxel prediction from 2D images and 3D model retrieval. Extensive experimental analysis demonstrates the usefulness and versatility of this embedding.
1 Introduction
The paper seeks an object embedding that is both generative in 3D and predictable from 2D images. The TL-embedding network joins an autoencoder and an image-based ConvNet to learn this representation and supports reconstruction, voxel prediction, and retrieval.
- Existing discriminative models distinguish image classes but do not necessarily encode 3D properties or provide strong generative capabilities.
- Existing 3D generative models address object structure but do not guarantee that their representations can be inferred from images.
- The proposed embedding must reconstruct objects in 3D while remaining easily inferable from 2D images, despite these criteria often being at odds.
- The TL-embedding network joins a 3D autoencoder and a discriminatively trained ConvNet through a shared 64-dimensional embedding space.
- During training, paired voxel maps and rendered images jointly optimize voxel reconstruction and prediction of the intermediate embedding.
- Experiments on CAD data and natural images show generative 3D reconstruction, 2D-to-3D voxel prediction, fast retrieval, and useful embedding properties.
2 Related Work
Prior work separately models visual representations, explicit 3D structure, or image-to-3D inference. This paper connects these directions by learning a representation that can generate 3D voxel maps from images and support retrieval without assuming an exact match.
- Earlier embedding research largely focused on reconstructive or generative models of 2D pixels, with 3D properties modeled only implicitly.
- 3D generative approaches explicitly represent object structure but generally do not address mapping images to that 3D structure.
- Image-based 3D inference has commonly targeted global properties such as object viewpoint or class rather than complete object geometry.
- Unlike alignment-based or non-generative retrieval methods, the proposed approach generates a 3D voxel map and can potentially generalize to previously unseen objects.
- The TL-embedding network combines generative 3D modeling with image-predictable representations through its training and testing architecture.
3 Our Approach
The TL-embedding network is designed to make object representations both generative and predictable. It trains a voxel autoencoder alongside an image-to-embedding ConvNet, using rendered image–voxel pairs and staged optimization.
- The target representation should generate 3D voxels and be predictable from a 2D object image, because both properties support image understanding tasks.
- The TL-embedding network uses a 3D autoencoder for generative structure and a ConvNet with an embedding loss for image predictability.
- Training requires paired RGB images and voxel maps, obtained by rendering CAD models with varied random backgrounds.
- The autoencoder receives a 20 × 20 × 20 voxel grid, encodes it to a low-dimensional vector, and decodes it back to a voxel grid.
- The autoencoder is trained with cross-entropy against the original voxel input, while the image network regresses the 64D embedding using Euclidean loss.
- At test time, the encoder is removed and the image embedding is connected to the decoder to predict a 3D voxel map.
- Training proceeds in three stages: independently training the autoencoder, regressing image features to its embedding, and continuing network training jointly.
4 Experiments
The experiments test whether the learned representation is generative in 3D, predictable from images, and structurally meaningful across reconstruction, voxel prediction, and retrieval tasks.
- Embedding Analysis: Linear interpolation produces smooth, physically plausible transitions, while combining latent dimensions generates previously unseen models that blend properties from two objects.The latter is assessed by comparing generated reconstructions with nearest training models.
- Embedding Analysis: Scaling individual latent dimensions produces consistent reconstruction effects across objects, suggesting that some dimensions encode semantically meaningful properties.The analysis clamps all other dimensions and varies one selected dimension.
- Embedding Analysis: 97.6% AP is achieved for CAD reconstruction, while the method reduces average error rate by 25% relative to PCA using the same number of dimensions.The reconstruction advantage remains after joint training makes the embedding more predictable from image space.
- Embedding Analysis: 74.4% classification accuracy is obtained on ModelNet40 using the learned representation without class information during representation learning.This is within 2.6% of a voxel method that uses class information and discriminative fine-tuning.
- Voxel Prediction: The TL-Network outperforms direct voxel prediction and the no-joint-training ablation because its embedding is both lower-dimensional and explicitly forced to be predictable.The method performs strongly on CAD data and remains effective on challenging natural images with clutter, truncation, and occlusion.
- CAD Retrieval: On IKEA retrieval, the method beats the baseline on all categories except tables/desks, while ambiguous images containing multiple objects tend to produce retrieval errors.Fine differentiation is especially difficult for tables because many models are similar and the representation is coarse.
Learning a Predictable and Generative
This abridged version presents fewer, lower-resolution results than the full paper. A higher-resolution version is available on the project webpage.
- The document is an abridged version of the paper.
- It contains fewer results than the full version.
- The included results are low-resolution, with a full high-resolution version available online.
1 Reconstruction Results on Synthetic Test Data
The synthetic test-data section shows randomly selected images and their corresponding reconstructions from test-model renderings.
- The section presents randomly selected images from the synthetic test data.
- It pairs the selected images with corresponding reconstructions.
- The examples come from 23,975 renderings of 4,058 test models.
2 Reconstruction Results on Natural Images from IKEA Dataset
The IKEA-dataset section shows selected natural images alongside their 3D reconstructions.
- The section presents selected natural images from the IKEA dataset.
- Each example is paired with a reconstruction.
- The examples illustrate reconstruction results on IKEA images.
3 Nearest Neighbor on Natural Images from IKEA Dataset
The IKEA-dataset section presents selected natural images together with their nearest retrieved 3D models.
- The section shows selected natural images from the IKEA dataset.
- It displays 3D model nearest neighbors for the selected images.
- The examples connect natural-image inputs with nearest-neighbor retrieval results.
4 Comparison with Kar et al. [1] (3D Prediction)
This section compares 3D predictions with Kar et al. and presents qualitative reconstructions on randomly selected Chair/Sofa images from PASCAL 3D+.
- 4 Comparison with Kar et al. [1] (3D Prediction): The section reports a quantitative evaluation of 3D prediction, following the stated comparison and voxelization procedure.The supplied passages identify the evaluation section and its comparison setup but do not provide a numerical result.
- 4 Comparison with Kar et al. [1] (3D Prediction): The evaluation converts Kar et al.’s predictions and ground truth to OBJ files before alignment and voxelization for comparison.The output points are aligned using the authors’ evaluation script, then voxelized to match the proposed approach.
- 4 Comparison with Kar et al. [1] (3D Prediction): Qualitative comparisons use randomly picked Chair/Sofa images from PASCAL 3D+.Complete qualitative results are stated to cover all 254 chair/sofa images on the project website.
5 Comparison with Li et al. [2] (Image based 3D Model Nearest-neighbor search)
The proposed method is compared with Li et al. on image-based CAD model retrieval, where Li et al. has several task-specific advantages but the proposed method still performs strongly.
- 5 Comparison with Li et al. [2] (Image based 3D Model Nearest-neighbor search): 82% recall@10 is obtained versus approximately 95% for Li et al. on their 315-image, 105-model labeled evaluation set.The comparison uses the recall@10 metric and the approximate value reported from Li et al.’s Figure 8.
- 5 Comparison with Li et al. [2] (Image based 3D Model Nearest-neighbor search): Li et al.’s method uses hand-crafted features from underlying 3D models, whereas the proposed method learns from a coarse 20^3 voxel grid.These differences give Li et al.’s method an advantage in capturing fine-grained model details.
- 5 Comparison with Li et al. [2] (Image based 3D Model Nearest-neighbor search): Li et al.’s approach is designed for discrimination between similar models and is class-specific, while the proposed objective is purely reconstructive.The passage identifies both properties as advantages for nearest-neighbor retrieval.
6 More embedding space analysis
The embedding-space analysis includes additional interpolation results between randomly selected models.
- 6 More embedding space analysis: The section presents further analysis of the learned embedding space.The supplied passage identifies this as additional interpolation analysis.
- 6 More embedding space analysis: The reported interpolation results are generated between two randomly picked models.The passage describes the results as randomly picked.
- 6 More embedding space analysis: The analysis examines interpolation behavior rather than a single fixed model pair.Both endpoints are selected randomly for the reported results.