Learning to Generate Chairs, Tables and Cars with Convolutional Networks

Alexey Dosovitskiy; Jost Tobias Springenberg; Maxim Tatarchenko; Thomas Brox

Source-linked AI summary

Learning to Generate Chairs, Tables and Cars with Convolutional Networks

Alexey Dosovitskiy, Jost Tobias Springenberg, Maxim Tatarchenko, Thomas Brox

arXiv:1411.5928v4Nov 21, 2014cs.CVcs.LGcs.NE

TL;DR

The paper asks whether supervised networks can generate object images from high-level descriptions without merely memorizing training examples. It trains up-convolutional generators on rendered chairs, tables, and cars, finding generalization to unseen views and styles, interpolation across objects and classes, and correspondences between objects.

Problem
The paper addresses how to generate images from high-level object descriptions while testing whether the generator generalizes beyond memorizing training examples.
Method
The authors train supervised up-convolutional networks to map object style, viewpoint, and transformation parameters to RGB images and segmentation masks.
Results
The networks generalize to unseen viewpoints and object styles, transfer viewpoint knowledge across classes, interpolate objects, generate new styles, and find object correspondences.
Takeaways & Limitations
The learned representation supports meaningful morphing and suggests that networks trained across object classes capture aspects of 3D shape and geometry.
Takeaways & Limitations
Generation handles extreme color transformations more easily than large spatial changes, especially translations.

Abstract

from arXiv · show

We train generative 'up-convolutional' neural networks which are able to generate images of objects given object style, viewpoint, and color. We train the networks on rendered 3D models of chairs, tables, and cars. Our experiments show that the networks do not merely learn all images by heart, but rather find a meaningful representation of 3D models allowing them to assess the similarity of different models, interpolate between given views to generate the missing ones, extrapolate views, and invent new objects not present in the training set by recombining training instances, or even two different object classes. Moreover, we show that such generative networks can be used to find correspondences between different objects from the dataset, outperforming existing approaches on this task.

1 INTRODUCTION

The paper uses supervised up-convolutional networks to generate images from high-level object descriptions rather than recognize images. Experiments show generalization across unseen views, object styles, and even object classes.

1 INTRODUCTION: The authors address image generation conditioned on high-level descriptions, focusing on learning the generator rather than the image distribution.The network receives descriptive inputs and produces images, while training uses standard supervised learning.
1 INTRODUCTION: Given a model identity, viewpoint, and optional transformations, the network generates RGB projections of rendered 3D objects.Training minimizes Euclidean reconstruction error for the generated image.
1 INTRODUCTION: The experiments test whether the network generalizes beyond memorizing training examples.The paper specifically examines previously unseen inputs and views.
1 INTRODUCTION: The networks transfer viewpoint knowledge within and between object classes, interpolate between objects, perform feature arithmetic, and generate new styles.The reported transfers include using tables to infer unseen chair elevations.
1 INTRODUCTION: The paper also applies the generative networks to finding correspondences between different objects in the training set.This application is presented alongside the generalization experiments.

2 RELATED WORK

The paper differs from prior generative modeling work by assuming high-level labels and training a supervised, controllable generator. This enables larger output images and explicit selection of generated content, at the cost of requiring sufficiently complete labels.

2 RELATED WORK: Earlier generative-modeling work commonly learns an image distribution and latent representation without supervision, using models such as RBMs, DBMs, and directed graphical models.These approaches include undirected models, mixture models, autoregressive models, and stochastic neural networks.
2 RELATED WORK: Conditional and semi-supervised generative models incorporate labels, but prior examples were often limited to small models and images.The paper contrasts these approaches with its supervised generation setting.
2 RELATED WORK: Neural-network image-generation research also includes large up-convolutional and adversarial models capable of generating realistic or high-fidelity images.The cited developments include conditional convolutional generative models.
2 RELATED WORK: The paper assumes a given high-level latent representation and uses supervised training to generate 128 × 128 pixel images with direct control over which images are produced.Most previous works cited are described as reaching at most 48 × 48 pixels.
2 RELATED WORK: The main trade-off is that the approach requires a label that fully describes the image-generation conditions.Related work partially relaxes this requirement by assuming only partial knowledge of changing and fixed scene parameters.
2 RELATED WORK: Prior viewpoint-generation work used fully connected networks and random sampling, making it unsuitable for the large and diverse chair images modeled here.The comparison concerns modeling desired views of objects from image inputs.

3 MODEL DESCRIPTION

The model maps object identity, viewpoint, and transformation parameters to RGB images and segmentation masks through an up-convolutional generator. It is trained on rendered chairs, tables, and cars with reconstruction losses and augmented transformations.

3 MODEL DESCRIPTION: The generator produces object images from high-level style, orientation, color, brightness, and related parameters, reversing the usual recognition direction.The task converts descriptions into images instead of images into compressed representations.
3 MODEL DESCRIPTION: Each input combines one-hot model identity, sine-and-cosine viewpoint encoding, and transformation parameters; targets contain an RGB image and segmentation mask.The mask facilitates separating generated objects from the background.
3 MODEL DESCRIPTION: Randomized transformations augment training variation and reduce overfitting, including rotation, translation, zoom, stretching, hue, saturation, and brightness changes.The specified ranges include up to ±12° rotation, ±10% translation, and 100%–135% zoom.
3.1 Network architectures: The network first builds a shared hidden representation from the inputs, then expands it through fully connected and up-convolutional layers to generate images.The three input streams are processed independently before their representations are concatenated.
3. MODEL DESCRIPTION: Viewpoint angles use sine and cosine representations so periodic values such as 0° and 359° remain close.This encoding addresses the periodicity of camera angles.
3.1 Network architectures: The architecture reshapes a 16,384-dimensional vector into an 8 × 8 multichannel map and repeatedly upsamples and convolves it toward the output.Upsampling increases feature-map span, while subsequent convolutions process the expanded maps.
3.2 Network training: Training minimizes RGB reconstruction and segmentation losses, with λ controlling their trade-off.RGB loss is squared Euclidean distance; segmentation uses either squared Euclidean distance or negative log-likelihood with λ values 0.1 or 100.
3.3 Probabilistic generative modeling: The training data consist of rendered 3D chair, table, and car models, including 809 filtered chairs, 7,124 cars, and 1,000 tables.Chair renderings use 62 viewpoints, while car and table renderings use 180 viewpoints per model.

4 TRAINING PARAMETERS

The study compares up-convolutional architectures, training-set sizes, and data augmentation, using reconstruction quality and interpolation behavior to guide the chosen network and training setup.

Optimization: Training uses Caffe with Adam, mini-batches of 128, an initial learning rate of 0.0005, scheduled halvings, and 500,000 total iterations.The Adam momentum parameters are β1 = 0.9 and β2 = 0.999, with ϵ = 10^-6.
Image resolution: Most networks generate 128 × 128 images, while viewpoint interpolation uses 64 × 64 images and car experiments also test 256 × 256 images.The larger car-image networks train without observed complications.
Network architectures: The deeper 1s-S-deep network performs significantly better than the alternatives qualitatively and quantitatively, so it is used in most experiments.Figure 4 shows generated images and Table 1 reports reconstruction errors.
Training-set size and augmentation: Increasing the car training-set size and applying data augmentation have qualitatively similar effects on reconstruction errors.Training-set size is varied with the architecture fixed, and Table 2 supports the observations.
Training-set size and augmentation: Without augmentation, the network models training images better but interpolates less smoothly between one-hot style vectors.Augmentation worsens fine-detail reconstruction while improving the expected generalization behavior in the interpolation experiment.

5 EXPERIMENTS

The experiments show that supervised generative networks generalize beyond memorizing training images, generating unseen viewpoints and styles, transferring 3D-structure knowledge across object classes, and supporting correspondence finding and object generation.

Interpolation between viewpoints: The networks generate previously unseen views by interpolating between training views, indicating an internal representation that supports chair similarity judgments.With knowledge transfer, interpolation remains reasonable even when only one view is available, although some fine details can be lost.
Interpolation between viewpoints: All viewpoints in a source set dramatically improve missing-viewpoint prediction on target objects, especially when few target viewpoints are available.The evaluation compares average squared error with and without transfer and also examines nearest-neighbor baselines.
Elevation transfer and extrapolation: Training jointly on chairs and tables enables unseen-elevation generation, transferring viewpoint knowledge and aspects of 3D structure between object classes.Chairs provide elevations absent from their own dataset, while tables provide broader elevation coverage; generated elevations are not always exact.
Interpolation between styles: Interpolating style vectors invents realistic intermediate objects, combines features from three chairs, and can morph between chairs and tables despite no intermediate examples in training.Cross-class morphings are slightly worse than within-class morphings but remain overall very good.
Feature arithmetics: Feature-space arithmetic changes chair attributes such as armchair style or back type while preserving the corresponding alternative feature.Similar results arise when manipulating the input style vectors or FC1 and FC2 outputs.
Random chair generation: Variational-bound-based training provides a more principled route to Gaussian-based random chair generation, with quality and diversity roughly similar to multi-chair heuristics.Increasing the KL-divergence weight produces more diverse chairs, whereas the non-variational control is not very diverse.

6 ANALYSIS OF THE NETWORK

The analysis probes how information is organized across layers and shows that meaningful transformations emerge from coordinated hidden representations rather than isolated units alone.

Layer-wise representations: Single activations in class-stream layers produce chair-like images, with FC3 and FC4 containing more varied joint representations.FC1 and FC2 mainly vary chair type, while FC3 and FC4 show larger structural variation.
Transformation neurons: Specialized FC4 neurons control transformations including translation, zoom, stretching, rotation, saturation, and hue.Increasing one neuron while holding the other FC4 activations fixed changes the generated image accordingly.
Transformation neurons: The weights connected to transformation neurons separate spatial transformations from color and brightness changes.The analyzed weights show that different output channels contribute to these transformation categories.
Spatially distributed representations: Single high-level deconvolutional neurons produce blurry clouds, whereas sharp chairs emerge when neighboring FC5 activations are combined across larger spatial regions.The active region expands from 2 × 2 to 8 × 8, moving from near-single-neuron to whole-image activation.
Artifact compensation: Regular patterns in upconv3 feature maps compensate for high-frequency artifacts caused by fixed filter sizes and regular-grid upsampling.Zeroing these feature maps produces high-frequency artifacts in generated chairs.

7 CONCLUSIONS

The paper demonstrates supervised up-convolutional networks for image generation from high-level object information. The learned representation supports meaningful morphing, creative chair generation, and shared 3D-shape understanding across object classes.

Supervised convolutional networks generate images from high-level style, viewpoint, and lighting information, extending their use beyond discriminative tasks.
The networks learn an implicit representation that smoothly morphs between object views or instances with meaningful intermediate images.
Stochastic training enables the networks to invent new chair styles from random noise and suggests shared understanding of 3D shape across two object classes.

APPENDIX A: TRAINING USING A VARIATIONAL

The appendix formulates random chair generation as conditional probabilistic modeling with latent variables and trains an inference network jointly with the generator using a variational objective.

Random chair generation is formulated using latent states z conditioned on chair identity, viewpoint, and transformation parameters.The expanding RGB and segmentation generator components define image and segmentation likelihoods under the latent representation.
Image likelihoods use a multivariate Gaussian model, simplified by assuming diagonal covariance Σ = Iσ.
The marginal log likelihood combines image and segmentation-mask likelihoods while integrating over the latent distribution.
An approximate Gaussian inference distribution q(z | c) is parameterized by a two-layer fully connected network predicting latent means and variances.
The inference and generator networks are jointly optimized with a variational lower-bound objective using stochastic gradient descent and one latent sample per data point.