Source-linked AI summary
Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
Jimei Yang, Scott Reed, Ming-Hsuan Yang, Honglak Lee
TL;DR
The paper studies how to synthesize novel rotated views of a 3D object from one image despite ambiguous shape and pose inference. It introduces a recurrent convolutional encoder-decoder with action-controlled transformations, demonstrating view synthesis for faces and chairs and emergent identity–pose disentangling.
Problem
Single-image 3D view synthesis is difficult because projection loses information needed to infer an object’s shape and pose.
Method
A recurrent convolutional encoder-decoder uses identity units, recurrent pose units, and per-step rotation actions to render transformed views.
Results
The model generates smooth, high-quality novel-object rotations for faces and chairs, significantly outperforming KNN5 with 179 MSE versus 310 MSE and a 42% relative improvement.
Takeaways & Limitations
Training on rotation prediction produces view-invariant features and disentangles chair identity from pose without explicit disentangling regularization.
Abstract
from arXiv · showhide
An important problem for both graphics and vision is to synthesize novel views of a 3D object from a single image. This is particularly challenging due to the partial observability inherent in projecting a 3D object onto the image space, and the ill-posedness of inferring object shape and pose. However, we can train a neural network to address the problem if we restrict our attention to specific object categories (in our case faces and chairs) for which we can gather ample training data. In this paper, we propose a novel recurrent convolutional encoder-decoder network that is trained end-to-end on the task of rendering rotated objects starting from a single image. The recurrent structure allows our model to capture long-term dependencies along a sequence of transformations. We demonstrate the quality of its predictions for human faces on the Multi-PIE dataset and for a dataset of 3D chair models, and also show its ability to disentangle latent factors of variation (e.g., identity and pose) without using full supervision.
1 Introduction
The paper addresses single-image 3D view synthesis, an ill-posed problem caused by information loss during projection. It proposes recurrent, action-controlled transformations that support long trajectories and weakly supervised disentangling of identity and pose.
- Problem: Single-image prediction of rotated 3D object appearances is ill-posed because projecting 3D objects into images loses information.Existing geometry-based approaches may require multiple images or known category-specific 3D models and can suffer alignment and texture-synthesis problems.
- Approach: The proposed recurrent convolutional encoder-decoder models pose-manifold traversal as a sequence of locally controlled transformations.Its components include an encoder, shared identity units, recurrent pose units with rotation actions, and a decoder.
- Approach: Control signals at each time step allow rotation sequences of varying length rather than restricting training to one specific sequence.The action inputs instruct the model how to move locally along the pose manifold.
- Contributions: The model learns realistic rotation trajectories for human faces and 3D chair models while generating out-of-plane rotations.The contribution is framed as learning transformations for these two object domains.
- Contributions: Image synthesis training also yields view-invariant features and weakly supervised disentangling of identity and pose without class labels.The paper highlights this effect particularly for longer-term prediction.
2 Related Work
Related work spans transformation-aware image generation, chair rendering, inverse graphics, and latent-factor disentangling. The paper distinguishes its approach through recurrent multi-step prediction, per-step action controls, and encoding of novel input images.
- Transformation-aware generation: Transforming autoencoders use capsules to track visual-feature presence and position while applying affine transformations and 3D rotations.The present work instead uses convolutional architecture with action inputs and recurrence for repeated rotations.
- Chair generation: Prior chair-generation work maps shape, pose, and transformation labels directly to images and controls these variation factors.The paper adds an encoder for novel-image generalization, distributed appearance and pose representations, and recurrent long-term prediction.
- Inverse graphics: Inverse Graphics Networks combine image encoding with decoder-based graphics codes, whereas this model learns trajectories of multiple transformations with per-step controls.The paper also uses deterministic feed-forward training rather than a variational autoencoder framework.
- Disentangling: Disentangling methods such as bilinear models and disentangling Boltzmann Machines separate latent factors including style, content, identity, and pose.These approaches provide context for the paper’s learned separation of appearance-related factors.
3 Recurrent Convolutional Encoder-Decoder Network
The model separates invariant identity features from transformable pose features and updates pose recurrently using action inputs. Curriculum learning supports longer trajectories and encourages identity–pose disentangling without explicit regularization.
- Architecture: The encoder maps an input image into pose units and identity units, while a symmetric decoder reconstructs transformed views from them.Pose units receive transformations; identity units represent information intended to remain unchanged.
- Recurrent transformation: Action units provide local pose increments, but simply accumulating them can move pose representations off the nonlinear manifold during long rotations.The paper introduces recurrence to address this long-trajectory problem.
- Objective: The training objective decodes an image from identity features, action- and time-dependent pose features, and the target transformed image.The action sequence controls the pose transformation while identity features remain invariant across time steps.
- Training: The recurrent network is trained with backpropagation through time and ADAM, using curriculum learning that gradually increases trajectory length.The authors report that longer training sequences improve model performance and feature disentangling.
- Disentangling: Longer sequences encourage identity units to discard pose information because the same identity representation must support predictions across widely separated rotations.This produces invariant identity features and pose features that are not informative of identity without explicit disentangling regularization.
4 Experiments
Experiments evaluate recurrent view synthesis on faces and chairs, including novel-object rendering, curriculum length, reconstruction error, cross-view recognition, and latent-factor disentanglement. The model synthesizes long rotation trajectories and learns identity- and pose-related representations without class labels.
- Experimental setup: The experiments use Multi-PIE faces and rendered chair models to assess view synthesis, recognition, and interpolation across object instances.Multi-PIE supplies seven viewpoints from −45° to 45°, while the chair experiments use 809 selected CAD models rendered from 31 azimuth and two elevation angles.
- 3D view synthesis: RNN6 produces smooth Multi-PIE face rotations while preserving gender, expression, facial structure, and glasses across novel viewpoints.The model is evaluated on test identities not seen during training and is compared with a 3D morphable model for pose normalization.
- Curriculum training: Curriculum stages progressively improve long-horizon chair predictions: RNN1 degenerates after one step, while RNN4 and RNN8 generalize to longer trajectories.The stages train models for increasing sequence lengths and evaluate their ability to process more rotation steps.
- 3D view synthesis: 179 MSE is achieved by RNN16 versus 310 MSE for KNN5, corresponding to a 42% relative improvement.KNN5 is comparable to RNN4, whereas the longer-horizon RNN16 model significantly outperforms it.
- Cross-view recognition: The learned identity and pose units disentangle cross-view recognition: pose units reach 1.4% mean success when identity units achieve near-perfect Multi-PIE recognition.The 1.4% pose-unit rate is close to the 0.5% random-guess rate for 200 classes.
- Latent-factor disentanglement: Interpolating chair identity units yields smooth stylistic transformations while preserving each chair’s distinctive characteristics across rotated views.The interpolation experiment combines identity and pose representations before recurrent decoding.
5 Conclusion
The paper develops a recurrent convolutional encoder-decoder for synthesizing 3D views of unseen faces and chairs, with curriculum training improving appearance and recognition features. It also demonstrates disentanglement of chair identity and pose without explicit hidden-unit regularization.
- The recurrent convolutional encoder-decoder synthesizes 3D views of unseen object instances from single images.
- On Multi-PIE faces and 3D chair CAD models, the model predicts accurate renderings across trajectories of repeated rotations.
- Curriculum training with gradually increasing trajectory length improves image appearance and produces more discriminative pose-invariant recognition features.
- The model interpolates across chair identity while fixing pose and traverses pose while fixing identity.
- Identity–pose disentangling emerges from the recurrent rotation-prediction objective without explicit hidden-unit regularization.
Appendix
The appendix extends the evaluation to chairs and cars while documenting mask-stream training and 16-step car rotation synthesis. These experiments use convolutional encoder-decoder networks and curriculum training for increasingly long rotation sequences.
- Chair model: The chair model decodes binary masks alongside rotated chair images as an easier prediction objective that regularizes the network.The paper reports that the mask stream significantly improves prediction performance.
- Chair model: Learning curves compare RNN1 chair training with and without the mask stream.
- Cars: The car experiment uses the chair-model network structure with single-channel input and output layers and trains RNN1, RNN2, RNN4, RNN8, and RNN16 sequentially.Training uses curriculum stages and targets azimuth rotation.
- Cars: Figure 12 presents 16-step rotation synthesis examples for two car models, marking input images with red boxes.