Source-linked AI summary
View Synthesis by Appearance Flow
Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, Alexei A. Efros
TL;DR
Novel view synthesis asks how to generate new views of an object or scene from input images, despite missing visual content and the difficulty of direct pixel generation. The paper predicts appearance flows that copy pixels from the input, extends the framework to multiple views, and reports higher perceptual quality than previous CNN-based techniques.
Problem
Novel view synthesis must generate images from arbitrary viewpoints, while geometry-based methods require difficult structure estimation and learning-based generators often produce blurry images.
Method
A convolutional neural network predicts dense appearance-flow coordinates that specify which input-view pixels to sample for reconstructing each target-view pixel.
Results
The framework synthesizes object-rotation and scene-egomotion views, supports multiple input views, and achieves higher perceptual quality than previous CNN-based techniques.
Takeaways & Limitations
Reusing input pixels while learning appearance correlations provides a practical alternative to direct pixel synthesis and supports preserving the input instance's identity and structure.
Takeaways & Limitations
The method is not close to solving novel view synthesis in the general case and cannot hallucinate pixel values absent from the input.
Abstract
from arXiv · showhide
We address the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints. We approach this as a learning task but, critically, instead of learning to synthesize pixels from scratch, we learn to copy them from the input image. Our approach exploits the observation that the visual appearance of different views of the same instance is highly correlated, and such correlation could be explicitly learned by training a convolutional neural network (CNN) to predict appearance flows -- 2-D coordinate vectors specifying which pixels in the input view could be used to reconstruct the target view. Furthermore, the proposed framework easily generalizes to multiple input views by learning how to optimally combine single-view predictions. We show that for both objects and scenes, our approach is able to synthesize novel views of higher perceptual quality than previous CNN-based techniques.
1 Introduction
Novel view synthesis seeks to generate an object or scene from a new viewpoint using one or more input images. The paper combines learned appearance correlations with input-pixel reuse to avoid explicit 3D estimation and direct pixel generation.
- Problem and motivation: Novel view synthesis generates an image of an object or scene from a desired viewpoint transformation given one or more input images.It is described as the computational equivalent of mental rotation and has applications in computer graphics and virtual reality.
- Prior approaches: Geometry-based methods estimate or approximate 3D structure but can produce holes where the source image lacks visual content.Texture hole-filling is used in some cases but is not always effective.
- Prior approaches: Learning-based methods address the problem's underconstraint by modeling object classes, but direct parametric image generation often produces blurry results.A side view cannot determine a specific car's front without learned information from other cars.
- Proposed approach: The proposed approach learns approximate geometry implicitly while reusing input pixels, combining advantages of geometric and learning-based methods.It also learns appearance correlations between object parts, including correlations useful for synthesizing a backside.
- Proposed approach: Instead of generating target-view RGB values, a convolutional encoder-decoder predicts appearance-flow vectors identifying corresponding pixels to copy from the input view.The formulation is intended to make learning more tractable and preserve the input instance's identity and structure.
- Results and scope: The framework synthesizes views under object rotation and scene egomotion, extends to multiple input views, and reports quantitative and perceptual improvements.The paper applies the method to both objects and scenes.
2 Related work
Related work spans feature learning, geometric view synthesis, image-based rendering, and texture synthesis. The paper positions appearance-flow prediction as a way to reuse image content while learning correlations that support unseen views.
- Feature learning by disentangling pose and identity: Prior feature-learning work treats novel-view synthesis as disentangling pose and identity or transforming representations under viewpoint changes.Examples include capsules, spatial transformation layers, ego-motion feature synthesis, and autoencoders with semantic units.
- Feature learning by disentangling pose and identity: The paper reports that predicting appearance flows instead of pixels leads to significant improvements.
- Geometric view synthesis: Geometric view synthesis explicitly models 3D structure, but correspondence-based methods can break down with only a few views from very different viewpoints.Multi-view stereo can produce impressive visual quality when sufficient correspondences are available.
- Image-based rendering: Image-based rendering reuses corresponding input rays and can generate high-quality views, but it cannot create content absent from the available views.The cited example is inability to generate a car's rear view from frontal and side images.
- Texture synthesis and epitomes: Texture synthesis combines samples from an input texture, while texture transfer adds a constraint to mimic a secondary source image.
- Texture synthesis and epitomes: Epitomes use a condensed image as a palette for sampling patches, a perspective analogous to using the original image as an epitome for novel-view generation.
3 Approach
The approach learns appearance correlations to reconstruct novel views by sampling pixels from one or more input views rather than generating target pixels from scratch. A multi-view extension predicts confidence masks to combine single-view predictions.
- Single-view synthesis: A CNN predicts a dense appearance flow field that maps each target pixel to an input-view sampling coordinate.The field is trained end-to-end for cross-view reconstruction, with bilinear interpolation supporting sub-pixel sampling.
- Single-view synthesis: Appearance-flow prediction preserves instance color identity, reduces perceptual blurriness, and makes the reconstruction process directly interpretable.The synthesized view uses only pixels from the same input instance, while the flow field visualizes how target pixels are constructed.
- Single-view synthesis: The single-view network encodes the input image and viewpoint transformation, then decodes their features into an appearance flow field for bilinear sampling.Its architecture comprises input-view and viewpoint encoders followed by a synthesis decoder.
- Multi-view synthesis: When one view lacks information for a target region, the multi-view network predicts per-pixel confidence masks and combines the resulting hypotheses using normalized weights.Each input view uses a shared-weight single-view CNN, and the masks estimate relative prediction quality at each pixel.
- Multi-view synthesis: The multi-view framework accepts an arbitrary number of input views at training and test time, unlike methods requiring a fixed number of views.The paper identifies this flexibility as particularly beneficial when the number of test-time input views varies.
4 Experiments
Experiments evaluate appearance-flow synthesis on objects and urban scenes against direct pixel generation, using both quantitative and perceptual comparisons. The method generally preserves details and improves perceptual quality, while showing weaker performance for views with little shared appearance and occasional scene distortions.
- Experimental setup: Experiments use ShapeNet objects and KITTI urban scenes, comparing against a CNN baseline that directly generates pixels.Object experiments include cars, chairs, and aeroplanes; ShapeNet shapes are split 80% for training and 20% for testing.
- Object synthesis: Appearance flows produce sharper car and chair views that better match ground truth than direct pixel generation.They preserve detailed textures and edge boundaries lost by the baseline despite identical training losses.
- Object synthesis: Appearance flows are strongest for target views within ±45° azimuth or along symmetry planes, but slightly weaker for views sharing little input appearance.Figure 5 visualizes mean pixel error by input and target view, including common failure modes.
- Object synthesis: 95% of car trials and 93% of chair trials favor appearance flows over the baseline in perceptual comparisons.The study sampled 1,000 test tuples and averaged responses from five unique turkers per tuple.
- Multi-view synthesis: Multi-view synthesis combines independent predictions with normalized confidence masks, typically producing results more similar to ground truth than any single prediction.Each input view supplies a candidate target view and a selection mask before weighted combination.
- Scene synthesis: On KITTI scenes, the method significantly outperforms the baseline in single- and multi-view settings while preserving object textures, edges, and scene details.Severe distortions can still occur in failure cases.
5 Discussion
The framework performs well on benchmark evaluations but remains limited in generality, with unresolved challenges in hallucination, long-range correlations, datasets, metrics, and category dependence. The appearance-flow technique may also extend beyond novel view synthesis.
- Open challenges: The method cannot hallucinate pixel values absent from the input view, motivating hybrid systems that combine pixel generation with flow-based detail preservation.The authors note that the input color palette often remains rich enough, but identify combining both capabilities as beneficial.
- Open challenges: The network sometimes struggles to learn long-range appearance correlations because flow-derived gradients are highly local.Preliminary multiscale reconstruction-loss experiments alleviate this gradient locality to some extent.
- Open challenges: Large-scale datasets of diverse real-world objects and scenes, along with a suitable progress metric, are still lacking.The authors specifically note that L1 pixel error is not an ideal metric for measuring research progress.
- Open challenges: Existing learning-based view-synthesis approaches assume the object's category is known, leaving category-agnostic synthesis as an open direction.The proposed future direction would apply a learned method to any real-world image.
- Broader applicability: Appearance flows may also apply to image inpainting, video frame prediction, action-effect modeling, and super-resolution.The authors present these as tasks beyond novel view synthesis where the technique could be useful.