Source-linked AI summary
Multi-View Image Generation from a Single-View
Bo Zhao, Xiao Wu, Zhi-Qi Cheng, Hao Liu, Zequn Jie, Jiashi Feng
TL;DR
The paper addresses generating multi-view clothing images from a single-view input, especially for deformable objects with rich details. It proposes VariGANs, which combines variational inference with adversarial training in a coarse-to-fine process, and reports more plausible novel views than state-of-the-art methods on MVC and DeepFashion. The paper also notes remaining blocky artifacts and that the model focuses on clothes rather than humans.
Problem
Generating multi-view clothing images from one view remains underexplored for deformable objects with rich details.
Method
VariGANs combines variational inference for coarse global appearance with adversarial learning for detailed high-resolution target views.
Results
On MVC and DeepFashion, VariGANs generates more plausible multi-view clothing images than state-of-the-art methods.
Takeaways & Limitations
The coarse-to-fine design produces novel views with more consistent global appearance and richer, sharper details than existing approaches.
Takeaways & Limitations
The model focuses on clothes rather than humans, and some generated examples still contain blocky artifacts.
Abstract
from arXiv · showhide
This paper addresses a challenging problem -- how to generate multi-view cloth images from only a single view input. To generate realistic-looking images with different views from the input, we propose a new image generation model termed VariGANs that combines the strengths of the variational inference and the Generative Adversarial Networks (GANs). Our proposed VariGANs model generates the target image in a coarse-to-fine manner instead of a single pass which suffers from severe artifacts. It first performs variational inference to model global appearance of the object (e.g., shape and color) and produce a coarse image with a different view. Conditioned on the generated low resolution images, it then proceeds to perform adversarial learning to fill details and generate images of consistent details with the input. Extensive experiments conducted on two clothing datasets, MVC and DeepFashion, have demonstrated that images of a novel view generated by our model are more plausible than those generated by existing approaches, in terms of more consistent global appearance as well as richer and sharper details.
1. Introduction
The paper targets single-view generation of multi-view clothing images, a costly and underexplored problem for deformable objects with rich details. VariGANs combines variational inference and adversarial learning in a coarse-to-fine architecture, with evaluations on two clothing datasets.
- Motivation: Multi-view clothing images are useful for product presentation but expensive to produce, motivating automatic generation from a single view.The paper also identifies photo/video editing and AR/VR as potential applications.
- Motivation: Existing multi-view methods mainly synthesize rigid objects such as cars and chairs, leaving deformable, richly detailed clothing insufficiently explored.The limitation is stated as an underexplored generation setting rather than a complete absence of prior work.
- Method: VariGANs combines variational inference and adversarial training to generate target views through complementary coarse and fine stages.Variational inference captures rough shapes and colors, while adversarial learning fills details and rectifies defects in high-resolution images.
- Method: The proposed architecture adopts a coarse-to-fine strategy intended to capture global appearance and draw richer details consistent with the input.The low-resolution stage models basic contours; the high-resolution stage focuses on details and defects.
- Evaluation: Experiments evaluate VariGANs on the MVC and DeepFashion clothing datasets against state-of-the-art methods.The paper reports comprehensive evaluations and states that the model demonstrates superiority over other state-of-the-art approaches.
2. Related Work
Related work covers VAE- and GAN-based image generation alongside view synthesis using 3D modeling, view-variant autoencoders, and projection-based frameworks.
- Image Generation: VAE-based approaches generate images from probabilistic graphical models and optimize a variational lower bound of data likelihood.The passage also mentions extensions using disentangled latent variables and recurrent attention over patches.
- Image Generation: GANs are presented as a distinct generative approach within the broader image-generation literature.The supplied passage introduces GANs as differing from generative parametric approaches.
- View Synthesizing: View-synthesis methods generate object views through 3D modeling, view-variant autoencoders, learned 3D structures, and projection transformations.The related work includes transforming auto-encoders and frameworks that project reconstructed 3D volumes into 2D observations.
3. Proposed Method
The method formulates single-view multi-view generation around a user-specified target view and uses VariGANs to generate images coarse-to-fine. Variational inference models global appearance, while adversarial refinement produces high-resolution details conditioned on the input view.
- Network Architecture: The architecture contains coarse and fine generators plus a conditional discriminator, with skip connections supporting information transfer in the fine generator.The discriminator receives generated-image/source-image pairs as negative examples and target-image/source-image pairs as positive examples.
- Problem Setup: The task generates an image at target view vj from a source image at view vi, with vj selected by the user.The model learns p(Ivj|Ivi, vj) from paired source-target images.
- Variational GANs: VariGANs addresses GANs’ difficulty capturing global appearance by decomposing generation into coarse variational inference and fine adversarial refinement.The coarse component models contours and global appearance; the fine component generates details from its output.
- Coarse Image Generation: The coarse generator V uses the source image, target view, and latent variable z to model the distribution of a low-resolution target-view image.Variational Bayesian optimization uses an auxiliary posterior distribution to approximate the true posterior and maximize a variational lower bound.
- Fine Image Generation: The fine generator G maps the coarse image to a high-resolution image conditioned on the source image and trains against a conditional discriminator.An additional ℓ1 loss keeps generated images near the ground truth and is chosen to reduce over-smoothing relative to ℓ2 loss.
4. Experiment
Experiments evaluate VariGANs on MVC and DeepFashion using SSIM, Inception Score, qualitative comparisons, and ablations. Results show stronger quantitative and visual performance, while examples illustrate coarse-to-fine generation and remaining artifacts.
- Datasets and comparisons: Experiments compare VariGANs with conditional VAE and conditional GANs on MVC and DeepFashion clothing datasets.MVC contains 36,323 clothing items, while DeepFashion contains 8,697; both provide multiple views.
- Quantitative results: SSIM and Inception Scores improve from cVAE to cGANs and further to VariGANs on both datasets.The authors interpret these higher scores as evidence of more realistic single-view-conditioned novel-view images.
- Qualitative results: Qualitative comparisons show cVAE produces blurry, incorrectly colored images, whereas cGANs add detail but exhibit severe artifacts.VariGANs produces more natural low-resolution shape and contour, then adds correct color and richer texture.
- Qualitative results: VariGANs generates side and back views from front images and front and back views from side images across both datasets.The examples include coarse images, fine images, and ground-truth images for comparison.
- Limitations: Some generated examples contain blocky artifacts, and the model focuses on clothing generation without modeling humans.The authors identify sharper-detail generation as future work while describing current novel-view results as sufficiently detailed for users.
- Model analysis: Feature-map visualizations indicate that the coarse generator learns view-angle transitions and contours for different image parts.The visualization covers encoder convolution layers and corresponding decoder deconvolution layers.
- Ablation study: Removing or replacing model components lowers SSIM and Inception Score, while variants show missing parts, incomplete areas, poor texture, or smearing.The ablations test variational inference, dual-path U-Net, ℓ1 loss, and the conditional discriminator.
5. Conclusion
VariGANs synthesizes realistic clothing views through variational inference and adversarial learning in a coarse-to-fine pipeline. Experiments found more plausible results than state-of-the-art methods, while ablations supported the importance of each component.
- VariGANs combines variational inference with adversarial learning to synthesize realistic clothing images from an input view.
- The coarse image generator first produces a target-view object shape, after which the fine image generator adds details and corrects defects.
- VariGANs generated more plausible results than state-of-the-art methods in extensive experiments.
- Ablation studies verified the importance of each component in VariGANs.