Source-linked AI summary
Learning from Simulated and Unsupervised Images through Adversarial Training
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, Russ Webb
TL;DR
Synthetic images provide inexpensive annotations but suffer from a realism gap with real images. The paper proposes S+U learning and SimGAN, which refine simulator outputs using unlabeled real data while preserving annotations; experiments report improved realism and state-of-the-art results without labeled real data.
Problem
Synthetic images provide automatic annotations, but their distribution can differ from real images and limit generalization to real-image tasks.
Method
SimGAN refines synthetic images with an adversarial refiner trained alongside self-regularization and other stabilizing modifications, using unlabeled real data while preserving annotations.
Results
The framework significantly improves simulator realism and achieves state-of-the-art results without human annotation effort, including on gaze estimation.
Takeaways & Limitations
Refined synthetic images can support training deep neural networks with simulator annotations and no labeled real data.
Takeaways & Limitations
The method assumes that refined images should be generated without artifacts.
Abstract
from arXiv · showhide
With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulator's output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a 'self-regularization' term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.
1. Introduction
Synthetic images provide automatic annotations but can fail to generalize because they differ from real images. S+U learning uses unlabeled real data to improve simulator realism while preserving annotations, implemented through SimGAN and stabilized adversarial training.
- Motivation: Synthetic training data offers automatic annotations but may generalize poorly because its image distribution differs from real data.Improving the simulator itself can be computationally expensive and difficult to model comprehensively.
- S+U learning: S+U learning refines simulator outputs with unlabeled real data while preserving their annotation information.The framework is intended to produce realistic images without artifacts that could affect downstream models.
- SimGAN: SimGAN uses a refiner network trained with adversarial and self-regularization losses to make synthetic images more realistic without changing their annotations.The refiner operates convolutionally at the pixel level to preserve global image structure.
- SimGAN: The method modifies GAN training to stabilize optimization and prevent artifacts, including alternating updates of the refiner and discriminator.The introduction identifies artifact prevention and training stability as key design goals.
- Results: Qualitative, quantitative, and user-study experiments show improved simulator realism and state-of-the-art results without human annotation effort.The paper positions SimGAN as complementary to prior synthetic-data and domain-adaptation approaches.
2. S+U Learning with SimGAN
SimGAN refines synthetic images toward real-image appearance while preserving simulator annotations. It combines adversarial realism training with self-regularization, local patch discrimination, and historical refined-image updates to reduce artifacts and stabilize learning.
- S+U Learning with SimGAN: SimGAN refines simulator-generated images with a neural refiner trained adversarially so refined outputs become difficult to distinguish from real images.The discriminator classifies real versus refined images, while the refiner is optimized to fool it.
- Adversarial Loss with Self-Regularization: Self-regularization preserves simulator annotations by minimizing feature-space differences between synthetic and refined images.The loss uses an L1 difference, and the identity feature transform is used unless otherwise stated.
- S+U Learning with SimGAN: The refiner is fully convolutional without striding or pooling, modifying pixels while preserving global structure and annotations through alternating refiner and discriminator updates.The refiner and discriminator parameters are minimized alternately, with the other network held fixed during each update.
- Local Adversarial Loss: Local adversarial loss classifies image patches separately, limiting discriminator capacity and providing multiple training samples per image.A fully convolutional discriminator outputs a w×h probability map, and cross-entropy losses are summed over patches.
- Qualitative Results: Refined UnityEyes images qualitatively resemble real images more closely in skin texture and iris appearance than the original synthetic images.The example compares real MPIIGaze images with synthetic-image refinement results; no MPIIGaze labels are used during refiner training.
- Updating Discriminator using a History of Refined Images: A history buffer stabilizes adversarial training by training the discriminator on both current refined images and images generated by previous refiners.Each discriminator mini-batch samples half its images from the current refiner and half from the buffer.
3. Experiments
Experiments show that SimGAN makes synthetic gaze and hand-pose images more realistic while preserving annotations, improving downstream performance and human indistinguishability from real images.
- Appearance-based Gaze Estimation: SimGAN refines synthetic gaze images to capture real-image skin texture, sensor noise, and iris appearance while preserving gaze direction.
- Appearance-based Gaze Estimation: 51.7% human classification accuracy matched chance when distinguishing refined images from real images.The study used 10 subjects and 1,000 aggregate trials.
- Appearance-based Gaze Estimation: 22.3% absolute improvement was obtained when training the gaze estimator on SimGAN outputs rather than synthetic data.The comparison was evaluated at distance d = 7 degrees from ground truth.
- Appearance-based Gaze Estimation: 21% relative improvement over the state of the art was achieved on MPIIGaze by training on refined images.The reported improvement corresponds to a 2.1 degree reduction in mean eye-gaze estimation error.
- Hand Pose Estimation from Depth Images: SimGAN learned depth-discontinuity noise in NYU hand images without label information, and refined synthetic training data outperformed supervised real-image training by 8.8%.
- Ablation Study: Using a history of refined images reduced gaze estimation error from 12.2 degrees to 7.8 degrees and prevented severe artifacts.
- Ablation Study: A local adversarial loss removed artifacts and produced significantly more realistic images than a global adversarial loss.The global loss classifies the whole image, whereas the local loss evaluates spatial regions.
4. Conclusions and Future Work
The paper proposes Simulated+Unsupervised learning and SimGAN to add realism to simulator outputs while preserving synthetic annotations. It reports state-of-the-art results without labeled real data and identifies future extensions to multiple refinements and video.
- Simulated+Unsupervised learning adds realism to simulator outputs while preserving the annotations of synthetic images.
- SimGAN uses an adversarial network to refine synthetic images for S+U learning and achieves state-of-the-art results without labeled real data.
- Future Work: Future work will model noise distributions for multiple refined images per synthetic input and extend refinement from single images to videos.
Additional Experiments
Additional experiments evaluate SimGAN on gaze estimation and hand pose estimation using synthetic and real datasets. The qualitative results show improved realism while preserving task annotations, and training progressively produces more realistic hand-depth images.
- Appearance-based Gaze Estimation: The gaze experiment uses 1.2M synthetic UnityEyes images and 214K real MPIIGaze images, with MPIIGaze captured under extreme illumination conditions.
- Appearance-based Gaze Estimation: SimGAN improves synthetic eye images by capturing skin texture, sensor noise, and iris appearance while preserving gaze-direction annotations.
- Hand Pose Estimation: The NYU hand pose experiment uses 72,757 training frames and 8,251 testing frames with hand-pose labels used to create synthetic depth images.
- Hand Pose Estimation: Refined NYU hand depth images add realistic depth boundaries compared with the synthetic images and real dataset examples.
- Training Convergence: During training, self-regularization first produces smooth edges, adversarial training introduces artifacts, and later iterations generate realistic noise.