Source-linked AI summary
Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks
Remi Denton, Sam Gross, Rob Fergus
TL;DR
The paper tackles semi-supervised image learning when labeled data are limited. It uses context-conditional adversarial in-painting to train a discriminator with unlabeled images, achieving comparable or superior performance to existing methods on STL-10 and PASCAL, including a 3.8% improvement over the prior state of the art on PASCAL.
Problem
Large deep-learning models depend on manually annotated datasets, but large collections of labels may not be readily available.
Method
CC-GANs train a generator to fill missing image patches from surrounding pixels and adversarially train a discriminator on real versus in-painted images, combined with classification.
Results
3.8%: CC-GAN outperforms the current state of the art on the PASCAL test set, while performance is comparable or better than existing semi-supervised methods on two benchmarks.
Takeaways & Limitations
The framework can directly train large VGG-style discriminators semi-supervised using adversarial in-painting as a regularizer.
Takeaways & Limitations
The model operates at 128×128 resolution, half the resolution used by other approaches, leaving potential gains from higher-resolution generators.
Abstract
from arXiv · showhide
We introduce a simple semi-supervised learning approach for images based on in-painting using an adversarial loss. Images with random patches removed are presented to a generator whose task is to fill in the hole, based on the surrounding pixels. The in-painted images are then presented to a discriminator network that judges if they are real (unaltered training images) or not. This task acts as a regularizer for standard supervised training of the discriminator. Using our approach we are able to directly train large VGG-style networks in a semi-supervised fashion. We evaluate on STL-10 and PASCAL datasets, where our approach obtains performance comparable or superior to existing methods.
1 INTRODUCTION
The paper addresses semi-supervised image learning when manually labeled data are scarce by using unlabeled images and context-based in-painting. Its approach trains a discriminator on real versus generated in-painted images, distinguishing it from prior context encoders and other representation-learning methods.
- Large labeled datasets drive deep-network performance, but labels may be unavailable in many settings.
- The proposed method trains a generator to fill missing image regions and uses the resulting images to adversarially train a discriminator alongside supervised learning.The discriminator distinguishes real training images from in-painted images, while the learned features support object classification.
- Learning From Context: Unlike context encoders, the approach uses discriminator features rather than encoder features for classification.The paper reports that this design enables direct adversarial training of large VGG models, whereas prior work reported difficulty training an AlexNet encoder with that loss.
- Deep unsupervised and semi-supervised learning: Related semi-supervised approaches include autoencoders, pseudo-labeling, CatGAN, semi-supervised GANs, graph methods, and context-based representation learning.The cited methods use mechanisms such as reconstruction, predicted labels, entropy objectives, generated classes, or spatial-context prediction.
- Predictive generative models of videos: Unlike video predictive models, this method performs interpolation within an image rather than extrapolation over time.
2 APPROACH
The approach combines GAN-based in-painting with supervised classification to use unlabeled images for learning representations. CC-GANs condition generation and discrimination on image context, while CC-GAN2 broadens discriminator training with additional generated examples.
- 2.2 CONTEXT-CONDITIONAL GENERATIVE ADVERSARIAL NETWORKS: CC-GANs train a generator to fill a missing image patch from surrounding pixels, then use the completed image in adversarial discrimination.The discriminator receives the completed image rather than separate context and patch inputs, preventing reliance on missing-patch edge discontinuities.
- 2.3 COMBINED GAN AND CC-GAN: CC-GAN2 presents both the in-painted image and the generator’s full output as fake discriminator examples, providing a wider array of negative examples.In standard CC-GAN, the discriminator sees only the portion corresponding to the missing hole from the generator output.
- 2.4 SEMI-SUPERVISED LEARNING WITH CC-GANS: The semi-supervised objective combines real-image discrimination, in-painted-image discrimination, and a classification loss on labeled images.The hyperparameter λc balances the classification and adversarial losses.
- 2.5 MODEL ARCHITECTURE AND TRAINING DETAILS: The framework uses a GAN generator inspired by DCGAN and a VGG-A′ discriminator without fully connected layers.The generator uses convolutional layers with subsampling followed by fractionally-strided convolutional layers.
- 2.5 MODEL ARCHITECTURE AND TRAINING DETAILS: For larger image patches, the generator can additionally receive a low-resolution whole image, making its task super-resolution on part of the image.The discriminator does not receive the low-resolution image and still determines whether the in-painted region is real or fake.
3 EXPERIMENTS
Experiments on STL-10 and PASCAL evaluate CC-GAN variants, supervised and GAN baselines, and different in-painting conditions. The method matches or exceeds comparison approaches while producing semantically plausible in-paintings.
- STL-10 classification: STL-10 contains 5,000 labeled and 100,000 unlabeled 96×96 color images across 10 classes, providing a 1:100 labeled-to-unlabeled ratio.
- STL-10 classification: CC-GAN performs comparably to the state of the art on STL-10, while CC-GAN2 improves upon it.Models use 64×64 crops, 32×32 holes, and randomly varying hole locations during training.
- STL-10 classification: The purely supervised VGG-A’ baseline performs significantly worse than CC-GAN on STL-10.
- STL-10 classification: SSL-GAN performs almost as well as CC-GAN on STL-10, supporting the GAN objective as an unsupervised criterion.
- PASCAL VOC classification: CC-GAN outperforms other feature-learning approaches on PASCAL VOC 2007, beating the reported state of the art by 3.8%.The best model uses 128×128 images, 64×64 holes, and a 32×32 low-resolution input.
- PASCAL VOC classification: CC-GAN outperforms the supervised VGG-A’ baseline by 7% on PASCAL VOC 2007, reaching 62.2% versus 55.2%.The authors suggest the gains stem from CC-GAN rather than the larger feature-extractor architecture.
- PASCAL VOC classification: CC-GAN improves over the baseline consistently across image resolutions, while low-resolution conditioning begins to help when the hole is 64×64.
- In-painting: Adversarial training produces sharp, semantically plausible in-paintings across masking schemes, although alternative masking schemes do not improve classification results.
4 DISCUSSION
The framework combines adversarial in-painting with semi-supervised learning, producing semantically meaningful fills and discriminator performance comparable to or better than existing methods. Its use of lower-resolution images also leaves potential for further gains through higher-resolution generation.
- The CC-GAN framework uses in-painting with an adversarial loss for semi-supervised learning.
- The generator produces semantically meaningful in-paintings, while the discriminator matches or exceeds existing semi-supervised methods on two classification benchmarks.
- The model operates at 128×128 resolution versus 224×244 for other approaches, leaving potential gains from improving generator resolution.
- The approach exceeds Pathak et al. (2016) on PASCAL classification because real/fake in-painting discrimination is more closely related to object classification.