PaperlayerView exact revision on arXiv

Source-linked AI summary

RenderGAN: Generating Realistic Labeled Data

Leon Sixt, Benjamin Wild, Tim Landgraf

arXiv:1611.01331v5cs.NEcs.CV

TL;DR

Manual labeling limits supervised DCNN training, especially for complex annotations. RenderGAN combines a 3D model with GAN-learned, label-preserving augmentations to generate realistic labeled images without manual labels. In the honeybee application, DCNN-based identity assignment reached 96% versus 55% for a previous computer vision pipeline.

  • Problem

    Manual annotation is time-consuming and can be prohibitively expensive, limiting supervised DCNN training despite the value of more labeled data.

  • Method

    RenderGAN embeds a 3D model in a GAN and learns differentiable augmentations from unlabeled data while preserving labels supplied by the model.

  • Results

    96% accuracy in assigning the true identity was obtained with DCNN detections, compared with 55% using the previous computer vision pipeline.

  • Takeaways & Limitations

    The framework generates realistic labeled samples without manual labeling, using the simple 3D model as supervision.

  • Takeaways & Limitations

    The augmentation functions must be customized for each application, and a suitable 3D model must be available.

Abstract

from arXiv · show

Deep Convolutional Neuronal Networks (DCNNs) are showing remarkable performance on many computer vision tasks. Due to their large parameter space, they require many labeled samples when trained in a supervised setting. The costs of annotating data manually can render the use of DCNNs infeasible. We present a novel framework called RenderGAN that can generate large amounts of realistic, labeled images by combining a 3D model and the Generative Adversarial Network framework. In our approach, image augmentations (e.g. lighting, background, and detail) are learned from unlabeled data such that the generated images are strikingly realistic while preserving the labels known from the 3D model. We apply the RenderGAN framework to generate images of barcode-like markers that are attached to honeybees. Training a DCNN on data generated by the RenderGAN yields considerably better performance than training it on various baselines.

1 INTRODUCTION

RenderGAN addresses the high cost of manually labeling data for DCNNs by combining a 3D model with learned image augmentations. In the honeybee application, generated labeled images improved downstream identification accuracy.

  • Manual labeling is time-consuming and can be prohibitively expensive for complex annotations needed by supervised DCNN training.
  • The framework requires no manual labeling because the simple 3D model supplies the supervision for generated samples.
  • RenderGAN embeds a 3D model in a GAN and learns label-preserving blur, lighting, background, and image-detail augmentations from unlabeled data.
  • The honeybee application uses a 3D tag model to generate labeled images while learning lighting, background, and image details from unlabeled bee-tag images.
  • 96% accuracy in assigning true bee identities was achieved with DCNN detections, compared with 55% using the previous computer vision pipeline.

2 RELATED WORK

Prior approaches reduce labeling demands through augmentation, transfer learning, 3D-rendered data, or GANs, but each has limitations in labels, domain coverage, or controllability.

  • Data augmentation increases effective training data by applying label-preserving transformations such as translation, noise, and deformation.
  • Pre-training on datasets such as ImageNet helps transfer features, but requires a sufficiently related dataset and may still need costly task-specific labels.
  • 3D-CAD models and game engines can generate labeled images for viewpoint estimation, detection, and segmentation tasks.
  • Conventional GANs generate high-quality samples but do not provide a simple relationship between latent dimensions and labels.
  • Conditional GANs generate samples given labels, but their training requires labeled datasets.

3 RENDERGAN

RenderGAN reverses supervised prediction by generating images from labels through a differentiable 3D model followed by learned, label-preserving augmentations. GAN training tunes these transformations to produce realistic samples.

  • RenderGAN treats supervised prediction inversely by generating data from labels using a 3D model embedded in the GAN generator.
  • The augmentation function φ(x, d) modifies image x using parameters d while preserving the image’s labels.
  • The generator predicts 3D-model parameters and augmentation parameters, producing images that combine modeled labels with learned visual characteristics.
  • Augmentation functions must be differentiable with respect to both the image and their parameters so gradients can reach the generator.
  • Multiple augmentation functions are applied sequentially, with each generator output supplying parameters to one stage of the image transformation.
  • The 3D model must be differentiable for discriminator loss to backpropagate through it, and unsuitable augmentations may prevent convergence to the data distribution.

4 APPLICATION TO THE BEESBOOK PROJECT

In BeesBook, RenderGAN combines a parameterized 3D bee-marker model with learned differentiable augmentations to generate realistic training images while preserving marker labels. The generated samples model varied image conditions and support improved downstream bee tracking.

  • 3D model: The 3D marker model represents position, pitch, yaw, roll, and ID, producing marker images, background masks, and depth maps.
  • Learned augmentations: RenderGAN augments the model output with blur, lighting, background, and image detail learned from unlabeled tag images.The framework uses a fixed neural-network emulator of the 3D model so discriminator gradients can reach model parameters, while IDs are sampled uniformly.
  • Learned augmentations: Lighting changes pixel intensities using separate black- and white-region scaling plus a shift, while preserving the ordering needed to keep marker bits valid.
  • Learned augmentations: Background replacement is restricted to pixels identified by the 3D model’s segmentation mask, so the marker labels remain unchanged.
  • Learned augmentations: Detail augmentation adds bounded high-pass structure, with repeated filtering used to model spotlights without allowing arbitrary large modifications.The augmentation sequence is blur, lighting, background, then detail; discriminator feedback discourages unrealistic label-changing artifacts.
  • Generated data: The generator learns varied lighting, noise, and backgrounds, while unrealistic high-frequency artifacts receive low discriminator scores and can be discarded.

5 RESULTS

The evaluation tests whether RenderGAN-generated data improves DCNN performance over limited real data, handmade augmentations, and a prior computer vision pipeline. RenderGAN data performs strongly, and replacing the prior pipeline with DCNN detections raises tracking accuracy from 55% to 96%.

  • Evaluation setup: The evaluation compares RenderGAN data with real data, mixed data, handmade augmentations, and a prior computer vision pipeline.The study also asks whether learned augmentations are necessary compared with hand-designed alternatives.
  • Datasets: 5 million tags were generated with RenderGAN, and each training sample was used only twice without further augmentation.
  • Results: The RenderGAN-trained model achieved an MHD of 0.424, with a slight improvement when generated and real data were combined.The small performance gap after adding real data indicates the generated samples were effective.
  • Application outcome: 96% tracking accuracy was achieved with DCNN detections, compared with 55% using the previous computer vision pipeline.At this accuracy, the paper states that honeybee social behavior can be analyzed reliably.
  • Results: RenderGAN data performed considerably better than handmade augmentations, while the gap between HM 3D and HM LI highlights learned lighting augmentation.

6 DISCUSSION

RenderGAN extends GANs with a 3D model and learned augmentations, producing labeled images that can train DCNNs without manual labels. The approach generalizes to unseen real data and supports scalable relabeling, but requires domain-specific augmentation design and a suitable 3D model.

  • 6 DISCUSSION: RenderGAN learns complex image effects from unlabeled data while rendering samples from a basic 3D model.
  • 6 DISCUSSION: The generator provides explicit image information usable as supervised labels, while RenderGAN training requires no labels.
  • 6 DISCUSSION: A DCNN trained from scratch with only generated data generalized to unseen real data.
  • 6 DISCUSSION: Once adapted to a domain, RenderGAN can acquire arbitrary amounts of labeled data cheaply even when the data distribution changes.A tag-design change would require only small 3D-model and augmentation-parameter adaptations, unlike repeating manual annotation.
  • 6 DISCUSSION: RenderGAN requires carefully customized augmentation functions and a suitable 3D model to preserve high-level information.

7 FUTURE WORK

Future work proposes applying RenderGAN to other tasks with basic 3D models and developing additional augmentation functions for domains with complex annotations or no pre-trained models.

  • 7 FUTURE WORK: Future applications include human faces, pose estimation, and viewpoint prediction where basic 3D models exist.
  • 7 FUTURE WORK: Potential new augmentation functions include colorization, affine transformations, and diffeomorphism.
  • 7 FUTURE WORK: RenderGAN may be especially valuable where pre-trained models are unavailable or annotations are very complex.

A GENERATED IMAGES

The generated-image section examines realism, correspondence with the 3D model, and manually constructed augmentations. The augmentation pyramid controls image frequency while adding effects such as blur, lighting, background variation, noise, and spotlights.

  • Generated-image quality: The discriminator score orders generated samples from most realistic to least realistic, excluding images with artifacts from training.
  • Generated-image quality: Generated images are visualized for fixed bit configurations and compared with their corresponding 3D-model outputs.
  • Augmentations: Image pyramids combine weighted noise levels to generate random lighting, background, and noise while controlling frequency content.Each pyramid level contains pixels drawn from N(0, 1), and weighting levels controls the frequency domain.
  • Augmentations: Handmade augmentations include Gaussian blur, pyramid-based lighting and background, noise, and random Gaussian spotlights.
  • Augmentations: Slightly more unrealistic handmade images produced better DCNN performance than the selected manually tuned parameters.

C AUGMENTATIONS OF THE REAL DATA

Real-data augmentation combines random pixel-intensity changes, per-pixel noise, and geometric transformations; Table 3 lists the augmentation parameters.

  • Pixel intensities are randomly scaled and shifted using sI + t, where I is the image and s and t are scalars.
  • Per-pixel noise is sampled from a normal distribution, with its variance parameter drawn separately for each image.
  • Random rotation, scaling, translation, and shear transformations are applied to augment the real training data.
  • Table 3 specifies the parameters used for augmenting the real data.

D TRAINING SAMPLES

Figure 12 presents training samples from the different datasets.

  • Figure 12 compares training samples drawn from the different datasets.
Loading 1611.01331v5…