PaperlayerView exact revision on arXiv

Source-linked AI summary

FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery

Krishna Kumar Singh, Utkarsh Ojha, Yong Jae Lee

arXiv:1811.11155v2cs.CVcs.AIcs.GRcs.LG

TL;DR

Fine-grained discovery is difficult because prior unsupervised category clustering addresses entry-level categories, while fine-grained recognition relies on labeled categories. FineGAN hierarchically generates background, shape, and appearance, producing realistic, diverse images and learned features that form more accurate fine-grained clusters than JULE and DEPICT.

  • Problem

    Prior unsupervised object discovery clustered entry-level categories, while fine-grained recognition relied on ground-truth fine-grained annotations, which require expensive expert labeling.

    Source: §1 · p. 1

  • Method

    FineGAN is an unsupervised hierarchical GAN that generates fine-grained categories by disentangling background, object shape, and appearance.

    Source: §1 · p. 2

  • Results

    FineGAN generates realistic, diverse fine-grained images with favorable Inception Scores and FIDs versus baselines, and its features yield significantly more accurate clusters than JULE and DEPICT.

    Source: §1 · p. 2, §4.1.1 · p. 6, §4.2 · p. 8

  • Takeaways & Limitations

    FineGAN’s learned representation can be used to cluster real images for unsupervised fine-grained object category discovery.

    Source: §1 · p. 2

    Users must set the number of parent and child categories, which can be difficult when the true category count is unknown.

    Source: §5 · p. 8

Abstract

from arXiv · show

We propose FineGAN, a novel unsupervised GAN framework, which disentangles the background, object shape, and object appearance to hierarchically generate images of fine-grained object categories. To disentangle the factors without supervision, our key idea is to use information theory to associate each factor to a latent code, and to condition the relationships between the codes in a specific way to induce the desired hierarchy. Through extensive experiments, we show that FineGAN achieves the desired disentanglement to generate realistic and diverse images belonging to fine-grained classes of birds, dogs, and cars. Using FineGAN's automatically learned features, we also cluster real images as a first attempt at solving the novel problem of unsupervised fine-grained object category discovery. Our code/models/demo can be found at https://github.com/kkanshul/finegan

1. Introduction

Fine-grained discovery calls for grouping objects by both shared shape and finer appearance, but prior discovery and recognition work leaves unsupervised fine-grained clustering unaddressed. FineGAN hierarchically generates background, shape, and appearance, then uses its learned features to cluster real images.

1. Introduction

  • Fine-grained grouping must separate objects with shared backgrounds and shapes but different details, such as beak color and tail markings.The authors argue that this grouping is naturally hierarchical: first by shape, then by appearance.Source: §1 · p. 1
  • FineGAN proposes unsupervised fine-grained discovery because prior unsupervised discovery targets entry-level categories, while fine-grained recognition relies on annotated categories.Fine-grained annotations require domain experts, are expensive, and cannot be collected through standard crowdsourcing.Source: §1 · p. 1
  • The paper hypothesizes that a model able to generate fine-grained images hierarchically can also support fine-grained grouping of real images.The framework is presented as a way to discover fine-grained categories without manual fine-grained annotations.Source: §1 · pp. 1–2
  • FineGAN generates and stitches a background, parent image for one object factor, and child image for another, using mutual-information constraints to associate latent codes with generated content.Grouping multiple child codes under each parent is intended to make parent codes capture shape and child codes capture appearance.Source: §1 · p. 2
  • FineGAN’s learned features produce fine-grained clusters significantly more accurate than JULE and DEPICT, making this the paper’s first attempt at unsupervised fine-grained category discovery.The authors also report that FineGAN generates realistic and diverse images on CUB, Stanford-dogs, and Stanford-cars.Source: §1 · p. 2

2. Related work

Prior work addresses unsupervised discovery mainly at entry-level categories, while fine-grained recognition generally depends on annotations. FineGAN relates to unsupervised disentanglement and image generation, but extends them with hierarchical control over background, shape, and appearance.

2. Related work

  • Fine-grained recognition classifies subordinate categories and depends on expert annotations, whereas this work attempts unsupervised fine-grained category discovery.Some recognition methods also require part, attribute, or text annotations.Source: §2 · p. 2
  • Recent deep clustering methods perform well on high-variation categories, but FineGAN reports significantly better performance on fine-grained datasets by focusing on object details.Earlier discovery methods include handcrafted-feature clustering and natural-language dialogue.Source: §2 · p. 2
  • InfoGAN learns unsupervised disentangled representations by maximizing mutual information, while FineGAN extends this principle to hierarchically control background, shape, and appearance.The paper identifies explicit disentanglement and control as especially important for fine-grained categories.Source: §2 · p. 2
  • Unlike unconditional GANs [1], text-supervised FusedGAN, and LR-GAN, FineGAN offers unsupervised control over background, pose, shape, and appearance.FusedGAN cannot decouple pose and shape or explicitly control background; LR-GAN uses one random vector and does not separate shape from appearance.Source: §2 · pp. 2–3

3. Approach

FineGAN generates fine-grained images through background, parent, and child stages, using latent-code relationships and information-theoretic objectives to associate codes with image factors. Its learned features can also be used to cluster real images into fine-grained categories.

3.1. Hierarchical fine-grained disentanglement

  • The background, parent, and child stages generate a canvas, add object shape, then add appearance; their objectives are weighted and trained end-to-end.The background stage generates B, the parent stage stitches an object outline onto B, and the child stage fills the outline with color and texture.Source: §3.1 · p. 3
  • FineGAN uses fewer parent than child codes and ties multiple child codes to each parent, while matching background and child codes during training.The background-child constraint can be relaxed at test time to generate combinations such as a duck in a tree.Source: §3.1 · p. 3

3.1.1 Background stage

  • The background stage learns background images from detected non-object patches, using adversarial and auxiliary classification losses to keep foreground information out.Its discriminator evaluates patches, and the auxiliary classifier trains the generator to produce patches assigned high background probability.Source: §3.1.1 · p. 3

3.1.2 Parent stage

  • The parent stage maps noise and parent code to a foreground and mask, then stitches the masked foreground onto the background to form an object outline.The paper describes hierarchical features as potentially easier to interpret and useful for fine-grained generation and categorization.Source: §3.1.2 · p. 4
  • Because real intermediate images with shape but no appearance are unavailable, the parent discriminator maximizes information between p and masked foreground to associate p with shared shape-level attributes.The discriminator uses masked foreground rather than the full parent image so background does not influence its prediction; parent attributes are shared across fine-grained categories but do not capture all their details.Source: §3.1.2 · pp. 4–5

3.1.3 Child stage

  • Conditioned on parent features and child code, the child stage generates a foreground and mask that fills the parent outline with object color and texture.It stitches the masked child foreground over the portion of the parent image outside the child mask.Source: §3.1.3 · p. 5
  • The child-stage objective combines real-versus-fake discrimination with information-based recovery of child code c from masked foreground.The discriminator uses masked foreground so background does not influence the child-code prediction.Source: §3.1.3 · p. 5

3.2. Fine-grained object category discovery

  • For category discovery, FineGAN trains networks to predict parent and child codes on synthetic images, then concatenates their features for k-means clustering of real images.The parent-code predictor is intended to focus on shape, while the child-code predictor focuses on appearance.Source: §3.2 · p. 5

4. Experiments

Experiments evaluate image generation and real-image fine-grained clustering on CUB, Stanford Dogs, and Stanford Cars, using labels only for evaluation. FineGAN is assessed for generation quality, disentanglement, and the usefulness of its learned features for clustering.

4.1.1 Quantitative evaluation of image generation

  • FineGAN’s Inception Scores and FIDs compare favorably with baselines, indicating generated images closely match the real-data distribution.Source: Table 1 · p. 6, §4.1.1 · p. 6

How sensitive is FineGAN to the number of parents?

4.1.2 Qualitative evaluation of image generation

  • The background stage generates context, the parent stage produces an object-shaped hollow image, and the child stage adds appearance details without mask supervision.Source: §4.1.2 · pp. 6–7
  • Varying parent and child codes changes shape and appearance separately, while the random vector z controls viewpoint and pose; background codes can also vary independently.Source: §4.1.2 · pp. 7–8
  • InfoGAN’s complete-image code prediction can be biased by background; adding parent-code prediction still performs worse than FineGAN.Source: §4.1.2 · p. 8

4.2. Fine-grained object category discovery

  • FineGAN’s learned features outperform clustering baselines on all three datasets; using child-code features alone incurs a 0.017 drop in Accuracy on birds.Source: Table 3 · p. 8, §4.2 · p. 8

5. Discussion and limitations

FineGAN has several practical and interpretive limits: users must choose category counts, discovered groupings may differ from human labels, and background modeling currently uses bounding boxes.

5. Discussion and limitations

  • Users must set the numbers of parents and children, which can be difficult when the true number of categories is unknown.The model is robust to a wide range of parent counts, but both counts remain hyperparameters.Source: §5 · p. 8
  • Discovered latent modes may not match human-defined categories; for cars, children group by color rather than car model type.The paper notes this makes evaluation metrics for unsupervised methods important.Source: §5 · p. 8
  • The current implementation requires bounding boxes to model backgrounds, though preliminary experiments find border patches give reasonable results.Source: §5 · p. 8
  • FineGAN significantly outperforms unsupervised clustering methods but remains far behind fully supervised fine-grained recognition methods.Source: §5 · p. 8

Appendix

The appendix provides supplementary clustering results, mask-quality analyses, and implementation details.

Appendix

  • The appendix presents qualitative clustering results and additional clustering and image-generation results.Source: Appendix · p. 11
  • It reports additional quantitative analysis of the generated masks.Source: Appendix · p. 11
  • It provides architecture and training details.Source: Appendix · p. 11

1. Qualitative clustering results

FineGAN’s learned features form clusters that capture fine-grained appearance while varying in background, and separate bird classes with subtle differences. The cited qualitative comparison reports that JULE and DEPICT instead cluster mainly by background or shape and pose.

1. Qualitative clustering results

  • JULE and DEPICT clusters mainly follow background or object shape and pose, consistent with their worse quantitative performance in Table 3.The cited comparison describes JULE and DEPICT cluster rows; it provides no quantitative values.Source: §1 · p. 11
  • FineGAN forms separate clusters for bird classes with subtle differences, including Summer Tanager versus Vermilion Flycatcher and two woodpecker classes.The figure caption names the first pair “Summer Tanger” and “Vermillion Flycatcher.”Source: §1 · p. 11, Figure 9 · p. 12
  • FineGAN clusters show representative fine-grained shape and appearance while background variation indicates background is not a significant clustering cue.The learned features concatenate the penultimate-layer outputs of φp and φc.Source: §1 · p. 11, Figure 8 · p. 11

2. Quantitative evaluation of mask generation

FineGAN’s generated masks align substantially with CUB foreground annotations, and much of the parent mask remains within the child mask.

2. Quantitative evaluation of mask generation

  • 75.6% mIU: FineGAN’s predicted foreground masks on CUB achieve this score against ground-truth segmentation masks.A network trained on generated images predicts masks for real images across the entire CUB dataset.Source: §2 · p. 12
  • 0.70: the intersection of binarized parent and child masks, normalized by child-mask area, indicates most child-stage changes lie within the parent mask.The analysis also concludes that most of the background is retained at the child stage.Source: §2 · p. 12
  • The mask evaluation uses CUB because the dataset provides ground-truth foreground masks for birds.Source: §2 · p. 12

3. Attempting to model hierarchy with InfoGAN

InfoGAN conditioned on parent and child codes improves bird image generation only slightly and still underperforms FineGAN. Its child groups do not exhibit the intended shared attributes for a common parent.

3. Attempting to model hierarchy with InfoGAN

  • IS: 48.06 and FID: 12.84 for birds, yet this InfoGAN variant still performs worse than FineGAN.The variant conditions the generator on both parent and child codes and asks the discriminator to predict both.Source: §3 · p. 12
  • Simply adding a parent-code constraint to InfoGAN does not yield the hierarchical disentanglement achieved by FineGAN.Source: §3 · p. 12
  • With the same parent code, the four child groups show no noticeable common pattern in shape, color, or background.Figure 10 illustrates why adding a parent-code constraint alone does not produce FineGAN’s hierarchical disentanglement.Source: Figure 10 · p. 12, §3 · p. 12

4. Relaxing relationship constraints during inference

FineGAN’s training constraints keep generated background and child texture combinations compatible with the data. At inference, the constraints can be relaxed, allowing the learned factors to be recombined into unusual images.

4. Relaxing relationship constraints during inference

  • At inference, FineGAN can relax the training relationships because the generator no longer needs to fool the discriminator.Source: §4 · p. 12
  • Choosing a green, leafy background for a duck-like parent can produce images that the adversarial discriminator would classify as fake during training.CUB ducks mostly appear against water-like blue backgrounds; unusual combinations of blue, green, and red texture are also likely to be flagged.Source: §4 · p. 12
  • Because its factors are disentangled, FineGAN can compose images from arbitrary combinations, including inconsistent backgrounds and textures.The paper gives a red seagull as another example.Source: §4 · p. 12

5. Combining factors from multiple real images

FineGAN can combine background, shape, and appearance codes obtained from different real images to synthesize a new image. The authors suggest this could be useful for artists creating such combinations.

5. Combining factors from multiple real images

  • Codes for background, shape, and appearance can be computed from different real images and supplied together to FineGAN.Source: §5 · p. 13
  • In the illustrated composites, the generated image takes its background, shape, and appearance from the first, second, and third real-image columns, respectively.Source: §5 · p. 13
  • The authors suggest that factor recombination could be useful to artists generating images from multiple visual examples.Source: §5 · p. 13

6. Additional image generation visualizations

Supplementary visualizations show FineGAN’s stagewise generation and the control of shape, appearance, pose, and position across birds, dogs, and cars. The appendix also describes the model modules, training setup, and hard-negative training.

7.1. FineGAN architecture

  • The generators build background, parent, and child representations in stages, while the discriminators classify background patches, parent and child classes, and final-image realism.The generator modules use convolutional layers, upsampling, residual blocks, and code concatenation; the discriminator modules use convolutional layers and output class or real/fake scores.Source: §7.1 · p. 13
  • FineGAN can also synthesize an image combining the background, shape, and appearance of three separate real images.Source: Figure 12 · p. 13

7.2. Training details

  • Training uses Adam at 2 x 10^-4 for 600 epochs with batches of 16; hard-negative training then alternates normal and degenerate-image batches for 1–2 further epochs.The hard-negative batch comprises the 16 lowest-scoring images selected from 160 generated images, and this procedure alleviates degenerate images to a great extent.Source: §7.2 · pp. 13–14
  • Hard-negative training relies on class-conditional scores, so it cannot be applied to standard unconditional GANs; applying it to InfoGAN yielded no significant improvement.Source: §7.2 · p. 14

6. Additional image generation visualizations

  • For CUB, dogs, and cars, stagewise examples retain the background, generate a hollow object shape at the parent stage, and fill in appearance at the child stage.Source: Figure 13 · p. 15, Figure 14 · p. 15, Figure 17 · p. 18
  • Across birds, dogs, and cars, changing the parent code preserves shape while changing the child code changes appearance; the random vector controls pose and position.The visualizations show six parent groups with four children for birds and dogs, and four parent groups with three children for cars.Source: Figure 15 · p. 16, Figure 16 · p. 17, Figure 18 · p. 18
Loading 1811.11155v2…