PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images

Aron Yu, Kristen Grauman

arXiv:1612.06341v2cs.CV

TL;DR

Learning visual comparisons is limited by sparse pairwise supervision and incomplete coverage of subtle attribute differences. The paper uses an attribute-conditioned image generator to create slight, controlled attribute variations, augments real training pairs with them, and trains ranking models for novel real-image pairs. Results across faces and fashion show improved precise attribute prediction, while generated pairs are imperfect and fail about 15% of the time.

  • Problem

    Visual comparison learning faces sparse supervision because image-pair possibilities are quadratic and real images incompletely cover fine-grained attribute differences.

  • Method

    The method generates synthetic image pairs with slight controlled attribute changes and combines them with real pairs to train attribute ranking models.

  • Results

    Across face and fashion domains and two ranking models, augmenting real pairs with synthetic pairs improves precise attribute predictions.

  • Takeaways & Limitations

    Synthetic images can densify comparative supervision for learning to rank subtle visual attributes.

  • Takeaways & Limitations

    About 15% of generated pairs are invalid because the attribute change may be absent or indistinguishable, or the images may be unrealistic.

Abstract

from arXiv · show

Distinguishing subtle differences in attributes is valuable, yet learning to make visual comparisons remains non-trivial. Not only is the number of possible comparisons quadratic in the number of training images, but also access to images adequately spanning the space of fine-grained visual differences is limited. We propose to overcome the sparsity of supervision problem via synthetically generated images. Building on a state-of-the-art image generation engine, we sample pairs of training images exhibiting slight modifications of individual attributes. Augmenting real training image pairs with these examples, we then train attribute ranking models to predict the relative strength of an attribute in novel pairs of real images. Our results on datasets of faces and fashion images show the great promise of bootstrapping imperfect image generators to counteract sample sparsity for learning to rank.

INTRODUCTION

Visual comparison models must learn subtle attribute differences despite sparse pairwise labels and incomplete coverage of the attribute space. The paper proposes synthetic image pairs that densify supervision, and reports improved precise attribute predictions across faces and fashion.

  • Visual comparisons ask which of two images exhibits more of a specified attribute, often when the images differ only subtly.
  • Ranking supervision is sparse because possible image comparisons grow quadratically with the number of training images and require costly human judgments.Existing relative-attribute datasets sample less than 0.1% of all image pairs for ground-truth labels.
  • Training images also incompletely represent fine-grained attribute changes, leaving important regions of the attribute space undersampled.The paper identifies this curation problem separately from the lack of labeled pairs.
  • The method generates plausible synthetic image pairs with controlled attribute variations, optionally verifies their comparative labels, and combines them with real pairs to train rankers.The resulting model predicts attribute comparisons for novel pairs of real images.
  • Semantic jitter augments real training sets with high-level changes affecting image meaning rather than low-level geometric or photometric alterations.The paper illustrates this distinction with facial shape changes accompanying expression changes.
  • Across face and fashion domains and two ranking models, densified comparative supervision improves precise attribute predictions.

RELATED WORK

Prior work addresses attribute comparisons with ranking models and attribute-aware image features, while related synthetic-data methods mainly target rendered views or recognition tasks. This paper instead uses a learned 2D image synthesis engine to densify supervision for attribute ranking.

  • Attribute Comparisons: Relative-attribute systems use human-ordered image pairs to train ranking functions that predict attribute orderings in novel pairs.Approaches include linear and nonlinear rankers, feature-specific or local rankers, and comparison-tailored visual features.
  • Semantic Jitter: Standard image augmentation changes pixels through operations such as mirroring and scaling, whereas semantic jitter controls high-level semantic properties.
  • Attributes and Image Synthesis: Image-synthesis research has explored attribute-specific edits and generative models, including GANs and VAEs, primarily for face or scene domains.
  • Training Recognition Models with Synthetic Images: Synthetic images have also trained pose estimators and person detectors, commonly using graphics engines and 3D models to vary viewpoint or body pose.
  • Positioning of This Work: This work differs by using a learned 2D image synthesis engine for attribute ranking, offering variation beyond camera pose and lighting parameters.
  • Method Overview: The approach samples attribute-conditioned images, elicits dense synthetic pairs, and integrates them with real images to train attribute-comparison rankers.

Synthesizing Dense Supervision

The method seeks denser attribute-space coverage by generating images that differ subtly in one attribute while preserving other aspects. It uses Attribute2Image, a conditional generative model driven by attributes and latent factors.

  • Improving attribute-space coverage requires generating images with subtle differences in the target attribute while keeping other aspects constant.
  • The paper adopts Attribute2Image, an existing state-of-the-art system that generates images conditioned on specified attributes and latent factors.
  • Attribute2Image represents attribute strengths with y and latent factors with z, then generates realistic images x conditioned on both.
  • Its conditional variational auto-encoder uses multilayer perceptrons for entangled attribute and latent representations and a coarse-to-fine convolutional decoder for image pixels.

Generating Dense Synthetic Image Pairs

The paper generates synthetic image pairs by varying one attribute while holding other factors fixed, filling underrepresented regions of the attribute space and augmenting real comparison pairs.

  • Generation: Attribute2Image generates realistic images conditioned on attributes and latent factors to supply synthetic training images for visual comparison.The generator is used to fill underrepresented regions of image space.
  • Generation: The method samples novel identities, then modifies one attribute while keeping other variables constant to create closely matched image variants.The altered attribute is shifted by ±2σ_A, where σ_A is its standard deviation in real training images.
  • Generation: Because the generator models attribute co-occurrences, slight single-attribute changes yield plausible but previously unseen instances.The resulting images form smooth progressions and enable fine-grained pairs that are difficult to curate systematically from real images.
  • Pair construction: Synthetic pairs include intra-identity and inter-identity comparisons to increase coverage of the attribute space.Intra-identity pairs come from one identity’s spectrum, while inter-identity pairs combine different generated identities.
  • Quality control: About 15% of generated pairs are invalid because the attribute change is absent, indistinguishable, or visually unrealistic.The authors optionally use five crowdworkers per pair to correct erroneous order labels.
  • Quality control: Training with synthetic pairs alone performs much worse than augmenting real pairs, likely because real pairs help resist domain shift.The hybrid training strategy combines synthetic density with real-image characteristics.

Learning to Rank with Hybrid Comparisons

The approach augments two complementary ranking models with synthetic and real comparisons: local RankSVM selects relevant hybrid neighbors, while a high-capacity Siamese RankNet learns paired rankings.

  • Ranking models: The method is compatible with both local RankSVM and deep Siamese RankNet ranking models.This separates supervision densification from the choice of comparison model.
  • Local RankSVM: Local RankSVM trains a separate ranker for each test pair using its K nearest labeled image pairs.Relevance is determined by inter-pair image distance, so the selected neighborhood mixes real and synthetic comparisons.
  • Local RankSVM: Local learning benefits when training examples densely populate the space of subtle visual comparisons.The method therefore treats data density as important alongside the total quantity of supervision.
  • Deep RankNet: The deep ranker combines a paired-ranking CNN with a spatial transformer network that learns useful localized image patches.Its high capacity makes it especially data hungry and able to exploit denser supervision.
  • Supervision strategy: The synthetic pairs are intended to densify attribute space with plausible images in potentially undersampled regions rather than merely increase pair count.The same supervision strategy can support different ranking architectures.

A LEXICON OF FINE-GRAINED ATTRIBUTES

The paper constructs a fine-grained shoe-attribute lexicon through crowdsourcing and gathers dense pairwise supervision for each selected attribute.

  • Dataset construction: The authors address sparse comparison data and intuition-based attribute vocabularies by mining a fine-grained lexicon and collecting pairwise orderings.The study uses fashion images from UT-Zap50K.
  • Crowdsourcing: More than 1,000 workers proposed over 350 distinct words while viewing 4,000 shoe image pairs.Workers described subtle differences by completing a comparative sentence and providing rationales.
  • Lexicon: The final lexicon contains 10 attributes: comfort, casual, simple, sporty, colorful, durable, supportive, bold, sleek, and open.The authors post-processed suggestions using rationales and synonym merging.
  • Supervision: The study collects about 4,000 pairwise comparisons for each of the 10 attributes.This provides more than three times the comparison labels in the original dataset, although synthetic instances remain needed for best results.

EXPERIMENTS

The experiments evaluate dense synthetic supervision across shoe and face datasets using multiple ranking-model baselines and synthetic-data variants.

  • Datasets: Experiments cover UT-Zap50K shoes, synthetic shoe pairs, LFW-10 faces, and synthetic face pairs.The shoe data include four original attributes plus ten lexicon attributes; LFW-10 contains ten face attributes.
  • Implementation: Implementation uses downsized real images, a conditional image-generation system, mixed intra- and inter-identity pairs, and model-specific features or training settings.The synthetic images are 64 × 64 pixels for shoes and 35 × 35 pixels for faces.
  • Models and controls: The study compares RankSVM and DeepSTN with Classifier and Real+ baselines under a shared labeled-pair budget.Real+ augments real pairs with pseudo pairs bootstrapped from individual-image attribute-strength estimates.
  • Baselines: Real trains only on human-labeled real pairs, whereas Jitter adds low-level translation, scaling, rotation, contrast, and color transformations.Jittered pairs inherit the corresponding real pair’s label.
  • Synthetic supervision: DSynth replaces half of Real’s pairs with manually verified synthetic pairs, while DSynth-Auto adds automatically supervised synthetic pairs to all real pairs.DSynth-Auto derives noisy pairwise supervision from the absolute attribute strengths used to generate images.

Fashion Images of Shoes

On shoe comparisons, semantic synthetic pairs improve ranking across local and deep models, including fine-grained evaluations and comparisons with prior methods.

  • Setup: Fashion experiments use UT-Zap50K real pairs and UT-Zap50K-Synth synthetic pairs across ten attributes under domain shift.Real training and testing pairs come from the same dataset, while synthetic training images introduce a domain difference.
  • Label quality: 78% average agreement links automatically generated pair labels with human labels across attributes.This agreement is reported for the automatically supervised synthetic pairs.
  • Baselines: The Classifier baseline underperforms both rankers, indicating that the generator’s initial attribute-strength representation is insufficient.Real+ also provides no performance gain over Real because it does not resolve fine-grained comparison sparsity.
  • Results: Semantic jitter outperforms baselines for most RankSVM attributes and all DeepSTN attributes, while low-level jitter provides smaller or limited gains.About 85% of selected local neighbors are synthetic under the proposed method, compared with 55% for jittered pairs.
  • Interpretation: Overall gains occur without changing the underlying ranking models, image features, or experimental setup.This isolates dense synthetic supervision as the added training component.
  • Prior methods: The approach beats prior methods, including DeepSTN, on coarse and difficult fine-grained UT-Zap50K comparisons under matched 64 × 64 image conditions.The comparison keeps the experimental setup fixed except for adding dense synthetic pairs.

Human Faces

The face-domain experiment tests transfer from LFW training data and synthetic pairs to PFSmile test pairs, where comparisons vary smilingness within individuals.

  • Setup: Face experiments use LFW-10 for real training pairs, LFW-Synth for synthetic training pairs, and PFSmile for real testing pairs.The setup includes domain shift because the real training and testing images come from different datasets.
  • Results: The proposed method outperforms all baselines on the face comparisons.The result is reported for the PFSmile evaluation setting.
  • Results: DSynth-Auto improves over Real from 69.29% to 75.00% and from 81.52% to 84.36% on the reported evaluations.These correspond to relative gains of 8% and 3.5%, respectively.
  • Baseline comparison: Semantic jitter gives DeepSTN a 6-point accuracy boost over traditional low-level jitter.The classifier-posterior baseline underperforms the ranking models.

Synthetic Test Images

On human-labeled synthetic test pairs, the proposed method yields substantial gains, though this evaluation favors models trained on the same synthetic domain.

  • Evaluation: The synthetic test sets contain novel image pairs from 2,000 identities with human-verified ordering labels, evaluated using DeepSTN.RankSVM achieves similar results.
  • Results: The method achieves a 25-point absolute gain for the attribute sleek on these synthetic test images.The authors note that training and testing share the Attribute2Image synthetic domain in this evaluation.
  • Caveat: The authors qualify the gain because DSynth learns from the same synthetic domain, while Real must overcome that domain shift.They also report overcoming the reverse synthetic-to-real shift in the real-image results above.

CONCLUSION

The paper presents semantic jitter as a way to densify supervision for fine-grained attribute comparisons, while distinguishing sample density from simply increasing sample quantity.

  • CONCLUSION: Synthetic examples vary slightly in attributes and are mixed with real training data to provide denser supervision for fine-grained comparisons.The generated images target underrepresented fine-grained differences rather than merely increasing the number of images.
  • CONCLUSION: Simply gathering more real images does not provide the same fine-grained density because of the difficulty of curating appropriate comparisons.
  • CONCLUSION: Future work could reduce human effort by sending only the most questionable synthetic pairs for verification.

APPENDIX

The appendix describes lexical data collection for shoes, attribute-conditioned generators for shoes and faces, and qualitative synthetic pairs that preserve fine-grained structure such as facial identity.

  • Lexical study: The lexical study uses UT-Zap50K shoe images to identify words describing fine-grained visual differences.The study asks what words humans use when distinguishing subtle differences in similar shoe images.
  • Experimental Design: MTurk workers complete comparative sentences for individual images or groups, using category-subcategory pseudo-classes to organize questions.Design 1 compares two images, whereas Design 2 compares one image with a group of six.
  • Post-Processing: More than 1,000 MTurk workers produced over 350 distinct suggestions after post-processing merged spelling variants and visual synonyms or antonyms.
  • Generative model: The attribute-conditioned image generator is trained with a Conditional Variational Auto-Encoder using real-valued attribute strengths.
  • Fashion Images of Shoes: For shoes, 50 attributes combine metadata-derived pre-labels with 10 fine-grained relative attributes from the lexical study.
  • Human Faces: The face generator is trained on 11,154 LFW images using 73-dimensional attribute strengths.
  • Nearest Neighbors: Synthetic face pairs exhibit fine-grained differences while preserving identity, a combination valuable for learning and difficult to obtain from real pairs.
Loading 1612.06341v2…