PaperlayerView exact revision on arXiv

Source-linked AI summary

Binary Generative Adversarial Networks for Image Retrieval

Jingkuan Song

arXiv:1708.04150v1cs.CV

TL;DR

Exact nearest-neighbor search is often intractable for large, high-dimensional image collections, while strong deep hashing methods typically require scarce, subjective labels. BGAN learns binary image representations unsupervised through binary-conditioned GAN generation, sign-activation optimization, and retrieval-oriented losses; it outperforms other hashing methods across the evaluated datasets.

  • Problem

    Exact nearest-neighbor search is often intractable for large, high-dimensional image collections, and successful deep hashing methods usually require scarce, subjective labels.

  • Method

    BGAN uses binary image representations as GAN inputs, with generator–discriminator training, neighborhood structure, a retrieval-oriented loss, and a sign-activation strategy for direct binary coding.

  • Results

    BGAN significantly outperforms other deep and non-deep hashing methods across CIFAR-10, NUS-WIDE, and Flickr; BGAN_s exceeds supervised baselines on CIFAR-10 by 15.8%, 13.9%, 12.8%, and 11.9% for 12, 24, 32, and 48-bit codes.

  • Takeaways & Limitations

    The paper demonstrates that unsupervised hashing can use a GAN framework while producing reliable binary codes without class labels.

Abstract

from arXiv · show

The most striking successes in image retrieval using deep hashing have mostly involved discriminative models, which require labels. In this paper, we use binary generative adversarial networks (BGAN) to embed images to binary codes in an unsupervised way. By restricting the input noise variable of generative adversarial networks (GAN) to be binary and conditioned on the features of each input image, BGAN can simultaneously learn a binary representation per image, and generate an image plausibly similar to the original one. In the proposed framework, we address two main problems: 1) how to directly generate binary codes without relaxation? 2) how to equip the binary representation with the ability of accurate image retrieval? We resolve these problems by proposing new sign-activation strategy and a loss function steering the learning process, which consists of new models for adversarial loss, a content loss, and a neighborhood structure loss. Experimental results on standard datasets (CIFAR-10, NUSWIDE, and Flickr) demonstrate that our BGAN significantly outperforms existing hashing methods by up to 107\% in terms of~mAP (See Table tab.res.map.comp) Our anonymous code is available at: https://github.com/htconquer/BGAN.

1 INTRODUCTION

Large-scale image retrieval needs efficient approximate search, but deep hashing has largely depended on scarce labels and faces optimization difficulties when producing binary codes directly. BGAN addresses these challenges with an unsupervised GAN-based hashing framework designed for retrieval.

  • Motivation: Approximate nearest-neighbor search offers practical speedups over exact search for large, high-dimensional image collections.Exact nearest-neighbor search is often intractable because of dataset size and image dimensionality.
  • Motivation: Learning-based hashing supports scalable image retrieval through compact binary representations and efficient Hamming-distance computation.
  • Research gap: Deep hashing methods have achieved strong retrieval performance, but the most successful approaches typically require scarce and subjective labels.Unsupervised methods generally produce unsatisfactory performance relative to supervised deep hashing.
  • Proposed approach: BGAN restricts GAN noise to binary values and conditions it on each image’s features, jointly learning a binary representation and a plausibly similar generated image.The discriminator verifies generated images against training images, removing the need for supervision.
  • Proposed approach: BGAN uses a retrieval-oriented loss and a sign-activation strategy to learn accurate binary codes directly without relaxation.The loss is designed beyond image generation, while the optimization strategy is equivalent to the non-smooth sign function.

2 RELATED WORK

Prior hashing work separates unsupervised structure-preserving methods from usually stronger supervised label-based methods, while deep models and GAN variants leave binary representation and direct optimization challenges unresolved. BGAN targets these gaps by combining unsupervised deep hashing with binary GAN representations.

  • Hashing methods: Unsupervised hashing preserves properties such as pairwise or multi-wise similarity without requiring labels.
  • Hashing methods: Supervised hashing uses labels and usually achieves better performance, but label information is typically scarce.
  • Open challenges: The paper seeks to let unsupervised hashing benefit from deep learning while generating reliable codes without class labels.
  • Open challenges: Non-smooth sign activation makes standard back-propagation infeasible, while prior alternatives relax binary constraints before thresholding.The paper proposes a sign-activation strategy as an alternative to these approaches.
  • Generative adversarial networks: Earlier GAN representation-learning methods use non-binary representations, leaving binary image representations in GAN frameworks as an unaddressed challenge.

3 BINARY GENERATIVE ADVERSARIAL NETWORKS

BGAN learns compact binary codes and reconstructed images without labels by coupling an encoder, hashing layer, generator, and discriminator. Its training uses image generation and verification to steer retrieval-oriented hash learning.

  • Objective: The unsupervised objective learns binary codes and reconstructed images so codes preserve image content and are computed directly without relaxation.
  • Framework: BGAN trains an encoder, generator, and discriminator after constructing image neighborhoods, coupling generated-code reconstruction with discriminator-based verification.The architecture contains encoder, hashing, generator, and discriminator components.

3.1 Construction of Neighborhood Structure

BGAN constructs an image-neighborhood structure before training by combining direct K-nearest-neighbor relations with graph-inspired indirect similarities. The resulting matrix supplies unsupervised structural information for hash learning.

  • Initial neighborhood: The method extracts 2,048-dimensional pool5 features and uses K-nearest neighbors to construct an initial neighborhood matrix.Each feature vector corresponds to an image.
  • Initial neighborhood: The neighborhood representation is organized as a similarity matrix S containing pairwise image relations.
  • Framework context: Figure 1 places neighborhood construction before the encoder, hashing layer, decoder, and discriminator components of BGAN.
  • Neighborhood evaluation: Figure 2 evaluates the precision of constructed CIFAR-10 labels across different K values and methods.
  • Initial neighborhood: The initial neighborhood uses direct K1-nearest-neighbor relations, represented through matrix entries for neighboring images.
  • Neighborhood expansion: Increasing K1 rapidly decreases neighborhood precision, so graph-based expansion is used instead of simply enlarging the direct neighborhood.
  • Neighborhood expansion: Graph expansion infers a potential connection between images when one image connects to a second and the second connects to a third.
  • Neighborhood expansion: The method compares columns of the initial similarity matrix to measure image similarity, ranks K2 neighbors, and forms a second similarity matrix.

3.2 Architecture Structure

BGAN combines an encoder, hashing layer, generator, and discriminator to learn binary image codes without supervision. Its hashing strategy addresses direct sign-based binarization through continuation-style approximations, while adversarial training makes generated images resemble originals.

  • Overall architecture: BGAN consists of four components: encoder, hashing, generator, and discriminator.The architecture learns hash codes through image generation and discriminator-based verification against original training images.
  • Hashing: The encoder produces a continuous representation that the hashing layer converts directly into binary codes with values +1 or −1.The binary code is obtained by applying the sign function to the representation from the final fully connected hash layer.
  • Hashing: The sign function is difficult to optimize because it is non-smooth and non-convex, with zero gradients away from zero and an undefined gradient at zero.These properties make standard back-propagation infeasible for training the deep network.
  • Hashing: Continuation methods approximate the sign function with app(.) and gradually reduce smoothing so optimization converges toward the original sign activation.The paper also describes tanh(.) as an alternative approximation illustrated alongside app(.).
  • Generator and discriminator: The generator reconstructs an image from the binary hash code, while the discriminator distinguishes original images from reconstructed images in an adversarial min-max game.The generator receives the hash code as input, and the discriminator’s competition encourages increasingly plausible reconstructions.

3.3 Loss Function

BGAN uses a weighted loss combining neighborhood structure, content, and adversarial objectives. Together, these terms preserve similarity relations, perceptual reconstruction quality, and discrimination between original and reconstructed images.

  • Loss design: The overall loss is a weighted sum of neighborhood structure, content, and adversarial losses.The loss definition steers the min-max optimization and evaluates perceptually relevant characteristics rather than relying only on mean squared error.
  • Neighborhood structure loss: The neighborhood structure loss measures how well binary codes preserve the reference similarity relations among images.It is defined using the binary codes and a similarity matrix, with optimization bringing codes of similar images closer together.
  • Content loss: The content loss evaluates reconstructed-image quality using perceptual feature similarity rather than only pixel-wise error.The VGG loss measures Euclidean distance between feature maps from the discriminator’s last convolutional layer for reconstructed and original images.
  • Adversarial loss: The adversarial loss models discriminator misclassification of original and reconstructed images.This term follows the discriminator-based adversarial formulation used in the reconstruction process.

3.4 Learning

BGAN encodes each input image into a binary code, reconstructs an image from that code, and evaluates the reconstruction with a discriminator while optimizing all loss terms. Training uses back-propagation and staged continuation, increasing the approximation parameter until the sign activation yields exact binary codes.

  • Forward propagation: The encoder extracts features from each input image and the hash layer embeds those real-valued features into binary codes.The resulting binary code is then passed to the generator to reconstruct an image.
  • Forward propagation: The generator reconstructs an image from the binary code, and the discriminator assigns probabilities to original and generated images.The discriminator distinguishes actual training samples from images generated by the model.
  • Optimization: The network learns parameters for the encoder, generator, discriminator, and hash representation using back-propagation and stochastic gradient descent.Each iteration computes neighborhood, content, and adversarial losses before updating parameters on mini-batches.
  • Continuation training: Training starts with β_t = 1 and increases β_t after convergence at each stage, using the previous parameters to initialize the next stage.As β_t approaches infinity, app(βz) converges toward the sign activation.
  • Continuation training: β_t = 10 already achieves fast convergence for BGAN training.

4 EXPERIMENTS

The experiments evaluate BGAN for large-scale image retrieval using component ablations and comparisons with unsupervised and supervised hashing methods. Results use public image datasets, Hamming ranking, and mAP and precision metrics.

  • Datasets and evaluation: Experiments evaluate BGAN on large-scale image retrieval using CIFAR-10, NUS-WIDE, and Flickr.CIFAR-10, NUS-WIDE, and Flickr provide the three public benchmark datasets used for empirical evaluation.
  • Datasets and evaluation: Hamming ranking is evaluated with mean average precision and precision, including precision-recall curves and precision@K.Average precision averages precision over top-k results for each query before averaging across queries.
  • Baselines: The study compares BGAN with four non-deep and two deep hashing methods, plus additional methods evaluated using deep features or precision.The comparisons include ITQ, SH, LSH, spherical hashing, DeepBit, DH, PCAH, DGH, AGH, and UN-BDNH.
  • Component analysis: The ablation study examines neighborhood structure, content, and adversarial losses and reports results on CIFAR-10 in Table 2.The loss components are tested individually and in combinations to study their effects on performance.
  • Component analysis: Removing neighborhood structure loss produces identical hash codes for all images, while the full three-component loss achieves the best performance.The full combination is ℓN + ℓC + ℓA; compared with ℓN alone, reported improvements are 2.5%, 2%, and 1.3% for 24-, 32-, and 48-bit codes.
  • Overall comparison: BGAN significantly outperforms other deep and non-deep hashing methods across the evaluated datasets, with comparisons reported in Table 3.The study also investigates whether direct binary codes improve over relaxed resolution and examines efficiency.

4.3 Effect of Binary Optimization (RQ2)

BGAN’s direct binary optimization improves hash-code performance over a two-step solution, while its broader evaluation shows strong retrieval results across datasets and code lengths.

  • Binary optimization: 2.5%, 3.2%, and 2.3% improvements are achieved over the two-step solution for 24-, 32-, and 48-bit hash codes, respectively.The paper attributes this to direct binary optimization rather than continuous optimization followed by sign-function postprocessing.
  • Unsupervised comparison: BGAN significantly outperforms other unsupervised hashing methods across CIFAR-10, NUS-WIDE, and Flickr.The comparison uses mAP in Table 3 and precision in Figure 4.
  • Supervised comparison: 15.8%, 13.9%, 12.8%, and 11.9% mAP improvements over supervised hashing methods are reported for 12-, 24-, 32-, and 48-bit codes.The comparison is conducted on CIFAR-10 using BGAN_s.
  • Component interpretation: The reported supervised comparison indicates that BGAN’s improvement is not only due to the constructed neighborhood structure.The paper links the remaining improvement to the other components of the framework.

4.5 The Study of Efficiency (RQ4)

The efficiency study reports rapid convergence and fast test-time hash-code generation, although BGAN takes longer to train and test than DH because it has more parameters. Reconstruction results are qualitatively similar to the originals.

  • Convergence: BGAN converges after a few epochs on CIFAR-10.The paper presents this as evidence of the efficiency of its solution.
  • Runtime: BGAN takes longer for training and testing than DH because it has more parameters.The comparison is reported for CIFAR-10 in Table 6.
  • Runtime: BGAN remains very fast at generating hash codes for a test image.This conclusion is stated despite the longer training and testing time relative to DH.
  • Reconstruction: BGAN reconstructs CIFAR-10 images that are similar to the original images.Figure 6 compares random-input reconstructions, BGAN reconstructions, and original images.

5 CONCLUSION

The paper concludes that BGAN addresses direct binary-code learning and retrieval-oriented representation learning, with strong benchmark results but reconstruction accuracy remaining an area for improvement.

  • Contributions: BGAN addresses direct binary-code generation without relaxation and accurate image retrieval beyond vivid image generation.These are identified as two central problems in image hashing.
  • Contributions: The method combines smoothed sign-activation realizations with adversarial, content, and neighborhood structure losses.The learning strategy converges to the results of sign activation.
  • Results: BGAN doubles the performance of the state of the arts on CIFAR-10 and significantly outperforms other methods on NUS-WIDE and Flickr.These are the paper’s reported experimental conclusions.
  • Future work: The paper identifies improving BGAN’s reconstruction accuracy as future work.This is the stated scope boundary for further development.
Loading 1708.04150v1…