Source-linked AI summary
Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation
Benjamin Klein, Guy Lev, Gil Sadeh, Lior Wolf
TL;DR
Fisher Vectors conventionally use Gaussian mixtures, although heavy-tailed image-descriptor statistics motivate considering other distributions. This paper derives Laplacian and hybrid Gaussian-Laplacian mixture models and their Fisher Vectors; the authors report improved performance over existing state of the art and the mean-vector baseline, with hybrid-based variants strongest in their reported comparisons.
Problem
Fisher Vectors conventionally use Gaussian Mixture Models, while heavy-tailed descriptor statistics suggest that a Gaussian may not capture the empirical distribution well.
Source: §1 · p. 1Method
The paper derives Expectation-Maximization and Fisher Vector formulations for Laplacian and weighted-geometric-mean hybrid Gaussian-Laplacian mixture models.
Source: §1 · pp. 1–2Results
The Fisher Vector methods outperform existing state of the art and the mean-vector baseline; HGLMM-based vectors outperform GMM- and LMM-based vectors, and combining GMM with HGLMM performs best.
Source: §5 · p. 9, §6 · p. 10Takeaways & Limitations
The authors conclude that Fisher Vectors derived from LMM and HGLMM improve accuracy in central computer-vision tasks.
Source: §6 · p. 10The most suitable normalization techniques for Fisher Vectors derived from LMM and HGLMM remain unexplored.
Source: §6 · p. 10
Abstract
from arXiv · showhide
In the traditional object recognition pipeline, descriptors are densely sampled over an image, pooled into a high dimensional non-linear representation and then passed to a classifier. In recent years, Fisher Vectors have proven empirically to be the leading representation for a large variety of applications. The Fisher Vector is typically taken as the gradients of the log-likelihood of descriptors, with respect to the parameters of a Gaussian Mixture Model (GMM). Motivated by the assumption that different distributions should be applied for different datasets, we present two other Mixture Models and derive their Expectation-Maximization and Fisher Vector expressions. The first is a Laplacian Mixture Model (LMM), which is based on the Laplacian distribution. The second Mixture Model presented is a Hybrid Gaussian-Laplacian Mixture Model (HGLMM) which is based on a weighted geometric mean of the Gaussian and Laplacian distribution. An interesting property of the Expectation-Maximization algorithm for the latter is that in the maximization step, each dimension in each component is chosen to be either a Gaussian or a Laplacian. Finally, by using the new Fisher Vectors derived from HGLMMs, we achieve state-of-the-art results for both the image annotation and the image search by a sentence tasks.
1. Introduction
The paper focuses on pooling local image descriptors, where Fisher Vectors represent them through gradients of log-likelihood with respect to a fitted Gaussian Mixture Model. It introduces Laplacian and hybrid Gaussian-Laplacian alternatives, derives their fitting and Fisher Vector formulations, and evaluates the new variants for image-text matching.
1. Introduction
- Fisher Vectors pool local descriptors as gradients of their log-likelihood with respect to parameters of an unsupervisedly fitted Gaussian Mixture Model.The resulting representation is a concatenation of those gradients.Source: §1 · p. 1
- Heavy-tailed descriptor statistics suggest Gaussian distributions may fit poorly, motivating Fisher Vector variants based on the Laplacian distribution.Prior SIFT feature-matching work reported improvement with the Laplacian distribution and L1 distance instead of Euclidean distance.Source: §1 · p. 1
- The paper defines a diagonal-covariance Laplacian Mixture Model and derives its Expectation-Maximization and Fisher Vector expressions.It also approximates the diagonal Fisher Information Matrix to normalize the Fisher Vector dimensions.Source: §1 · pp. 1–2
- The Hybrid Gaussian-Laplacian Mixture Model uses a weighted geometric mean of Gaussian and Laplacian distributions, with each component dimension selected as one distribution or the other during maximization.The paper derives its EM and Fisher Vector expressions and approximates the diagonal Fisher Information Matrix.Source: §1 · p. 2
- Experiments represent images with Overfeat or VGG vectors and pool word2vec text vectors using GMM-, LMM-, or HGLMM-based Fisher Vectors before Canonical Correlations Analysis matching.The paper describes this combination as potent.Source: §1 · p. 2
2. Previous work
Previous work develops Fisher Vector improvements, image-text matching and description-generation systems, and methods for representing words and sentences. This paper's distinction is to derive Fisher Vectors from distributions beyond the conventional Gaussian mixture.
Previous work
- Earlier Fisher Vector work improved normalization and combined Fisher methods with deep learning, while the present work derives Fisher Vectors from LMM and HGLMM distributions.The paper states that these variants can surpass conventional Fisher Vector performance on some tasks.Source: §2 · p. 2
Image Annotation and Image Search
- Image-sentence retrieval approaches either map images and sentences into a common space or align image and sentence fragments, while this method pools word vectors and maps them with image vectors using CCA.The paper reports significant improvement over state of the art on Pascal1K, Flickr8K, and Flickr30K.Source: §2 · pp. 2–3
- Image-description generation work includes region-alignment and recurrent models, encoder-decoder pipelines, and CNN-to-RNN approaches.Examples include models by Karpathy and Fei Fei [15], Kiros et al. [18], and Vinyals et al. [50].Source: §2 · p. 3
Representing text as vectors
- The paper uses word2vec vectors for words and notes that its experiments employ the Skip-gram architecture.The related work describes word2vec as a log-linear network and contrasts Skip-gram with CBOW.Source: §2 · p. 3
3. Laplacian Mixture Model (LMM)
The LMM models data as a weighted sum of multivariate Laplacians under a diagonal covariance assumption. The section derives its EM fitting procedure and Fisher Vector, then motivates ICA as a descriptor transformation for that assumption.
3. Laplacian Mixture Model (LMM)
- The LMM is a weighted sum of multivariate Laplacians, each assumed to have a diagonal covariance matrix.The paper notes that the multivariate Laplacian is not uniquely defined.Source: §3 · p. 4
3.1. Fitting a LMM
- LMM parameters are estimated with EM, whose location parameter is a weighted median rather than the explicitly given GMM mean.The proposed efficient solution sorts coordinate values and selects a value minimizing the gap between the two sides of the defining equation.Source: §3.1 · p. 4
3.2. Fisher Vector of a LMM
- The LMM Fisher Vector is derived from gradients of the data log-likelihood with respect to LMM parameters, with the diagonal Fisher Information Matrix approximated for dimension normalization.The section identifies terms corresponding to location and scale parameter gradients.Source: §3.2 · pp. 4–5
3.3. LMM and ICA
- ICA is chosen to transform descriptors toward dimension-independent non-Gaussian signals that better fit the diagonal covariance assumption.For the reported image-text matching experiments, the authors say ICA also seems preferable when using GMM.Source: §3.3 · p. 5
4. Hybrid Gaussian-Laplacian Mixture Model
The HGLMM combines Gaussian and Laplacian distributions through a weighted geometric mean. Its EM fitting selects one distribution per dimension and component, while its Fisher Vector retains the GMM’s length.
4. Hybrid Gaussian-Laplacian Mixture Model
- The HGLMM is built from a weighted geometric mean of Gaussian and Laplacian distributions, with dimensions assumed independent for the multivariate model.Source: §4 · p. 5
4.1. Fitting a HGLMM
- EM fitting yields a Gaussian or Laplacian distribution for each dimension in each component, and the resulting hybrid mixture is normalized.Source: §4.1 · pp. 5–6
4.2. Fisher Vector of a HGLMM
- The HGLMM Fisher Vector has length 2KD, matching the GMM Fisher Vector despite the HGLMM’s additional parameters.Its coordinates are computed from log-likelihood gradients; the diagonal Fisher Information Matrix is approximated for normalization as in the GMM and LMM.Source: §4.2 · p. 6
5. Results
Across the reported benchmarks, HGLMM Fisher Vectors and their fusion with GMM outperform the compared methods. The paper also applies CCA-aligned image and word representations in an RNN for sentence synthesis.
5. Results
- Experiments cover Pascal1K, Flickr8K, Flickr30K, and COCO, containing 1,000, 8,000, 30,000, and 123,000 images, respectively.The experiments use dataset-specific training, validation, and test splits.Source: §5 · p. 6
- Evaluation covers image annotation, image search, and sentence similarity; annotation and search report recall@1/5/10 and mean or median first-ground-truth rank, while similarity reports mean rank.Source: §5 · p. 9
- HGLMM Fisher Vectors outperform GMM- and LMM-based variants, and combining GMM with HGLMM outperforms the other methods; Fisher Vector methods beat existing state of the art and the mean-vector baseline.The accompanying tables cover Flickr8K, Flickr30K, Pascal1K, and COCO.Source: Table 1 · p. 7, Table 2 · p. 7, Table 3 · p. 8
5.1. Sentence Synthesis
- For sentence synthesis, an LSTM RNN starts from a CNN image vector projected into CCA space, then feeds each predicted word through word2vec, HGLMM Fisher Vector, and sentence CCA representations.Images and words share the CCA space, so training does not require an extra layer to project words into image space.Source: §5.1 · pp. 9–10, Figure 2 · p. 10
- The sentence-synthesis RNN was trained on Flickr8K for 300 epochs, using SGD with a 0.00001 learning rate and 0.5 momentum; validation results selected the stopping point.Its training set comprised 6,091 images, each described by five or fewer sentences.Source: §5.1 · p. 10
6. Discussion
The paper reports improved accuracy from Fisher Vectors derived from LMM and HGLMM, and presents HGLMM as a flexible way to combine Gaussian and Laplacian modeling. It leaves open which normalization techniques best suit these variants.
6. Discussion
- Fisher Vectors derived from LMM and HGLMM improve accuracy on central computer-vision tasks; the authors expect the improvements may extend to other tasks.Source: §6 · p. 10
- Which normalization techniques best suit LMM- and HGLMM-derived Fisher Vectors remains unexplored, although Perronnin et al.’s normalization improvements raised standard Fisher Vector performance.Source: §6 · p. 10
- HGLMM lets each dimension in each component use whichever of the Gaussian or Laplacian distributions is more suitable, and the geometric-mean approach can extend beyond two distributions.Source: §6 · p. 10