PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Max Pooling

Naila Murray, Florent Perronnin

arXiv:1406.0312v1cs.CV

TL;DR

Frequent descriptors can dominate sum-pooled image representations even when rare descriptors carry more discriminative information, while max-pooling is limited to count-based encodings. GMP equalizes each patch’s similarity to the pooled representation, extending max-pooling to general encodings such as FV and EMK; across five public benchmarks, it can outperform heuristic alternatives.

  • Problem

    Sum-pooling overweights frequent, often uninformative descriptors, whereas max-pooling is not directly applicable to correlated higher-order encodings such as FV.

  • Method

    GMP equalizes the similarity between each patch encoding and the pooled representation, equivalently re-weighting per-patch encodings before sum-pooling.

  • Results

    Across five public benchmarks, GMP can provide significant performance gains over heuristic alternatives such as power normalization.

  • Takeaways & Limitations

    GMP provides a general pooling mechanism that equalizes frequent and rare descriptor influence and applies to encodings including EMK and FV.

Abstract

from arXiv · show

State-of-the-art patch-based image representations involve a pooling operation that aggregates statistics computed from local descriptors. Standard pooling operations include sum- and max-pooling. Sum-pooling lacks discriminability because the resulting representation is strongly influenced by frequent yet often uninformative descriptors, but only weakly influenced by rare yet potentially highly-informative ones. Max-pooling equalizes the influence of frequent and rare descriptors but is only applicable to representations that rely on count statistics, such as the bag-of-visual-words (BOV) and its soft- and sparse-coding extensions. We propose a novel pooling mechanism that achieves the same effect as max-pooling but is applicable beyond the BOV and especially to the state-of-the-art Fisher Vector -- hence the name Generalized Max Pooling (GMP). It involves equalizing the similarity between each patch and the pooled representation, which is shown to be equivalent to re-weighting the per-patch statistics. We show on five public image classification benchmarks that the proposed GMP can lead to significant performance gains with respect to heuristic alternatives.

1. Introduction

Patch-based image representations encode local descriptors and pool the resulting codes into fixed-length vectors. GMP addresses the imbalance caused by frequent descriptors while extending max-pooling beyond count-based encodings.

  • Representation framework: Patch-based image representations map local descriptors into a higher-dimensional space, then aggregate their codes with a pooling function.The framework includes BOV, Fisher Vector, VLAD, Super Vector, and EMK representations.
  • Motivation: Sum-pooling is general but lets frequent, potentially uninformative descriptors outweigh rare, potentially discriminative descriptors.This imbalance is especially problematic when background patches are frequent and class-specific details are localized.
  • Motivation: Existing remedies are heuristic or encoding-specific; max-pooling is not directly applicable to FV, VLAD, SV, or EMK because their dimensions are correlated and should be treated jointly.Max-pooling only makes sense for BOV and related count-based extensions.
  • Contribution: GMP equalizes the similarity between each patch and the pooled representation, generalizing max-pooling to arbitrary encodings including EMK and FV.The method is designed to give frequent and rare descriptors comparable influence.
  • Contribution: GMP has a primal matching formulation and an equivalent dual formulation that re-weights per-patch encodings using patch-to-patch similarities.Experiments on five public benchmarks compare GMP with heuristic alternatives such as power normalization.

2. Related Work

Related work addresses descriptor-frequency effects through pooling modifications or post-hoc normalization. These approaches include useful alternatives but are often heuristic or restricted to particular representation families.

  • Local descriptor pooling: Pooling aggregates local descriptor encodings while trading perturbation invariance against potential information loss.Spatial pyramids and suitable embeddings can constrain pooling in geometric or descriptor space.
  • Local descriptor pooling: Sum-pooling is widely used but assumes descriptor contributions can be added despite dependencies among descriptors.This independence assumption can make frequent descriptors disproportionately influential.
  • Local descriptor pooling: Max-pooling suits BOV and soft- or sparse-coding representations but is not directly applicable to higher-order statistics such as the Fisher Vector.Its operation requires embeddings that encode descriptor-to-codeword association strength.
  • Local descriptor pooling: Extensions such as ℓp-, softmax-, and weighted pooling provide transitions or reweighting, but prior weights may rely on external information.GMP is also a weighted-pooling instance, while its weights are derived within the representation framework.
  • Image-level descriptor normalization: Post-hoc normalization and burstiness corrections are often heuristic or limited to finite-vocabulary representations, excluding codebook-free methods such as EMK.A principled latent-model alternative is specific to Gaussian Mixture Model representations, whereas GMP is generic to aggregation-based representations.

3. GMP as equalization of similarities

GMP generalizes max-pooling by requiring every patch encoding to have equal dot-product similarity with the pooled representation. The resulting optimization projects sum-pooling through patch-encoding correlations and can be regularized toward sum-pooling.

  • 3. GMP as equalization of similarities: For BOV, max-pooling gives every patch encoding the same dot-product similarity with the pooled representation, equalizing frequent and rare patches.Sum-pooling instead weights a patch by its codeword frequency.
  • 3. GMP as equalization of similarities: GMP enforces a constant dot-product similarity c between every patch encoding and the pooled representation.The constant is set to 1 because the final representation is typically ℓ2-normalized.
  • 3. GMP as equalization of similarities: Because the equalization equations may be unsolvable or underdetermined, GMP uses least squares with a minimum-norm constraint.The solution uses the pseudo-inverse of the patch-encoding system.
  • 3. GMP as equalization of similarities: The GMP representation projects the sum-pooled representation using the uncentered patch correlation matrix ΦΦT.This differs from decorrelation methods that operate on data more generally; here the correlation matrix comes from patches in one image.
  • 3. GMP as equalization of similarities: In the hard-coded BOV case, GMP is strictly equivalent to max-pooling.This equivalence follows from designing GMP to reproduce max-pooling’s equal-similarity property.
  • 3. GMP as equalization of similarities: Regularized GMP uses ridge regression and smoothly interpolates between the unregularized GMP solution and sum-pooling as λ increases.The method can be computed with conjugate gradient descent, while block-sparse encodings can reduce computation.

4. GMP as weighted pooling

GMP can be interpreted as weighted pooling: patch encodings are re-weighted using weights derived from patch-to-patch similarities, while structured encodings can reduce computation.

  • 4. GMP as weighted pooling: GMP represents the pooled descriptor as a linear combination of patch encodings with learned weights.This makes GMP an instance of weighted pooling and yields the representation through re-weighted per-patch encodings.
  • 4. GMP as weighted pooling: The dual formulation computes the weights from the kernel matrix of patch-to-patch similarities rather than directly from the patch encodings.The kernel is K = ΦT Φ, and the resulting weights depend only on this similarity kernel.
  • 4. GMP as weighted pooling: For VLAD and hard-assignment Fisher Vectors, block sparsity makes the kernel matrix block-diagonal and reduces its computation cost with an inverted-file structure.The stated cost decreases from O(N^2D) to O(N^2D/C^2) when matching only patches assigned to the same codeword.
  • 4. GMP as weighted pooling: The primal formulation is more efficient experimentally, whereas the dual formulation helps visualize how GMP weights patches.After weights are computed, the representation is obtained by linearly re-weighting the encodings; the experiments instead use the primal formulation.
  • 4. GMP as weighted pooling: Weight-based topographic maps aggregate patch weights over image locations and highlight rare regions without guaranteeing foreground/background segmentation.Highly weighted regions can contain rare patches, while simple backgrounds may lead to foreground-background separation.

5. Experimental Evaluation

Experiments evaluate GMP with EMK and FV representations across five public image-classification benchmarks. GMP consistently improves performance, surpassing or complementing power normalization and benefiting especially fine-grained tasks.

  • Experimental setup: Experiments use EMK and FV representations across four fine-grained datasets and VOC-2007 under standard training, validation, and test protocols.The datasets are CUB-2010, CUB-2011, Oxford Pets, Oxford Flowers, and PASCAL VOC 2007.
  • EMK results: EMK with GMP improves performance by 3%–20% on every dataset, with an average improvement of 16% on fine-grained datasets.The comparison is against baseline EMK using sum-pooling.
  • FV results: 30.8% top-1 accuracy is achieved by GMP on CUB-2011 with FV, compared with 13.2% for sum-pooling.This result is reported for the CUB-2011 fine-grained classification dataset.
  • FV results: GMP outperforms power normalization across datasets, with average late-fusion improvements of 2.8% for G = 16 and 0.4% for G = 256.Combining GMP with power normalization yields larger average improvements of 3.8% and 2.5%, respectively.
  • FV results: With spatial pyramids on VOC-2007, GMP reaches 62.0% versus 60.2% for the power baseline, a 1.8% increase.The comparison uses late fusion with G = 256.
  • FV results: Increasing G from 16 to 256 consistently improves performance, while GMP remains complementary and is particularly attractive for low-dimensional FVs.The experiments therefore support using GMP alongside larger Gaussian mixtures when possible.

6. Conclusions

The paper presents GMP as a general pooling method that equalizes the influence of frequent and rare descriptors across encoding techniques. It also distinguishes GMP from related democratic aggregation and notes that their practical comparison remains unresolved.

  • GMP equalizes the influence of frequent and rare descriptors while remaining applicable to any encoding technique.
  • GMP extends the desirable behavior of max-pooling beyond count-based representations such as BOV and its coding extensions.
  • GMP can be computed efficiently in the primal, whereas democratic aggregation can only be computed in the dual.
  • The practical comparison between GMP and democratic aggregation remains to be established.

A. GMP and Max-Pooling

For finite codebook encodings with orthonormal codewords, GMP depends on which codewords appear rather than their frequencies. In the BOV hard-coding case, this produces the standard max-pooled representation, but the equivalence does not extend to soft- or sparse-coding.

  • GMP is analyzed for encodings drawn from a finite codebook whose codewords form an orthonormal matrix.
  • GMP does not depend on codeword proportions, only on whether each codeword is present in the image.
  • In BOV hard-coding, GMP becomes a binary representation indicating the presence or absence of each codeword, exactly matching max-pooling.
  • The equivalence between standard max-pooling and GMP does not hold for soft- or sparse-coding, while GMP is invariant to rotations of the encodings.
Loading 1406.0312v1…