PaperlayerView exact revision on arXiv

Source-linked AI summary

Compact Bilinear Pooling

Yang Gao, Oscar Beijbom, Ning Zhang, Trevor Darrell

arXiv:1511.06062v2cs.CV

TL;DR

Bilinear pooling offers strong visual recognition performance but produces features too large for practical subsequent analysis. This paper introduces two kernelized compact bilinear representations with efficient back-propagation, achieving comparable performance at far lower dimensionality. Experiments support their utility across image classification settings.

  • Problem

    Bilinear features are highly discriminative but typically have hundreds of thousands to a few million dimensions, limiting their practicality for subsequent analysis.

  • Method

    The paper uses a kernelized view of bilinear pooling to develop Random Maclaurin and Tensor Sketch compact representations with efficient back-propagation.

  • Results

    An 8K-dimensional Tensor Sketch feature matches a 262K-dimensional bilinear feature while achieving 96.5% compression.

  • Takeaways & Limitations

    Compact Tensor Sketch representations can retain bilinear performance while reducing feature size and supporting end-to-end optimization.

  • Takeaways & Limitations

    PCA is impractical for the 262K-dimensional bilinear feature because solving it usually requires O(d3) operations.

Abstract

from arXiv · show

Bilinear models has been shown to achieve impressive performance on a wide range of visual tasks, such as semantic segmentation, fine grained recognition and face recognition. However, bilinear features are high dimensional, typically on the order of hundreds of thousands to a few million, which makes them impractical for subsequent analysis. We propose two compact bilinear representations with the same discriminative power as the full bilinear representation but with only a few thousand dimensions. Our compact representations allow back-propagation of classification errors enabling an end-to-end optimization of the visual recognition system. The compact bilinear representations are derived through a novel kernelized analysis of bilinear pooling which provide insights into the discriminative power of bilinear pooling, and a platform for further research in compact pooling methods. Experimentation illustrate the utility of the proposed representations for image classification and few-shot learning across several datasets.

1. Introduction

Bilinear pooling is discriminative but impractically high-dimensional, motivating compact representations that preserve performance while enabling end-to-end learning. The paper introduces kernelized compact pooling methods with efficient back-propagation.

  • Motivation: Bilinear pooling improves recognition but produces features with hundreds of thousands to millions of dimensions, creating storage and processing challenges.These challenges include expensive model or feature storage, costly feature concatenation, and difficult classifier regularization in few-shot settings.
  • Contribution: The paper proposes two compact bilinear pooling methods that reduce dimensionality by two orders of magnitude with little-to-no performance loss.The methods are motivated by a kernelized viewpoint of bilinear pooling.
  • Method: The proposed methods build on low-dimensional feature maps for polynomial kernels and investigate Random Maclaurin and Tensor Sketch approximations.The kernelized analysis connects bilinear features to polynomial kernels.
  • Contribution: The compact representations support efficient back-propagation, allowing end-to-end optimization of the recognition network.This preserves joint optimization across the visual recognition pipeline.

2. Related work

Prior visual recognition methods trade off compactness, discriminative power, and end-to-end learnability. Compact bilinear pooling is presented as a method designed to combine these properties more favorably.

  • Bilinear models: Bilinear models achieve state-of-the-art results across visual tasks but have not been widely adopted partly because their extracted features are prohibitively high-dimensional.Second-order pooling has been used in semantic segmentation and fine-grained recognition with hand-tuned and learned features.
  • Feature encoding: Fisher vectors and related encodings improve over hard vector quantization by incorporating second-order information into visual descriptors.Improved Fisher Vector encoding has achieved state-of-the-art performance on many datasets.
  • Pooling trade-offs: Fully connected pooling is compact and learnable end-to-end but requires fixed input image sizes and is less discriminative than other methods.The comparison highlights a compactness–discriminativeness trade-off in existing pooling approaches.
  • Pooling trade-offs: Fisher encoding is more discriminative than fully connected pooling but is high-dimensional and cannot be learned end-to-end.Bilinear pooling is discriminative and tuneable but very high-dimensional.
  • Proposed approach: Compact bilinear pooling is described as combining bilinear pooling’s effectiveness with substantially lower dimensionality.This positioning directly addresses the limitations summarized for fully connected, Fisher, and bilinear pooling.

3. Compact bilinear models

Compact bilinear pooling approximates the second-order polynomial-kernel behavior of full bilinear pooling with much lower-dimensional features, while retaining efficient end-to-end training. The proposed Random Maclaurin and Tensor Sketch methods reduce memory and computation demands while achieving similar classification performance.

  • Bilinear pooling: Bilinear pooling forms a global descriptor by summing outer products of local descriptors across spatial locations.The resulting c × c matrix is treated as a c^2-dimensional vector.
  • Kernelized view: Bilinear descriptors give a linear classifier the discriminative power of a second-order polynomial kernel.This follows because the descriptor comparison equals the sum of squared inner products between local descriptors.
  • Compact bilinear pooling: Low-dimensional feature maps that approximate the polynomial kernel can produce compact bilinear features with output dimension d much smaller than c^2.The paper investigates Random Maclaurin and Tensor Sketch as such approximations.
  • Compact bilinear pooling: Random Maclaurin estimates the polynomial-kernel quantity in expectation, with estimator variance decreasing by a factor of 1/d as output dimension increases.Tensor Sketch uses sketching functions and convolution to improve projection complexity and often provide better approximations in practice.
  • Back propagation: Back-propagation through both compact pooling methods has computational and storage complexity similar to the corresponding forward pass.Random Maclaurin uses matrix multiplications, while Tensor Sketch uses FFT, inverse FFT, and matrix multiplication operations.
  • Properties: 98% redundancy in the bilinear representation is suggested by similar performance from compact representations using only 2% of the bilinear feature dimension.The reported comparison uses a 1000-class VGG-VD classification setting with a compact output dimension of 10,000.
  • Properties: Tensor Sketch is linear in d log d + c, whereas bilinear pooling is quadratic in c and Random Maclaurin is linear in cd.For a 448×448 image, the reported forward-backward times are 0.77ms for bilinear pooling and 5.03ms for Tensor Sketch with d = 4096.
  • Alternative dimension reduction: PCA is impractical as an alternative when bilinear features have dimension d = 262K because solving PCA usually involves O(d^3) operations.The cited alternative also requires an initial sweep over the whole dataset to obtain principal components.

4. Experiments

The experiments assess compact pooling design choices, compare against PCA-based pooling, and benchmark the methods against state-of-the-art approaches.

  • The experiments investigate projection dimensionality and whether projection parameters should be tuned.
  • They compare the proposed pooling methods with a PCA-based compact pooling baseline.
  • They evaluate bilinear pooling and compact methods against state-of-the-art approaches on three common computer vision tasks.

4.1. Experimental details

The experiments use VGG-M and VGG-D convolutional features, compare bilinear, compact bilinear, fully connected, and Fisher encoding methods, and fine-tune the recognition networks under specified optimization settings.

  • Experiments use VGG-M and VGG-D convolutional layers as local descriptor extractors.The setups retain conv5 + ReLU from VGG-M and conv5_3 + ReLU from VGG-D.
  • Full bilinear pooling produces approximately 250K dimensions from the 512-channel final convolutional layer.
  • Compact bilinear pooling uses user-defined dimension d and random projection parameters W, with d = 8000 typically sufficient for near-maximum accuracy.Tuning W provides a positive but small boost in practice.
  • The comparison includes fully connected pooling and improved Fisher encoding as alternatives to bilinear pooling.The fully connected baseline replaces the final classification layer before fine-tuning, while Fisher encoding uses 64 GMM components.
  • Fine-tuning updates the whole network using logistic loss, a learning rate of 10^-3, weight decay of 5 × 10^-4, and batch sizes of 32 for VGG-M and 8 for VGG-D.Convergence typically occurs in fewer than 100 epochs.
  • For compact methods, back-propagation can tune the deeper network layers, the projection parameters W, or both.

4.2. Configurations of compact pooling

On CUB200, increasing the compact projection dimension brings RM and TS toward full bilinear performance, while fine-tuning substantially improves extremely low-dimensional representations.

  • On CUB200, increasing projection dimension brings both compact bilinear methods toward full bilinear performance.The experiments use 11,788 images across 200 bird species with VGG-M and no part annotations.
  • At d = 16K without fine-tuning, TS has 1.7% lower error than bilinear features while using 6.1% of their dimensions.
  • With fine-tuning, TS at d = 16K reaches 22.66% error versus 22.44% for bilinear pooling.
  • At lower dimensions, RM outperforms TS, especially when tuning W, whereas TS outperforms RM when d > 2000.The passage attributes RM's low-dimensional advantage to its greater number of parameters and learning capacity.
  • Fine-tuning restores much of the discriminative capability lost by fixed extremely low-dimensional representations.At d = 512, the representation achieves 25.54% error, a 3.1% drop relative to the 250,000-dimensional bilinear feature.
  • The experiments suggest using 2000–8000 features and tuning W mainly for extremely low-dimensional representations.They also emphasize the importance of fine-tuning and projection methods that support it.

4.3. Comparison to the PCA-Bilinear baseline

The PCA-Bilinear baseline is compared with Tensor Sketch at matched output dimensions using CUB and VGG-M, with Top 1 error reported without and with fine-tuning.

  • PCA-Bilinear reduces channel dimensions with a 1 × 1 convolution, making the final bilinear feature dimension k^2.
  • 23.8% error for PCA at 16K dimensions exceeds 23.2% error for TS at 4K dimensions.The gap is especially large at small dimensions and without fine-tuning; fine-tuning narrows it.
  • Table 3 reports Top 1 error rates for PCA-reduced features and Tensor Sketch without and with fine-tuning.

4.4. Evaluation across multiple data-sets

Across fine-grained, scene, and texture recognition, compact bilinear pooling generally approaches or matches full bilinear pooling while using far fewer dimensions.

  • Evaluation across multiple data-sets: Compact representations perform on par with bilinear pooling across three distinct image-classification tasks while requiring orders of magnitude less memory.The comparison covers CUB-200-2011, MIT indoor scenes, and the Describable Texture Dataset.
  • Bird species recognition: 19.90% error for full bilinear versus 21.83% for RM and 20.50% for TS before fine-tuning on CUB with VGG-D.The feature dimension falls from 250k to 8192; after fine-tuning, all bilinear methods reach 16.0% error.
  • Indoor scene recognition: 3.03% better than Fisher vector, full bilinear pooling outperformed Fisher vector on MIT scene recognition.Compact TS was 0.94% worse than full bilinear but 2.09% better than Fisher vector.
  • Indoor scene recognition: Fine-tuning negatively affected full and compact bilinear methods by about 2% on MIT scene recognition.The authors attribute this to the small training set and large number of VGG-D convolutional weights, while noting that it needs further attention.
  • Texture classification: 32.29% error made compact TS the strongest reported result in the texture experiment with VGG-D.This was 2.18% better than Fisher vector and slightly better than full bilinear at 32.50%.

4.5. An application to few-shot learning

The paper evaluates compact bilinear pooling for learning from very few labeled examples, motivated by the relationship between feature dimensionality and classifier hypothesis-space size.

  • An application to few-shot learning: Few-shot learning generalizes from very few labeled training samples, which matters when labels are expensive or time-consuming to acquire.The evaluation uses the CUB dataset and compares full bilinear pooling with TS pooling.
  • An application to few-shot learning: 2.5% remained the gap between TS and bilinear pooling with three samples per class, or 600 training images.As the number of shots increased, the scores converged around 15 images per class.

5. Conclusion

The conclusion presents kernelized bilinear pooling and two compact, back-propagatable representations as practical alternatives to full bilinear features.

  • Conclusion: An 8K-dimensional TS feature matched a 262K-dimensional bilinear feature, enabling 96.5% compression.The conclusion also states that TS is more compact than Fisher encoding and achieves stronger results.
  • Conclusion: Both compact representations allow back-propagation of gradients for end-to-end optimization of the classification pipeline.The kernelized framework relates bilinear pooling to a pairwise polynomial kernel over local descriptors.
  • Conclusion: The authors identify image retrieval, further processing, and embedded deployment as potential applications, while leaving those studies for future work.They also suggest exploring alternative kernels in deep visual recognition systems.
Loading 1511.06062v2…