PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning the Best Pooling Strategy for Visual Semantic Embedding

Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, Changhu Wang

arXiv:2011.04305v5cs.CV

TL;DR

VSE retrieval systems must select pooling strategies across modalities and feature extractors, a costly task that becomes harder for variable-sized features. The paper introduces GPO, which learns pooling coefficients automatically, and builds VSE∞ by replacing standard aggregators with GPOs. VSE∞ outperforms previous VSE methods on image-text retrieval and reaches state-of-the-art video-text retrieval results, while experiments show GPO discovers effective pooling strategies.

  • Problem

    Selecting the best pooling function for each modality and feature extractor requires costly manual search, especially for variable-sized feature sets.

  • Method

    GPO learns coefficients for sorted feature vectors with a sequence-model generator, and VSE∞ plugs separate GPOs into the standard VSE framework.

  • Results

    VSE∞ outperforms previous VSE methods on image-text retrieval benchmarks and achieves state-of-the-art results on two video-text retrieval datasets.

  • Takeaways & Limitations

    GPO automatically discovers effective pooling strategies and serves as a plug-and-play aggregation module for VSE models.

  • Takeaways & Limitations

    The optimal combination of visual and textual pooling functions can require n × n search, increasing to O(n^3) with multiple visual feature extractors.

Abstract

from arXiv · show

Visual Semantic Embedding (VSE) is a dominant approach for vision-language retrieval, which aims at learning a deep embedding space such that visual data are embedded close to their semantic text labels or descriptions. Recent VSE models use complex methods to better contextualize and aggregate multi-modal features into holistic embeddings. However, we discover that surprisingly simple (but carefully selected) global pooling functions (e.g., max pooling) outperform those complex models, across different feature extractors. Despite its simplicity and effectiveness, seeking the best pooling function for different data modality and feature extractor is costly and tedious, especially when the size of features varies (e.g., text, video). Therefore, we propose a Generalized Pooling Operator (GPO), which learns to automatically adapt itself to the best pooling strategy for different features, requiring no manual tuning while staying effective and efficient. We extend the VSE model using this proposed GPO and denote it as VSE$\infty$. Without bells and whistles, VSE$\infty$ outperforms previous VSE methods significantly on image-text retrieval benchmarks across popular feature extractors. With a simple adaptation, variants of VSE$\infty$ further demonstrate its strength by achieving the new state of the art on two video-text retrieval datasets. Comprehensive experiments and visualizations confirm that GPO always discovers the best pooling strategy and can be a plug-and-play feature aggregation module for standard VSE models. Code and pre-trained models are available at https://vse-infty.github.io.

1. Introduction

Visual Semantic Embedding aligns visual and textual data in a shared space, but selecting effective pooling functions is costly across modalities and feature extractors. GPO learns this choice automatically and underpins VSE∞, which achieves strong retrieval results with minimal added computation.

  • Motivation: VSE learns a shared semantic embedding space where paired visual and textual instances are close, supporting image- or video-by-text retrieval.Its pipeline extracts features, aggregates them into holistic embeddings, and computes matching scores.
  • Problem: Finding the optimal pooling strategy requires costly, repetitive manual tuning across modalities and feature extractors, especially when feature sets have varying sizes.Grid-searching combinations becomes combinatorial and is particularly difficult for variable-length text or video features.
  • Method: GPO predicts coefficients for sorted feature vectors and computes their weighted sum, adapting pooling to each modality and feature extractor.Its tiny sequence-model generator handles variable-sized features while adding negligible computational cost.
  • Results: VSE∞ extends standard VSE with GPO-based visual and text aggregation and achieves state-of-the-art results across image-text and video-text retrieval benchmarks.The reported benchmarks include COCO, Flickr30K, MSR-VTT, and VaTeX.
  • Empirical finding: Carefully selected simple pooling functions can outperform complex visual aggregators in prior VSE methods for image-text matching.The finding motivates replacing more elaborate aggregation designs with simpler pooling choices.

2. Visual Semantic Embedding for Multi-modal Matching

VSE converts visual and textual features into holistic embeddings and compares them using cosine similarity. Its inference ranks candidates efficiently from precomputed embeddings, while training uses hard-negative triplet ranking; the section frames this against complex aggregators and reports pooling advantages.

  • Framework: VSE uses visual and text embedding functions to produce sets of visual and textual features from input data.Visual features may come from grids, regions, or spatial pyramids, while text features are contextualized word-token representations.
  • Framework: Visual and textual aggregators map extracted feature sets into holistic embeddings v and u in a shared space.These embeddings are the representations used for subsequent cross-modal matching.
  • Matching: The compatibility score is cosine similarity between the visual embedding v and text embedding u.The score compares the direction of the two holistic embeddings.
  • Inference: At inference, VSE ranks candidate images by matching scores against a query text, using precomputed embeddings and matrix multiplication for efficiency.The same scoring procedure supports retrieval over candidate items.
  • Learning: VSE training commonly uses a hinge-based triplet ranking loss with online hard-negative mining.Hardest negative text and image examples are selected within each mini-batch according to the learned model.

3. VSE∞with Generalized Pooling Operator

This section finds that carefully selected simple pooling can outperform complex aggregators and introduces GPO to learn suitable pooling strategies automatically across modalities, feature extractors, and variable-sized inputs.

  • 3.1. Simple Pooling Works the Best: Carefully selected simple pooling functions outperform complex aggregators in VSE image-text matching.Across grid and region features, MaxPool or K-MaxPool achieves the best reported recall@1 results among the compared pooling candidates.
  • 3.2. Generalizing over Different Pooling Strategies: GPO learns pooling coefficients over sorted feature values to generalize AvgPool, MaxPool, K-MaxPool, and other strategies.The coefficients are normalized with Softmax and weighted values are summed to produce the pooled representation.
  • 3.2. Generalizing over Different Pooling Strategies: GPO uses a coefficient generator g(k, N), allowing pooling to adapt when feature-set size N varies.This addresses the limitation of fixed trainable coefficient vectors for variable-length text and video features.
  • 3.3. Implementing Generalized Pooling Operator: The generator encodes positions with trigonometric positional features and predicts coefficients using a small BiGRU and MLP.Size Augmentation randomly drops 20% of input vectors during training to expose the generator to diverse feature-set sizes.
  • 3.4. Building up VSE∞using GPO: VSE∞ replaces standard visual and text AvgPool aggregators with GPOs while retaining the standard VSE matching framework.The two GPOs independently produce holistic image and text embeddings for computing matching scores.

4. Related Works

VSE methods learn joint image-text embeddings for efficient retrieval, while V+L BERTs use richer cross-modal interactions at higher retrieval cost. VSE∞ is reported to approach the best V+L BERT image-text matching performance while remaining much faster for large-scale retrieval.

  • Visual Semantic Embedding: VSE learns a joint embedding space where image-text compatibility is computed from holistic vectors.Its efficiency depends on learning strong image and text embedding functions.
  • Experimental Comparisons: Table 2 compares VSE-based methods on COCO and Flickr30K across different visual and textual backbones using RSUM for ranking.The table marks the best and second-best RSUM results and includes ensemble and pretraining annotations.
  • Vision-Language BERTs: V+L BERT methods use cross-modal attention and typically require BERT inference over image-text pairs.This interaction pattern is contrasted with the retrieval efficiency of VSE models.
  • Comparison: VSE∞ approaches the best V+L BERT image-text matching performance while being much faster in large-scale multi-modal retrieval.The comparison is presented for the best VSE∞ model in the referenced experiments.

5. Experiments

Experiments evaluate VSE∞ across image-text, video-text, alternative pooling, retrieval speed, and semantic matching settings. Across these evaluations, GPO improves retrieval performance while avoiding manual pooling searches and retaining efficient VSE inference.

  • Image-text Retrieval: VSE inference uses pre-computed holistic embeddings and matrix multiplication, making VSE methods much faster than V+L BERTs as image-candidate sets grow.Figure 3 measures single-GPU text-based image-retrieval inference time, where lower values are better.
  • Image-text Retrieval: VSE∞ outperforms VSE baselines across all three image-feature-extractor combinations without complicated aggregators.The strongest empirical results use WSL+BERT features and improve over the second-best feature combination by a large margin.
  • Image-text Retrieval: VSE∞ is no worse than three of five V+L BERTs with the same feature extractors and approaches OSCAR when using WSL features.The comparison uses COCO 5K because the reported V+L BERT 1K results use only the first fold rather than five-fold averages.
  • Video-text Retrieval: VSE∞ and its HGR∞ variant improve video-text retrieval over their corresponding baselines in RSUM.HGR∞ replaces global-matching pooling with GPO, while VSE∞ replaces frame and text AvgPool operations.
  • Comparing GPO to Alternative Poolings: GPO outperforms alternative learnable pooling methods across visual and textual feature combinations, while BERT’s default CLS aggregation is not optimal for multi-modal matching.The results support GPO as a plug-and-play per-modality feature aggregator.
  • Visualizing and Understanding GPO: 520.8 RSUM for GPO exceeds the 520.4 best result from a K-MaxPool grid search on BUTD region and BiGRU features.GPO therefore avoids the costly repetitive search, although its BiGRU strategy is not significantly better than the best selected K-MaxPool.
  • Visualizing and Understanding GPO: The optimal visual-textual pooling combination is entangled, requiring n × n search and potentially O(n^3) search complexity with multiple visual extractors.This interaction makes manual tuning more expensive than independently selecting pooling functions.

6. Conclusion

The paper proposes GPO as an adaptive pooling operator within VSE∞ and reports strong retrieval performance across image-text and video-text benchmarks.

  • GPO automatically adapts pooling to different data and feature backbones, replacing manual selection of pooling strategies.VSE∞ extends standard VSE by using GPO as the feature aggregator.
  • VSE∞ significantly outperforms previous VSE methods on image-text retrieval benchmarks across popular feature extractors.
  • Variants of VSE∞ achieve state-of-the-art results on two video-text retrieval datasets with simple adaptations.

2. More experiments and results, including

The appendix adds ablations, expanded benchmark comparisons, synthetic validation, and explorations of more complex GPO variants.

  • The appendix includes ablation studies analyzing GPO’s effects in VSE∞.
  • Additional experiments extend Table 2 with full COCO 5K results, more feature-backbone combinations, and more related baselines.
  • Synthetic experiments verify GPO design choices and motivate Size Augmentation, while complex GPO variants explore data-dependent and per-dimensional pooling.

A. Additional Implementation Details

The appendix details model settings, training procedures, implementation choices, evaluation corrections, and experimental configurations for image-text and video-text retrieval.

  • Model settings: Joint embeddings use dimensionality 1024, while feature dimensions vary across CNN, BiGRU, and BERT backbones before projection when necessary.CNN features use d1=2048; BiGRU uses d2=1024, and BERT uses d2=768.
  • Training details: Training uses scheduled learning-rate decay, warm-up strategies, and end-to-end optimization after the first epoch.The first warm-up epoch freezes the ConvNet for grid features and uses all batch negatives; later training uses linear learning-rate warm-up.
  • Evaluation corrections: The CVSE re-evaluation fixes concept-label leakage and inconsistent COCO 1K evaluation relative to other image-text methods.Valid concept labels are restricted to each caption, and COCO 1K results are averaged over five test folds.
  • Video-text and retrieval setup: Video-text experiments plug GPO into HGR’s video and text aggregation, while retrieval speed comparisons pre-compute unconditioned embeddings and use matrix multiplication for VSE scoring.The video-text models follow HGR settings; VSE similarity scores are computed after pre-computation.

B.1. More Ablation Studies

Ablations examine whether GPO is needed for both modalities, the effect of Size Augmentation, and alternative sequence models within GPO.

  • Using GPO for either visual or text features improves aggregation, while applying it to both modalities yields a larger improvement.The comparison replaces BUTD region AvgPool and BERT CLS aggregation with GPO in different combinations.
  • Size Augmentation is evaluated by randomly dropping visual or text inputs to measure its effect across modalities.
  • A simple BiGRU provides sufficient capacity and computational efficiency for the sequence model, whereas more complex multi-head attention mechanisms can hurt performance.

B.2. More results and comparisons for image-text matching

VSE∞ consistently improves image-text retrieval across ResNet-152 grid features and outperforms non-VSE methods using fine-grained cross-modal interactions, without complex cross-modal modeling.

  • VSE∞ consistently outperforms improved VSE++ and other baselines on ResNet-152 grid features.The improved VSE++ uses 512 × 512 images, following an empirical study.
  • VSE∞ surpasses SCAN, CAAN, and IMRAM under the same experimental setups without complex cross-modal modeling.These non-VSE methods use fine-grained cross-modality interactions with BUTD and BiGRU feature extractors.

B.3. Synthetic Experiment for Verifying GPO Design

Synthetic experiments compare GPO designs across pooling patterns and feature-set sizes, finding Cos/Sin+BiGRU strongest overall while revealing challenges on unseen and smaller inputs.

  • Evaluation Protocol: GPO is evaluated on AvgPool, K-MaxPool, Top-K% Pooling, and linearly-decayed pooling patterns.Training feature-set sizes range from 20 to 100, while test sizes range from 10 to 120, including seen and unseen sizes.
  • Image-Text Matching Extensions: The image-text matching extensions report results across COCO and Flickr30K using different visual and textual backbones.Table 10 additionally identifies ensemble results of two models, while Table 9 marks best and second-best RSUM results.
  • Different GPO Designs: Cos/Sin+BiGRU achieves the best overall performance among the compared GPO designs.The alternatives include interpolation, Cos/Sin+MLP, and Index+BiGRU designs.
  • Different GPO Designs: Interpolation has clear difficulty handling K-Max Pooling, while Index+BiGRU performs slightly worse than Cos/Sin+BiGRU.The comparison reports results using RMSE, where lower is better.
  • Different GPO Designs: Generalizing to unseen feature sizes is challenging, with smaller feature sizes harder than larger ones.Size Augmentation is proposed to improve GPO generalization across different input sizes.

B.4. Complex Variants of GPO

Experiments find that increasing GPO complexity does not improve VSE∞: per-dimension and data-dependent coefficient variants provide no gains, supporting position-based coefficients.

  • Per-Dimension Pooling: Per-dimension pooling coefficients provide no improvement over two feature-extractor combinations, potentially due to over-fitting.This variant generates separate coefficients for each feature dimension instead of shared coefficients.
  • Data-Dependent Coefficients: Adding the feature itself to the coefficient generator does not improve GPO; the position index alone suffices.The modification makes GPO a special form of self attention by conditioning coefficients on feature values and position.
  • Interpretation: The experiments reinforce that complicated feature aggregation does not necessarily improve VSE models.The paper suggests feature extractors may already provide adequate information for multi-modal matching, reducing the need for further contextualization.
Loading 2011.04305v5…