Source-linked AI summary
Top-down Visual Saliency Guided by Captions
Vasili Ramanishka, Abir Das, Jianming Zhang, Kate Saenko
TL;DR
Captioning models accurately describe images and videos but make their region-to-word mappings difficult to interpret, while conventional top-down saliency cannot use natural-language queries. The paper introduces Caption-Guided Visual Saliency, which extracts word-level spatial or spatiotemporal heatmaps from encoder-decoder models without explicit attention layers; it matches existing captioning performance and yields more accurate saliency heatmaps.
Problem
Captioning models obscure how visual regions map to generated words, and existing top-down saliency methods cannot use natural-language sentences as task inputs.
Method
Caption-Guided Visual Saliency measures each region’s or frame’s information gain for generating a word, producing spatial or spatiotemporal heatmaps for predicted or arbitrary captions.
Results
The method achieves comparable captioning performance to existing methods while producing more accurate saliency heatmaps and outperforming the compared methods on attention correctness.
Takeaways & Limitations
Caption-Guided Visual Saliency can analyze the visual-word correspondences learned by existing encoder-decoder captioning architectures without adding explicit attention layers.
Takeaways & Limitations
The reported evaluation includes protocol differences between baseline tables, including central cropping of test and training images for fair comparison with prior work.
Abstract
from arXiv · showhide
Neural image/video captioning models can generate accurate descriptions, but their internal process of mapping regions to words is a black box and therefore difficult to explain. Top-down neural saliency methods can find important regions given a high-level semantic task such as object classification, but cannot use a natural language sentence as the top-down input for the task. In this paper, we propose Caption-Guided Visual Saliency to expose the region-to-word mapping in modern encoder-decoder networks and demonstrate that it is learned implicitly from caption training data, without any pixel-level annotations. Our approach can produce spatial or spatiotemporal heatmaps for both predicted captions, and for arbitrary query sentences. It recovers saliency without the overhead of introducing explicit attention layers, and can be used to analyze a variety of existing model architectures and improve their design. Evaluation on large-scale video and image datasets demonstrates that our approach achieves comparable captioning performance with existing methods while providing more accurate saliency heatmaps. Our code is available at visionlearninggroup.github.io/caption-guided-saliency/.
1. Introduction
Captioning models describe images and videos well but obscure how visual regions support individual words. Caption-Guided Visual Saliency addresses this by extracting spatial or spatiotemporal word-level saliency from encoder-decoder models without explicit attention layers.
- Motivation: Existing top-down saliency methods use high-level semantic inputs such as object categories but cannot directly guide search with natural-language sentences.This limits language-driven localization of subjects, verbs, and other sentence components.
- Motivation: Captioning models are difficult to interpret because their internal mappings between visual regions and generated words are not transparent.The paper asks whether words such as “woman” or “meat” arise from visual recognition or language-model regularities.
- Contribution: Caption-Guided Visual Saliency produces spatial or spatiotemporal heatmaps for predicted captions or arbitrary input sentences using encoder-decoder captioning models.The approach exposes how captioning models correspond visual content with words and can support future model design.
- Approach: The method estimates saliency through information gain by measuring how a single temporal frame or spatial region affects the probability of generating a word.It applies signal-dropout-inspired analysis to LSTM encoder-decoder models rather than adding explicit attention layers.
- Results: The method achieves comparable captioning performance to explicit soft-attention models while requiring less computational overhead and enabling more accurate word localization.Experiments evaluate localization on Flickr30kEntities and captioning on MSR-VTT.
2. Related Work
Prior top-down saliency methods recover pixel importance for class labels, while soft-attention captioning models learn explicit input weights during generation. Caption-Guided Visual Saliency extends saliency analysis from isolated object labels to linguistic sentences.
- Top-down neural saliency: Top-down saliency methods use class-label supervision or activation backprojection to recover class-specific importance over image pixels.These methods analyze CNN representations using partial derivatives or deconvolution through max-pooling layers.
- Soft Attention: Soft-attention architectures compute learned weights over visual inputs before generating each word, but require modifying the network and may not scale well.The paper contrasts this explicit attention mechanism with its post hoc saliency analysis.
- Contribution: The paper extends top-down saliency from isolated object labels to natural-language sentences.This reframes saliency as localization of visual concepts corresponding to words in a caption.
3. Background: Encoder-Decoder Model
The encoder-decoder captioning model converts video frames into feature descriptors, encodes them into a representation, and sequentially generates words. Soft attention instead retains the descriptor sequence and dynamically weights it using the decoder state.
- Model framework: The base video description model is an encoder-decoder sequence-to-sequence framework adapted from neural machine translation.The paper later applies the same base model to still-image captioning.
- Encoder: The encoder converts a sequence of video frames into high-level feature descriptors before producing a fixed-length representation for decoding.A CNN typically extracts the descriptors, which are then encoded by a potentially nonlinear function or LSTM.
- Encoder: Encoder representations can be formed from the last LSTM state or by averaging all feature descriptors.These alternatives compress the visual sequence before decoding.
- Decoder: The decoder generates each target word conditionally from the encoded representation and previously generated words.The decoder LSTM produces a conditional probability distribution sequentially across the output sequence.
- Soft Attention: Soft attention retains the full descriptor sequence and feeds the decoder a dynamically weighted sum at each timestep.The weights depend on the previous decoder state and the encoded visual sequence, enabling a search for relevant concepts over the video.
4. Approach
Caption-Guided Visual Saliency extracts word-specific spatial and temporal importance from encoder-decoder captioning models by measuring how localized visual inputs affect word probabilities. It supports predicted or arbitrary query sentences for images and videos without adding explicit attention layers.
- 4. Approach: Caption-Guided Visual Saliency produces spatial or temporal heatmaps for captions predicted by the model or for arbitrary input sentences.The same framework supports still images and videos and can analyze word-level correspondences in both settings.
- 4.1. Video Saliency: For video, temporal saliency is computed per frame and word, while spatial saliency recovers region importance from convolutional activations discarded by spatial pooling.The encoder-decoder processes frame descriptors sequentially; localized activations are then evaluated through information loss for each word.
- 4. Approach: The method estimates each visual item's saliency for a word from the information lost when the model processes only that item rather than the full input sequence.A forward pass with a single descriptor yields the word-probability change used to quantify its contribution.
- 4.1. Video Saliency: The resulting loss values are mapped to receptive-field locations, scaled into saliency values, normalized across inputs, and summed across words for phrases.Lower information loss corresponds to higher saliency, and phrase saliency aggregates the values of its constituent words.
- 4.1. Video Saliency: The approach requires repeated forward passes for spatial and temporal maps, but independent loss computations can be batched efficiently.For a video, temporal maps require m passes and spatial maps r × m additional localized evaluations, with batching over these computations.
5. Experiments
Experiments evaluate caption-guided saliency for image and video localization, while comparing captioning performance with soft attention. The method localizes noun phrases against annotated boxes and produces spatial, temporal, and spatiotemporal visualizations.
- Captioning performance is evaluated with METEOR across MSVD, MSR-VTT, and Flickr30k against soft attention.Table 3 reports the comparison for two video datasets and one image dataset, with higher values indicating better performance.
- Compared with random pointing, saliency localization is better on average and across every noun-phrase category.The center-baseline improvement is smaller on average, but some categories show significant gains.
- The method outperforms uniform and soft-attention baselines on Flickr30kEntities attention correctness.Table 2 compares the proposed method with uniform attention, soft attention, and a supervised soft-attention variant.
- Video visualizations provide spatial and temporal saliency for individual caption words, including frames ranked as most relevant.Darker grey indicates higher relative frame saliency, and selected frames are highlighted by matching colors.
- Image visualizations cover arbitrary query sentences and predicted captions, with noun-phrase maps formed by summing token responses.Maps use red for highest saliency, blue for lowest, asterisks for maximum points, and white boxes for ground truth.
- Some qualitative errors reflect visual ambiguity or normalization artifacts rather than consistently incorrect localization.The model sometimes attends to a gate resembling a window, while concentrated saliency for function words can result from normalization.
6. Conclusion
Caption-guided top-down saliency helps explain image and video captioning decisions without adding explicit attention layers.
- The approach explains complex image and video captioning decisions without modifying models with explicit attention layers.
- It maintains good captioning performance while producing more accurate heatmaps than existing methods.
- The method generalizes across a wide variety of encoder-decoder architectures.