PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-aware Captions from Context-agnostic Supervision

Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, Gal Chechik

arXiv:1701.02870v3cs.CVcs.AI

TL;DR

The paper asks how context-aware, discriminative image descriptions can be produced without collecting context-specific training data. It introduces joint inference over a generic captioning model and a listener-like discriminator, and reports stronger discrimination than baseline generative and speaker-listener approaches across justification and image-captioning evaluations.

  • Problem

    Context-aware captions require descriptions that distinguish related images or concepts, but collecting discriminative training utterances is costly and contexts may be unknown in advance.

  • Method

    The paper uses an introspective speaker with joint inference over a context-agnostic captioning model and a listener-like discriminator, without additional training.

  • Results

    Evaluations on CUB-Justify and COCO human studies show that the approach outperforms baseline generative and speaker-listener approaches at inducing discrimination.

  • Takeaways & Limitations

    Pragmatic, discriminative captions can be induced at inference time from generic context-agnostic captioning models for justification and image identification tasks.

  • Takeaways & Limitations

    Exact inference over the exponentially large sentence space is intractable, motivating approximate search procedures.

Abstract

from arXiv · show

We introduce an inference technique to produce discriminative context-aware image captions (captions that describe differences between images or visual concepts) using only generic context-agnostic training data (captions that describe a concept or an image in isolation). For example, given images and captions of "siamese cat" and "tiger cat", we generate language that describes the "siamese cat" in a way that distinguishes it from "tiger cat". Our key novelty is that we show how to do joint inference over a language model that is context-agnostic and a listener which distinguishes closely-related concepts. We first apply our technique to a justification task, namely to describe why an image contains a particular fine-grained category as opposed to another closely-related category of the CUB-200-2011 dataset. We then study discriminative image captioning to generate language that uniquely refers to one of two semantically-similar images in the COCO dataset. Evaluations with discriminative ground truth for justification and human studies for discriminative image captioning reveal that our approach outperforms baseline generative and speaker-listener approaches for discrimination.

1. Introduction

The paper studies pragmatic vision-language tasks that require descriptions to distinguish a target from a similar distractor, while avoiding costly context-specific training data. It introduces an introspective speaker approach and evaluates it on justification and discriminative image captioning.

  • The paper addresses pragmatic descriptions that emphasize image aspects useful for distinguishing a target from a similar distractor.A literal caption may describe an image correctly yet fail to disambiguate it from a contextually similar image.
  • Collecting context-aware training utterances is costly because possible objects and contexts are numerous, and the relevant context may be unknown in advance.
  • The two studied tasks are justification between fine-grained categories and discriminative captioning between semantically similar images.Justification targets a category-versus-distractor explanation, whereas discriminative captioning identifies one image among two similar images.
  • The proposed introspective speaker provides pragmatic behavior without additional training by using a unified inference procedure over a context-free captioning model.The approach reuses the generative model’s sampling distribution rather than training a separate discriminativeness model.
  • Evaluations on CUB-Justify and human studies on COCO show that the approach outperforms baseline approaches at inducing discrimination.The paper also introduces CUB-Justify, containing five captions for 3,161 image, target-class, and distractor-class triplets.

2. Related Work

The paper situates its approach within pragmatic language modeling, vision-language pragmatics, and work beyond ordinary image captioning. It differs from prior pragmatic systems by avoiding a separately trained listener and targeting vision tasks requiring explicit discrimination.

  • Prior pragmatic language work includes rule-based models, reinforcement-learning approaches, and neural systems that derive pragmatic behavior from context-free data.
  • The paper applies pragmatics to vision tasks that provide justifications and discriminative image captions for human-facing or interaction settings.
  • Unlike the most closely related abstract-scene approach, this method does not require training a second listener model and supports more efficient inference.
  • The proposed tasks extend image captioning toward fine-grained justification and descriptions that distinguish one image from a set of similar images.

3. Approach

The approach induces context-aware language from a context-agnostic captioning model by jointly favoring target-consistent words and suppressing words likely under a distractor. It implements this introspective speaker with emitter-suppressor beam search for justification and discriminative image captioning.

  • 3.1. Reasoning Speaker: For justification, the model generates a sentence for an image of target concept ct that discriminates it from distractor concept cd without receiving distractor-class images.The generic captioner is conditioned on both the image and target concept before contextual inference is applied.
  • 3.1. Reasoning Speaker: The reasoning-speaker objective balances linguistic adequacy from p(s|I, ct) against discriminativeness through a listener score, with λ controlling the trade-off.The formulation selects sentences that are both good according to the generative model and discriminative for the target against the distractor.
  • 3.2. Introspective Speaker: The introspective speaker reuses a generic language model to score target-consistent and distractor-inconsistent utterances without training an explicit listener.Its listener uses the same generative model conditioned on the target and distractor, enabling context-aware outputs from existing language models.
  • 3.3. Emitter-Suppressor (ES) Beam Search for RNNs: Emitter-suppressor beam search selects tokens likely under the target-conditioned emitter and unlikely under the distractor-conditioned suppressor.The emitter and suppressor share the previously generated word history, and beam search retains sentences with the highest emitter-suppressor ratio.
  • 3.3. Emitter-Suppressor (ES) Beam Search for RNNs: The trade-off parameter must be tuned because λ = 1 ignores context, whereas very low λ values can produce repeated or ungrammatical sentences.The authors report that an intermediate range producing both discriminative and well-formed sentences exists in practice.
  • 3.4. Discriminative Image Captioning: For discriminative image captioning, the emitter is conditioned on the target image and the suppressor on the distractor image, reusing the same emitter-suppressor search.This adapts the justification formulation from competing concepts to semantically similar images.

4. Experimental Setup

The experiments evaluate justification on the new CUB-Justify dataset and discriminative captioning on confusing COCO image pairs. The setup uses context-agnostic bird captions, human-authored justifications, and image-pair construction based on visual and caption similarity.

  • Justification: CUB-Justify evaluates fine-grained bird justifications using target and distractor classes drawn from shared hyper-categories.The dataset samples related species within 37 hyper-categories and reserves approximately 15 test images per CUB class.
  • Justification: The CUB source contains 11,788 images spanning 200 North American bird species, each annotated with five context-agnostic captions.
  • Justification: Human workers wrote target-image descriptions intended to prevent confusion with six distractor-class images while consulting bird morphology diagrams.
  • Evaluation: Justification quality is measured with CIDEr-D against discriminative CUB-Justify captions, while the speaker's context-agnostic captioning reaches 50.2 validation CIDEr-D using image and class inputs.
  • Discriminative image captioning: For COCO captioning, confusing image pairs are constructed from nearest neighbors in VGG-16 FC7 space and from combined visual-caption similarity.

5. Results

Across justification and discriminative captioning, introspective inference improves discrimination by combining generation with context-sensitive reasoning. Performance is strongest at intermediate context weights, while overly aggressive discrimination can degrade fluency and context-agnostic supervision has clear limits.

  • Justification: IS(λ) and semi-blind-IS(λ) outperform baselines across mid-range λ values, while semi-blind-IS(λ) remains stable across a wider λ range.The introspector's access to the image changes how discriminative λ must be to overcome image-conditioned signals.
  • Justification: As λ decreases from 1, methods first become more discriminative and then deteriorate when very low λ values produce rare, badly formed outputs.The degradation is stronger for IS(λ) than RS(λ), which samples from the generator and ranks the sampled outputs.
  • Justification: Both introspective-speaker models outperform baselines on CUB-Justify test results, with semi-blind-IS(λ) outperforming other methods.The comparison uses each model's best validation-selected λ and reports CIDEr-D and METEOR, with higher scores better.
  • Justification: Using 100 listener-reranked samples gives RS(λ) 18.8 CIDEr-D, comparable to semi-blind-IS(λ) with ten beams but with lower METEOR and higher sampling cost.The authors attribute the efficiency advantage to joint greedy inference over speaker and introspector rather than to a stronger listener.
  • Justification: Context changes which visual attributes are selected: descriptions emphasize brown coloring and a red throat for a Rufous Hummingbird rather than its nondiscriminative long beak.
  • Justification: Context class changes the selected cue, shifting Tennessee Warbler descriptions toward a white eyebrow or green color depending on the distractor.The approach can also repeat words when inference strongly encourages discrimination without discriminative training data.
  • Justification: Visual grounding improves justification: semi-blind-IS(λ) identifies a green crown and avoids an inaccurate red-chest claim that blind-IS(λ) produces.The result reflects substantial within-category variation in fine-grained bird species.
  • Discriminative Image Captioning: On COCO, human judgments favor introspective speaker captions over the speaker baseline for identifying target images across both easy and hard confusing pairs.

7. Conclusion

The paper introduces inference-time pragmatic language generation that uses distractor context without discriminative training data. It applies this approach to justification and discriminative image captioning, with experiments showing advantages over generative and adapted prior baselines.

  • The proposed technique derives context-aware pragmatic language from recurrent neural network captioning models during inference.It accounts for a distractor class or image without requiring discriminative training data.
  • The paper evaluates pragmatic reasoning on justification and discriminative image captioning tasks.Justification contrasts related categories, while discriminative captioning distinguishes closely related images.
  • Experiments show the method outperforms generative baselines and adaptations of previous work.

Appendix

The appendix documents supplementary analyses, model adaptations, optimization details, evaluation metrics, and dataset collection procedures. It also organizes supporting material for visual explanations and performance analysis.

  • The appendix analyzes performance when unrelated images serve as distractors.
  • It documents adaptations for visual explanations, including architectural changes and optimization details for justification models.
  • It provides details on justification metrics and CUB-Justify data collection.

1. COCO Qualitative Results

On difficult COCO image pairs, introspective-speaker captions often identify the target more clearly than standard speaker captions. The examples also examine how discrimination changes as distractors become less visually similar.

  • The introspective speaker often produces more discriminative captions than the baseline speaker on hard COCO confusion examples.For example, it generates “a delta passenger jet flying through a clear blue sky,” distinguishing the target better than the baseline caption.
  • The qualitative analysis compares target images with nearest-neighbor, tenth-nearest-neighbor, and random distractors.
  • As distractors become less relevant, the examples assess how the quality of introspective-speaker discriminative captions varies.

2. Comparison to previous work on Generating Visual Explanations [16]

The paper adapts visual-explanation captioning to justification and adds emitter-suppressor inference to model a specific distractor class. This adaptation improves CUB-Justify performance while highlighting the distinction between context-independent and context-specific discrimination.

  • The vis-exp-IS approach conditions caption generation on the target class and uses emitter-suppressor beam search to account for context.
  • Qualitative comparisons examine target images against distractors at varying distances and show cases where introspective captions are more discriminative or inaccurate.
  • Unlike the baseline visual-explanation approach, the method can explicitly reason about a particular distractor class at inference time.
  • 21.52 CIDEr-D with a standard error of 0.17, improving 1.16 CIDEr-D over vis-exp on CUB-Justify.The vis-exp-IS model applies emitter-suppressor inference to the visual-explanation setup.
  • The visual-explanation baseline already captures context-independent discriminative signals through policy-gradient training.

3. CUB Captioning Model Architecture

The CUB captioning model extends Show, Attend and Tell with class information at multiple stages, while a blind variant relies only on the class embedding.

  • 3. CUB Captioning Model Architecture: The CUB adaptation embeds each of 200 class labels into a 512-dimensional vector and adds it to the Show, Attend and Tell architecture.
  • 3. CUB Captioning Model Architecture: The model incorporates the class embedding into the initial LSTM state, recurrent computation, and deep output layer.Class information is combined with averaged image annotations for initialization, added alongside the attention context during recurrence, and used to predict output words.
  • 3. CUB Captioning Model Architecture: The architecture changes preserve the original attention-based image context while conditioning language generation on the target class.
  • 3. CUB Captioning Model Architecture: The blind-IS(λ) variant removes attention and generates sentences using only the class embedding.

4. Optimization Details

The CUB captioning network uses Rmsprop training and largely follows the original Show, Attend and Tell design choices.

  • 4. Optimization Details: The network is trained with Rmsprop, batch size 32, learning rate 0.001, and learning-rate decay every five epochs.
  • 4. Optimization Details: Word embeddings have 512 dimensions, while the LSTM hidden and cell states each have size 1800.

5. Metrics for Justification

The justification evaluation compares caption metrics and inference strategies, highlighting both metric limitations and the efficiency advantage of joint inference over sampling and reranking.

  • 5. Metrics for Justification: SPICE can be problematic because discriminative justifications trade off sentence likelihood against discrimination rather than matching well-formed discriminative references.
  • 5. Metrics for Justification: SPICE scores are sometimes difficult to interpret for fine-grained bird descriptions, likely because WordNet-based soft similarity is poorly calibrated for specialized terms.
  • 5. Metrics for Justification: IS(λ) reaches the performance of RS(λ) with λ = 0.5 at 100 samples while using beam size 10, indicating lower-cost search.
  • 5. Metrics for Justification: 18.4±0.2 CIDEr-D for semi-blind-IS(λ) versus 16.2±0.3 for RS(λ)-TL demonstrates the benefit of joint inference.
  • 5. Metrics for Justification: At λ = 0, the trained listener scores 14.7±0.2, RS(0) scores 13.9±0.2, and the chance ranker scores 13.4±0.2.
Loading 1701.02870v3…