PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-Aware Visual Policy Network for Fine-Grained Image Captioning

Zheng-Jun Zha, Daqing Liu, Hanwang Zhang, Yongdong Zhang, Feng Wu

arXiv:1906.02365v1cs.CV

TL;DR

Fine-grained captioning requires compositional visual reasoning, but existing methods do not explicitly make visual policy depend on visual history. CAVP feeds previous visual attentions into current visual decisions for sentence and paragraph generation. Experiments report improvements across SPICE compositional scores and state-of-the-art performance on the captioning tasks described by the paper.

  • Problem

    Existing captioning methods focus on language policy rather than visual policy, leaving visual context important for compositional reasoning insufficiently represented, especially in longer paragraphs.

    Source: §1 · pp. 1–2

  • Method

    CAVP makes previous visual outputs available as context for current visual decisions, with hierarchical visual policies for paragraph generation.

    Source: §1 · p. 2

  • Results

    The paper reports significantly improved object, relation, and attribute SPICE scores without optimizing SPICE, and state-of-the-art performance under all metrics on the “Karpathy” test split.

    Source: §1 · p. 3, §4.3.2 · p. 8

  • Takeaways & Limitations

    CAVP can attend to complex visual compositions over time, and the paper reports this as beneficial for image captioning.

    Source: §6 · p. 12

Abstract

from arXiv · show

With the maturity of visual detection techniques, we are more ambitious in describing visual content with open-vocabulary, fine-grained and free-form language, i.e., the task of image captioning. In particular, we are interested in generating longer, richer and more fine-grained sentences and paragraphs as image descriptions. Image captioning can be translated to the task of sequential language prediction given visual content, where the output sequence forms natural language description with plausible grammar. However, existing image captioning methods focus only on language policy while not visual policy, and thus fail to capture visual context that are crucial for compositional reasoning such as object relationships (e.g., "man riding horse") and visual comparisons (e.g., "small(er) cat"). This issue is especially severe when generating longer sequences such as a paragraph. To fill the gap, we propose a Context-Aware Visual Policy network (CAVP) for fine-grained image-to-language generation: image sentence captioning and image paragraph captioning. During captioning, CAVP explicitly considers the previous visual attentions as context, and decides whether the context is used for the current word/sentence generation given the current visual attention. Compared against traditional visual attention mechanism that only fixes a single visual region at each step, CAVP can attend to complex visual compositions over time. The whole image captioning model -- CAVP and its subsequent language policy network -- can be efficiently optimized end-to-end by using an actor-critic policy gradient method. We have demonstrated the effectiveness of CAVP by state-of-the-art performances on MS-COCO and Stanford captioning datasets, using various metrics and sensible visualizations of qualitative visual context.

1 INTRODUCTION

Image captioning needs to move beyond fluent word prediction toward fine-grained descriptions requiring compositional reasoning. CAVP brings visual history into visual policy decisions for sentence and paragraph captioning, with reported gains on compositional SPICE scores.

1 INTRODUCTION

  • Conventional supervised captioning can learn data bias rather than machine reasoning, a challenge made more acute by the longer, finer-grained paragraphs expected from images.The paper also identifies an RL-based visual-policy gap: existing frameworks do not make visual policy a sequence of decisions informed by historical visual actions.Source: §1 · pp. 1–2
  • CAVP explicitly uses previous visual attentions as context during sequential visual reasoning, supporting compositions such as object relationships across words and sentences.It can attend to complex visual compositions over time rather than fixing a single region at each step.Source: §1 · p. 2
  • For paragraph generation, hierarchical CAVP uses visual context at both sentence and word levels, alongside paragraph-level and sentence-level rewards.CAVP is decomposed into four sub-policies; the paper says this reduces search complexity to linear time and stabilizes Monte Carlo policy rollout.Source: §1 · p. 2
  • The framework is trained end-to-end with actor-critic policy gradients and a self-critic baseline.The paper reports experiments on MS-COCO sentence captioning and the Stanford paragraph captioning dataset.Source: §1 · pp. 2–3
  • CAVP significantly improves every SPICE compositional score, including object, relation, and attribute, without optimizing SPICE.The paper also reports qualitative visual-policy reasoning over generation.Source: §1 · p. 3

2 RELATED WORK

Prior sentence-captioning attention focuses on the current visual step, while paragraph-captioning work does not account for visual context over time. The paper frames visual and language actions as sequential decisions and uses policy-gradient training with a self-critic value.

2.1 Image Sentence Captioning

  • Earlier image-sentence captioning attention methods focus on current-step visual attention; the paper proposes adding historical visual attention as context.The cited approaches include soft and hard attention, channel-wise attention, a visual sentinel, and bottom-up attention.Source: §2.1 · p. 3

2.2 Image Paragraph Captioning

  • Prior paragraph-captioning approaches are described as neglecting visual context over time, and the paper introduces hierarchical CAVP to address paragraph coherence and exposure bias.The paper identifies a lack of correlation among paragraph sentences as a consequence of omitting visual context.Source: §2.2 · p. 3

2.3 Sequential Decision-Making

  • The paper formulates each word prediction as a policy based on CAVP's action and optimizes with policy gradients using a self-critic value.This training can directly optimize non-differentiable metrics such as CIDEr.Source: §2.3 · p. 3

3 APPROACH

The approach section lays out the model and training strategy in sequence: task formulation, CAVP, language policy, then sequence training.

3 APPROACH

  • The section first formulates image captioning as sequential decision-making and profiles the proposed models.This is covered in Section 3.1.Source: §3 · p. 3
  • It then introduces the Context-Aware Visual Policy network and the language policy network.These are covered in Sections 3.2 and 3.3, respectively.Source: §3 · p. 3
  • The section concludes its framework overview with the sequence-training strategy for the full model.This is covered in Section 3.4.Source: §3 · p. 3

3.1 Overview

The model treats captioning as sequential decision-making: CAVP chooses a visual representation from the image and visual history, and the language policy predicts words. Paragraph captioning adds sentence-level topic and stop decisions that guide word generation.

3.1 Overview

  • The proposed captioning agent consists of CAVP and a language policy, acting in an image environment to generate a sentence or paragraph.CAVP's action is a visual representation, while the language policy's action is a predicted word.Source: §3.1 · p. 4
  • At each step, CAVP uses the image and prior visual context to produce a visual representation, which the language policy combines with the preceding word to predict the next word.The agent's state also includes the words predicted so far, and its reward can be an evaluation metric comparing prediction with ground truth.Source: §3.1 · p. 4
  • For paragraph captioning, a sentence-level CAVP-LP produces a visual representation, topic vector, and stop probability; the topic vector then constrains word-level generation.The paragraph process first produces topic vectors and then translates each into a sentence.Source: §3.1 · p. 4

3.2 Context-Aware Visual Policy Network

CAVP decomposes visual decision-making into four soft-attention sub-policies, using prior visual outputs as context when relevant to the current task. For paragraphs, sentence- and word-level CAVPs maintain visual context at their respective levels.

3.2.1 Sub-Policy Networks

  • CAVP uses four sub-policy networks—single, context, composition, and output—to select visual features through soft attention.Each sub-policy encodes an observed state with an RNN using an LSTM cell; the output SP produces the visual feature used by the language policy and as later visual context.Source: §3.2 · p. 4, §3.2.1 · pp. 4–5
  • The single SP selects a feature from the detected image regions, while the composition SP operates on context features.The single SP’s output is fed to the output SP; the composition SP’s output supplies composition features for producing the visual output.Source: §3.2.1 · p. 5
  • The context SP selects informative historical visual context and combines it with detected region features for composition.The context SP considers historical visual outputs; an alternative representation uses only the preceding time step as context.Source: §3.2.1 · p. 5

3.2.2 Hierarchical Context-aware Visual Policy Network

  • For paragraph captioning, sentence-level CAVP uses previous sentence representations, and its output guides the word-level policy and sentence LSTM.The sentence-level CAVP produces a visual representation for each sentence from region features and prior visual context.Source: §3.2.2 · p. 5
  • Word-level CAVP generates sentence words using previous word-level visual representations as context, enabling repeated generation across the paragraph.The model describes a sentence as generally corresponding to a particular image region.Source: §3.2.2 · pp. 5–6

3.3 Language Policy Network

The language policy uses CAVP’s context-aware visual representations to generate words. For paragraphs, hierarchical language policies produce sentence topics and stopping decisions, then generate each sentence’s words.

3.3 Language Policy Network

  • At each step, the language policy uses a CAVP visual representation and its hidden state to update the LSTM, then predicts a distribution over vocabulary words.The sentence probability is calculated as the product of the conditional word distributions across time steps.Source: §3.3 · p. 6

3.3.1 Hierarchical Language Policy Network

  • For paragraph captioning, sentence-level and word-level language policies pair with their corresponding CAVPs; the sentence policy produces a topic vector and predicts when to stop.The word-level policy generates sentence words from the topic vector and visual representation; its LSTM predicts vocabulary-word probabilities.Source: §3.3.1 · p. 6
  • The paragraph framework has sentence-level and word-level CAVP-LP components.Figure 4 presents the overview of this hierarchical framework.Source: Figure 4 · p. 6

3.4 Sequence Training

Training combines supervised pre-training with reinforcement-learning fine-tuning. For paragraphs, the method uses sentence-level rewards and can initialize the output policy by cloning an expert before end-to-end training.

3.4 Sequence Training

  • Training first minimizes cross-entropy, then uses self-critical REINFORCE fine-tuning to optimize sequence-level metrics and address exposure bias.The method compares a sampled sequence’s reward with a greedy sequence’s relative baseline; the reward can be CIDEr, BLEU, or SPICE.Source: §3.4 · p. 6
  • CAVP decomposes visual search into four sub-policy tasks with O(N) complexity each, reducing the overall search complexity exponentially.This decomposition is presented as a way to make policy rollout more efficient.Source: §3.4 · p. 7

3.4.1 Hierarchical Sequence Training

  • Paragraph training adds sentence-ending prediction to word prediction and uses sentence-level rewards as a trade-off between shared paragraph rewards and unstable per-word rewards.The sentence reward is obtained by teacher-forcing preceding ground-truth sentences, then sampling or greedily searching the next sentence.Source: §3.4.1 · p. 7

3.4.2 Behavior Cloning

  • Behavioral cloning initializes the output policy by minimizing its KL-divergence from an expert policy alongside captioning loss; the expert is discarded before test time.The paper notes that the expert need not be optimal and continues training the model end-to-end with policy gradients.Source: §3.4.2 · p. 7

4 EXPERIMENTS ON SENTENCE CAPTIONING

The sentence-captioning experiments use MS-COCO and report both Karpathy-split and online-server evaluations. The section also describes evaluation metrics, preprocessing, and model settings.

4 EXPERIMENTS ON SENTENCE CAPTIONING

  • The experiments section covers settings, implementation details, quantitative and qualitative evaluations, and ablation studies.The supplied section passages do not report the evaluation outcomes.Source: §4 · p. 7

4.1.1 Dataset

  • The experiments use MS-COCO, reporting results on the Karpathy offline split and the MS-COCO online test server.The dataset has 82,783 training and 40,504 validation images; the Karpathy split has 113,287 training, 5,000 validation, and 5,000 test images.Source: §4.1.1 · p. 7

4.1.2 Metric

  • Evaluation includes n-gram similarity metrics and SPICE, which scores semantic tuples such as objects, relations, and attributes.The paper notes reported bias between conventional metrics and human judgment.Source: §4.1.2 · pp. 7–8

4.2.1 Data Pre-processing

  • Preprocessing lowercases and tokenizes captions, removes words occurring fewer than five times, and limits captions to 16 words.The resulting vocabulary contains 10,369 words; the model uses the top 36 region features per image.Source: §4.2.1 · p. 8

4.2.2 Parameter Settings

  • The model uses one-layer LSTMs with 1,300 hidden units, 1,024-unit attention, and 1,000-dimensional word embeddings.It starts reinforcement learning after 37 epochs and uses beam search with size 5 at inference.Source: §4.2.2 · p. 8

4.3 Comparisons to State-of-The-Arts

CAVP outperforms non-context methods across captioning benchmarks and improves every SPICE category over Up-Down. Its visualizations show how prior regions inform both word-level and compositional decisions.

4.3.1 Comparing Methods

  • Comparisons include traditional cross-entropy captioners and RL-based methods trained with different sequence-level reward returns.The traditional methods use encoder-decoder training without sequence training; the RL-based methods use sequence training with varied rewards.Source: §4.3.1 · p. 8

4.3.2 Quantitative Analysis

  • CAVP improves every SPICE semantic category over Up-Down, including relation and visual-comparison categories such as attribute, color, and size.The authors describe relation scores as reflecting object relationships and attribute, color, and size scores as reflecting visual comparisons.Source: §4.3.2 · p. 8, Figure 5 · p. 8
  • On the online MS-COCO test server, the single CIDEr-optimized model outperforms published works on every reported metric except BLEU-1.On the Karpathy test split, CAVP achieves state-of-the-art performance under all metrics and is a single model while other compared approaches use ensembles.Source: §4.3.2 · p. 8, Table 2 · p. 9

4.3.3 Qualitative Analysis

  • In qualitative examples, prior regions such as a boy’s hand inform the choice of “brushing,” while different contexts support the same words in different scenes.A separate visualization shows policy decisions across a sentence, including “standing” connecting “man” with “snow.”Source: §4.3.3 · p. 9

4.4 Ablation Studies

The ablations examine architecture, parameter sharing, computational cost, and reward choice. They indicate that retaining historical visual context improves performance over using only the last visual feature.

4.4.1 Architecture

  • The architecture variants compare single-policy Up-Down, scratch-trained and cloned output policies, and a non-sharing configuration against CAVP.The context sub-policy uses the last visual feature in the scratch and cloning variants; the non-sharing variant does not share weights across the four sub-policies.Source: §4.4.1 · p. 9
  • CAVP outperforms scratch and cloning variants that use only the last visual feature, supporting the value of memorizing historical visual context for later generation.The authors report comparable performance for scratch and cloning, and say the output policy can be learned from scratch without expert-policy guidance.Source: §4.4.1 · p. 10
  • CAVP slightly increases parameters and training and testing costs over Up-Down, while parameter sharing reduces both parameters and computational cost versus no sharing.The comparison reports parameter number, training time, and testing time per image.Source: §4.4.1 · p. 10, Table 4 · p. 10

4.4.2 Reward

  • Optimizing CIDEr or SPIDEr gives the best overall performance, but SPIDEr is more time-consuming because SPICE evaluation is slow; most experiments therefore optimize CIDEr.Optimizing a specific metric generally yields the best performance on that same metric.Source: §4.4.2 · p. 10, Table 5 · p. 10

5 EXPERIMENTS ON PARAGRAPH CAPTIONING

Experiments on Stanford image-paragraph captioning compare Hierarchical CAVP with existing methods and evaluate its components. Results favor Hierarchical CAVP on most metrics, while ablations compare visual-context and training choices.

5.1 Experiment Settings

  • The Stanford dataset contains 14,575 training, 2,487 validation, and 2,489 test images, each paired with one paragraph averaging 5.7 sentences.Images were represented by 50 region features of 4,096 dimensions; the vocabulary contained 4,237 words.Source: §5.1 · p. 10

5.2 Comparison to State-of-the-Arts

  • Hierarchical CAVP outperforms state-of-the-art methods on most Stanford paragraph-captioning metrics, including RTT-GAN* with additional training data; human descriptions lead especially on CIDEr and METEOR.The comparison includes automatic baselines and a human evaluation using additional paragraphs for 500 randomly chosen images.Source: Table 6 · p. 11, §5.2 · p. 11

5.3 Ablation Studies

  • Sentence CAVP beats Single Policy, hierarchical sentence- and word-level visual policies improve further, and BLEU optimization outperforms CIDEr optimization despite trade-offs against cross-entropy.Compared with cross-entropy training, CIDEr- or BLEU-optimized variants improve some metrics and degrade others; the authors attribute this to having only one reference paragraph per image.Source: Table 7 · p. 11, §5.3 · pp. 11–12

5.2 Comparison to State-of-the-Arts

  • The comparison covers sentence concatenation, flat and hierarchical paragraph models, template generation, DenseCap concatenation, and RTT-GAN variants.The authors also collected human paragraphs for 500 randomly selected images.Source: §5.2 · p. 11
  • Generated paragraphs can form a storyline by shifting attention from a nearby sign to roadside mountains and then a distant mountain.The examples show paragraphs with a variable number of sentences.Source: §5.2 · p. 11, Figure 8 · p. 11

6 CONCLUSION

The conclusion presents CAVP for image sentence and paragraph generation, using visual context for compositional reasoning over time. Experiments report top-ranking performance on MS-COCO and the Stanford paragraph-captioning dataset, and the authors identify further applications and search strategies for future work.

6 CONCLUSION

  • CAVP targets fine-grained image sentence and paragraph generation, incorporating visual context into sequential visual reasoning rather than fixing attention on one region per step.The authors describe it as the first RL-based image-captioning model to incorporate visual context into sequential visual reasoning.Source: §6 · p. 12
  • Experiments report that CAVP boosts RL-based image-captioning performance and achieves top-ranking results on MS-COCO and Stanford image-paragraph captioning.Source: §6 · p. 12
  • Future work will integrate visual and language policies into Monte Carlo search and apply CAVP to visual question answering and visual dialog.Source: §6 · p. 12
Loading 1906.02365v1…