PaperlayerView exact revision on arXiv

Source-linked AI summary

Recurrent Fusion Network for Image Captioning

Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, Tong Zhang

arXiv:1807.09986v3cs.CV

TL;DR

Image captioning models typically rely on one CNN view, limiting how comprehensively they represent image semantics. RFNet fuses complementary outputs from multiple CNN encoders through a two-stage recurrent procedure, and it achieves leading MSCOCO performance, including CIDEr gains over Up-Down.

  • Problem

    Image captioning models generally use one CNN encoder, limiting the diversity and comprehensiveness of image representations.

  • Method

    RFNet uses a two-stage recurrent fusion procedure to combine multiple CNN representations into thought vectors for the decoder.

  • Results

    RFNet achieved leading performance on the MSCOCO evaluation server, with C5 and C40 CIDEr scores improved by 5.0 and 4.6 over Up-Down.

  • Takeaways & Limitations

    RFNet demonstrates that fusing complementary information from multiple image encoders can support leading image-captioning performance.

Abstract

from arXiv · show

Recently, much advance has been made in image captioning, and an encoder-decoder framework has been adopted by all the state-of-the-art models. Under this framework, an input image is encoded by a convolutional neural network (CNN) and then translated into natural language with a recurrent neural network (RNN). The existing models counting on this framework merely employ one kind of CNNs, e.g., ResNet or Inception-X, which describe image contents from only one specific view point. Thus, the semantic meaning of an input image cannot be comprehensively understood, which restricts the performance of captioning. In this paper, in order to exploit the complementary information from multiple encoders, we propose a novel Recurrent Fusion Network (RFNet) for tackling image captioning. The fusion process in our model can exploit the interactions among the outputs of the image encoders and then generate new compact yet informative representations for the decoder. Experiments on the MSCOCO dataset demonstrate the effectiveness of our proposed RFNet, which sets a new state-of-the-art for image captioning.

1 Introduction

Image captioning uses CNN encoders and RNN decoders, but relying on one encoder limits the diversity of image representations. RFNet addresses this gap by fusing complementary information from multiple CNNs before decoding.

  • Motivation: Image captioning must understand image content, object relationships, and image-language interactions to generate natural sentences.The task has practical applications including improving image retrieval and helping visually impaired people understand images.
  • Existing Framework: The encoder-decoder framework typically uses a CNN to extract image representations and an RNN to generate captions.A pre-trained image-classification CNN is usually used as the encoder.
  • Research Gap: Existing models generally use one encoder, making performance depend heavily on that CNN’s expressive ability.The paper identifies ResNet and Inception-X as examples of established CNN encoders.
  • Research Gap: Multiple encoders can provide diverse, complementary representations for more complete image semantics, but prior image-captioning models had not exploited this behavior.The paper frames multi-encoder representation as a way to better depict and characterize image meaning.
  • Proposed Approach: RFNet inserts a recurrent fusion procedure between multiple CNN encoders and the decoder to create more comprehensive thought vectors.The first stage models interactions among encoder-specific components, while the second stage compresses their outputs into one set for the decoder.

2 Related Works

Related work improves encoder-decoder captioning through attention, additional visual information, reinforcement learning, multi-task learning, and fusion. RFNet is positioned as an intermediate-fusion method that exploits relationships among multiple visual representations.

  • Encoder-Decoder Extensions: Attention mechanisms extend encoder-decoder captioning by selecting image subregions to focus on during generation.The related work describes spatial attention as an extension of the basic framework.
  • Additional Visual Information: Additional visual information has been incorporated through word-occurrence prediction and region-based multi-label classification.These approaches use predicted visual semantics to initialize or improve recurrent captioning models.
  • Reinforcement Learning: Policy-gradient reinforcement learning directly optimizes non-differentiable captioning metrics such as CIDEr.The cited system replaces cross-entropy loss with negative CIDEr score and trains with REINFORCE.
  • Multi-Task Learning: Multi-task sequence-to-sequence learning shares encoders or decoders across translation and image-captioning tasks to transfer knowledge.The related approach formulates translation and captioning together with a shared decoder.
  • Information Fusion: RFNet performs intermediate fusion by preprocessing representations from different CNN views through their relationships before supplying them to the target model.This differs from input concatenation and output ensembles, which combine representations or decoder results at other stages.

3 Background

The captioning background describes a CNN-RNN encoder-decoder pipeline: CNN features represent an image globally and regionally, while an attention-equipped LSTM translates them into a sentence.

  • Encoder: A pretrained image-classification CNN extracts both a global representation and representations for image subregions.The global representation usually comes from a fully connected layer, while subregion representations come from a convolutional layer.
  • Encoder: The global and subregion features are denoted a0 and A = {a1, . . . , ak}, where k is the number of subregions.These representations provide the visual inputs used by the decoder.
  • Decoder: A decoder based on a GRU or LSTM translates the image representations into a natural sentence.This paper uses an LSTM equipped with an attention mechanism.
  • Decoder: The attention-equipped LSTM uses its hidden state, current input, and context vector to produce recurrent outputs.The context vector is a weighted sum of subregion features, with weights determined by an attention model.
  • Caption Generation: Captioning generates C = (y1, y2, · · · , yN) for an image and models the conditional probability of the next word.At each step, the word embedding of the current word is used as the recurrent input, and a softmax produces the next-word distribution.

4 Our Method

RFNet fuses representations from multiple CNN encoders through two recurrent stages, producing compact thought vectors for the decoder. The stages enable component interactions, multi-attention review, and final information compression before caption generation.

  • Architecture: RFNet uses two fusion stages to combine multiple encoder representations into more representative thought vectors for the decoder.Stage I forms multiple thought-vector sets, while Stage II compresses them into one set.
  • Architecture: Multiple CNN encoders provide diverse image representations, which RFNet processes jointly rather than relying on a single encoder.The encoder representations include global and subregion representations extracted by each CNN.
  • Fusion Stage I: In fusion stage I, each review component receives the previous hidden states of all components, enabling communication and complementary-information extraction.Each component outputs hidden states as thought vectors, and the LSTM units can differ across components and time steps.
  • Fusion Stage II: Fusion stage II applies multiple independent attention models to stage-I thought vectors and combines them into a single thought-vector set.The resulting vectors are intended to provide more information for the decoder.
  • Decoder: The decoder inherits the final fusion-stage hidden state and memory cell, then translates the fused information into natural sentences with attention.The decoder uses the same encoder-decoder-style recurrent formulation for caption generation.
  • Discriminative Supervision: Discriminative supervision is combined with captioning loss, and its weight λ is selected to balance the two objectives.The complete loss includes both the discriminative supervision loss and captioning loss.

5 Experiments

Experiments evaluate RFNet on MSCOCO using multiple CNN encoders, standard captioning metrics, and ablations of its fusion design. RFNet generally outperforms competing models, with both fusion stages and first-stage interactions contributing to performance.

  • Dataset and setup: MSCOCO contains 82,783 training, 40,504 validation, and 40,775 test images, each paired with five human captions.The conventional Karpathy split instead uses 113,287 training, 5,000 validation, and 5,000 test images.
  • Dataset and setup: RFNet extracts five representation groups using ResNet, DenseNet, Inception-V3, Inception-V4, and Inception-ResNet-V2 encoders.Each group includes a global feature vector and subregion feature vectors.
  • Performance comparisons: RFNet significantly outperformed existing models except Up-Down on cross-entropy evaluation, while Up-Down used an encoder trained with additional Visual Genome information.RFNet exceeded Up-Down on METEOR, ROUGE-L, and SPICE but trailed it on BLEU-1, BLEU-4, and CIDEr.
  • Performance comparisons: RFNetΣ, an ensemble of four RFNets, significantly outperformed all compared ensemble models.A single RFNet also surpassed other ensemble models such as NIC and Att2in and was comparable to ReviewNetΣ.
  • Performance comparisons: 5.0 and 4.6 CIDEr points improved the C5 and C40 scores, respectively, over the Up-Down model under reinforcement-learning evaluation.RFNet also achieved the best performance in the reported online evaluation using an ensemble of seven models.
  • Ablation study: Both fusion stages and the interactions in fusion stage I are important for RFNet’s performance.The first stage improves thought vectors through interactions, while the second stage refines and compresses them into compact, informative vectors.

6 Conclusions

The paper concludes that RFNet fuses complementary representations from multiple image encoders through a two-stage recurrent procedure before decoding. It reports leading performance on the MSCOCO evaluation server.

  • Conclusion: RFNet exploits complementary information from multiple image representations for image captioning.The architecture inserts a recurrent fusion procedure between the encoders and decoder.
  • Conclusion: The first fusion stage incorporates complementary information into separate thought-vector sets, which the second stage compresses into decoder inputs.Each stage is treated as a special RNN.
  • Conclusion: RFNet achieved leading performance on the MSCOCO evaluation server.The conclusion presents this result as corroborating the effectiveness of the network architecture.

A.1 Qualitative Analysis

RFNet’s qualitative gains are mainly associated with recognizing salient objects that single-CNN ReviewNet models miss. Across examples, recurrent fusion combines multiple representations to identify these objects more effectively.

  • RFNet more accurately recognizes salient objects than ReviewNet variants using individual CNN encoders.Examples include children, a kite, a beach, zebra, giraffe, police officer, beds, and a lamp.
  • Recurrent fusion combines multiple representations to find salient objects neglected by individual CNN representations.
  • ReviewNet-ResNet, ReviewNet-DenseNet, and ReviewNet-Inception-V3 each miss or misidentify salient objects in the displayed examples.

A.2 Ablation Study on Different CNNs.

The ablation compares CNN encoder contributions using validation-set CIDEr scores and qualitative captions. ResNet contributes most and DenseNet least overall, while the proposed method produces captions that more consistently match the depicted content.

  • Ablation results: ResNet contributes the most to final image-captioning performance, whereas DenseNet contributes the least overall.The comparison uses CIDEr scores for different CNNs on the validation set.
  • Qualitative examples: Our method correctly captions children flying a kite, animals standing together, and a police officer on a motorcycle, while ReviewNet variants introduce object or identity errors.The corresponding references support children flying a kite, a zebra and giraffe together, and a police officer on a motorcycle.
  • Qualitative examples: Our method describes the hotel room with two beds and a lamp, while the ReviewNet variants differ in bed or lamp details.The ground-truth captions also emphasize two beds and a lamp.
  • Qualitative examples: Our method identifies a flock of birds flying over water, while all three ReviewNet variants reduce the scene to a single bird.The reference captions consistently describe multiple birds over water.
Loading 1807.09986v3…