PaperlayerView exact revision on arXiv

Source-linked AI summary

Meshed-Memory Transformer for Image Captioning

Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, Rita Cucchiara

arXiv:1912.08226v2cs.CVcs.CL

TL;DR

Image captioning requires multimodal modeling, yet fully attentive architectures had been comparatively under-explored for this task. The paper introduces the M2 Transformer, combining memory-augmented multi-level image encoding with meshed decoding, and reports state-of-the-art COCO performance plus novel-object evaluation.

  • Problem

    Fully attentive architectures were state-of-the-art in sequence modeling, but their application to multimodal image captioning remained under-explored.

  • Method

    The M2 Transformer uses persistent memory for multi-level image-region relationships and learned meshed connectivity to combine low- and high-level visual features during decoding.

  • Results

    The model achieves a new state of the art on COCO, including the Karpathy test split and online test server, while also being evaluated for novel-object captioning.

  • Takeaways & Limitations

    The experiments support the model's memory-based region encoding and meshed connectivity through ablations and evaluate its use for describing novel objects.

Abstract

from arXiv · show

Transformer-based architectures represent the state of the art in sequence modeling tasks like machine translation and language understanding. Their applicability to multi-modal contexts like image captioning, however, is still largely under-explored. With the aim of filling this gap, we present M$^2$ - a Meshed Transformer with Memory for Image Captioning. The architecture improves both the image encoding and the language generation steps: it learns a multi-level representation of the relationships between image regions integrating learned a priori knowledge, and uses a mesh-like connectivity at decoding stage to exploit low- and high-level features. Experimentally, we investigate the performance of the M$^2$ Transformer and different fully-attentive models in comparison with recurrent ones. When tested on COCO, our proposal achieves a new state of the art in single-model and ensemble configurations on the "Karpathy" test split and on the online test server. We also assess its performances when describing objects unseen in the training set. Trained models and code for reproducing the experiments are publicly available at: https://github.com/aimagelab/meshed-memory-transformer.

1. Introduction

Image captioning has mainly relied on recurrent models, while fully attentive architectures remain comparatively under-explored for this multimodal task. The M2 Transformer addresses this gap with multi-level visual relationship encoding, learned memory, and meshed decoder connectivity, achieving leading COCO results.

  • Image captioning requires modeling relationships between visual and textual elements while generating a sequence of words.
  • Fully attentive models replace recurrent relations with self-attention, creating new architectural opportunities for image captioning.
  • The M2 Transformer encodes image-region relationships at multiple levels and incorporates learned a priori knowledge through persistent memory vectors.
  • Its decoder uses learned gating to connect low- and high-level visual relationships across layers in a mesh-like structure.
  • On COCO, the model achieves a new state of the art on the Karpathy test set for both single-model and ensemble configurations.
  • The work compares fully attentive architectures, evaluates novel-object captioning on nocaps, and releases code and trained models for reproducibility.

2. Related work

Earlier image-captioning systems commonly combined recurrent language models with CNN-derived visual features and attention mechanisms. Recent work introduced fully attentive and Transformer-based alternatives, but these approaches remained an emerging direction for captioning.

  • Earlier methods generated captions from templates filled by object-detector or attribute-predictor outputs.
  • Most deep-learning captioning systems used RNN language models conditioned on visual information from one or more CNN layers.
  • Training evolved from time-wise cross-entropy objectives toward reinforcement learning with non-differentiable caption metrics.
  • Visual encoding progressed from spatial attention over CNN grids to attention over extracted image regions.
  • Because RNNs have limited representation power and sequential computation, recent work investigated fully attentive paradigms for image captioning.
  • Transformer-based captioning extended the architecture with mechanisms such as geometric relations between detected object pairs.

3. Meshed-Memory Transformer

The M2 Transformer uses memory-augmented encoding to model multi-level visual relationships and meshed decoding to combine representations from all encoder layers during caption generation.

  • Architecture: The model stacks attentive encoder and decoder layers, replacing recurrence with scaled dot-product attention for image–text interactions.The decoder generates captions word by word from image-region encodings and previously generated words.
  • Memory-Augmented Encoder: Self-attention encodes pairwise relationships between image regions but cannot represent input-independent a priori knowledge.The limitation arises because attention weights depend solely on pairwise similarities among projected inputs.
  • Memory-Augmented Encoder: Memory-augmented attention adds learnable key and value slots, enabling retrieval of learned knowledge not already embedded in the image features.The additional memory vectors are independent of the input set and are updated through stochastic gradient descent.
  • Memory-Augmented Encoder: Stacked encoding layers produce multi-level image representations, allowing higher layers to refine relationships identified by earlier layers.Residual connections and layer normalization encapsulate the memory-augmented attention and position-wise feed-forward components.
  • Meshed Decoder: Meshed cross-attention connects decoder inputs to every encoding layer, then sums gated contributions from those layers.The gates modulate each layer’s contribution and their relative importance using relevance between cross-attention outputs and the decoder query.
  • Meshed Decoder: The decoder combines masked self-attention, meshed cross-attention, feed-forward processing, residual connections, and layer normalization to predict the next word.At time t, the output represents a prediction conditioned on the preceding sequence Y≤t; training uses crossentropy pre-training followed by reinforcement learning.

4. Experiments

Experiments evaluate M2 Transformer and Transformer-based alternatives on COCO, including standard, ensemble, online-server, and novel-object settings. Ablations show benefits from multi-level encoding, meshed connectivity, sigmoid gating, and memory vectors, while qualitative examples indicate more accurate and descriptive captions.

  • Datasets and evaluation: COCO experiments use Karpathy splits, the online test server, and standard BLEU, METEOR, ROUGE, CIDEr, and SPICE metrics.COCO includes more than 120,000 images with five captions each; the online test server contains 40,775 images without public annotations.
  • Transformer baselines: 123.6 CIDEr is achieved by a three-encoder, three-decoder Transformer, compared with 121.8 CIDEr for the original six-layer Transformer.The original Transformer slightly exceeds Up-Down at 121.8 versus 120.1 CIDEr.
  • Ablation study: 129.2 CIDEr versus 123.6 CIDEr shows that multi-level encoding improves over using only the last encoder layer.The meshed connectivity then increases performance from 129.2 CIDEr to 131.2 CIDEr, for a total 7.6-point gain over the standard Transformer.
  • Ablation study: Removing memory vectors reduces performance by around 1 CIDEr point in both one-to-one and meshed connectivity settings.This ablation supports using learned a priori knowledge when encoding image regions.
  • State-of-the-art comparisons: A four-model ensemble improves by 2.5 CIDEr points over the current state of the art, while online evaluation improves by 1.4 CIDEr points over the best performer.The four-model ensemble is best on all reported metrics in the ensemble comparison and surpasses the current state of the art on all online-server metrics.
  • Novel objects and qualitative analysis: The model surpasses Up-Down on both in-domain and out-of-domain novel-object images, while Constrained Beam Search particularly improves out-of-domain captioning.Qualitative examples also report more accurate and descriptive captions integrating fine-grained details and object relations.

5. Conclusion

The conclusion presents M2 Transformer as a Transformer-based image-captioning architecture combining memory-based region encoding with meshed encoder–decoder connectivity. Experiments report state-of-the-art COCO performance, ablation validation, and evaluation on novel objects.

  • 5. Conclusion: M2 Transformer combines memory-vector region encoding with meshed connectivity between encoding and decoding modules.The architecture uses a priori knowledge through memory vectors and connects encoding and decoding modules in a mesh.
  • 5. Conclusion: The model achieves a new state of the art on COCO and ranks first on the online leaderboard.
  • 5. Conclusion: Ablation studies validate the model components, while additional experiments assess performance when describing novel objects.

A. Supplementary material

The supplementary material adds implementation details, further experiments, and visualizations for the M2 Transformer.

  • A. Supplementary material: The supplementary material provides additional material about the M2 Transformer model.
  • A. Supplementary material: It includes additional training and implementation details.
  • A. Supplementary material: It also reports further experimental results and visualizations.

A.1. Additional implementation details

The implementation details describe efficient autoregressive decoding and preprocessing choices. Reusing decoder states reduces training and inference time substantially on the reported GPU.

  • Decoding optimization: Naive decoding recomputes attention states over all previous timesteps at every iteration.Intermediate results depending on wt are recomputed T−t times, with T equal to 20 in the experiments.
  • Decoding optimization: The implementation stores previously computed decoder keys and values and processes only the preceding generated word at each timestep.
  • Decoding optimization: A NVIDIA 2080Ti implementation reduced training and inference times by approximately a factor of 3.
  • Vocabulary and tokenization: Captions are lowercased, punctuation is removed, and spaCy tokenization is used for vocabulary construction.Words appearing fewer than 5 times in training are removed from the vocabulary.

A.2. Additional experimental results

Additional experiments examine memory-vector counts, encoder–decoder depth, and SPICE subcategories. The reported analyses select 40 memory vectors, favor three-layer symmetric stacks, and show gains in several semantic categories.

  • Memory vectors: 40 memory vectors produce the best BLEU, METEOR, ROUGE, and CIDEr results, while 80 provide slightly better SPICE.The main-paper experiments therefore use 40 memory vectors.
  • Encoder and decoder layers: Three encoding and three decoding layers yield the best performance.
  • Encoder and decoder layers: Non-symmetric encoder–decoder architectures do not produce significant performance improvements.
  • SPICE F-scores: The model significantly improves SPICE F-scores for objects, attributes, and relationships between objects.The comparison uses the Up-Down approach and a three-layer base Transformer on the Karpathy test split.

A.3. Qualitative results and visualization

Qualitative comparisons show that the proposed model produces more correct and detailed captions than the original Transformer, while attention visualizations expose its region-level attribution patterns.

  • The proposed model improves caption correctness and provides more detailed, exhaustive descriptions compared with the original Transformer.
  • Integrated Gradients assigns attribution scores to feature channels of each input region, which are averaged and renormalized for visualization.
  • The nocaps examples compare model captions with the baseline Transformer and display object-detector constraints used during generation.

A.4. Novel object captioning

On nocaps images, the M2 Transformer better incorporates object-detector constraints while maintaining fluent, proper sentences. Qualitative examples show more accurate recognition of objects, attributes, counts, and relations than the baseline Transformer.

  • The M2 Transformer better incorporates object-detector constraints while maintaining the fluency and properness of generated sentences.
  • The nocaps evaluation uses the top-3 detected Open Images objects as generation constraints after filtering classes and overlapping detections.
  • The object detector is identified as the TensorFlow Faster R-CNN Inception ResNet v2 Atrous OIDv2 model.
  • Additional examples show M2 recovering scene objects and relations that the Transformer misidentifies, including a van, truck, kites, suitcases, boat, and knife.
  • Compared with the Transformer, M2 captions more accurately describe objects and attributes, including a toaster oven, Santa hat, and three people.
  • The supplementary figures provide additional qualitative comparisons and attention-state visualizations for captions generated by the M2 Transformer.
Loading 1912.08226v2…