Meshed-Memory Transformer for Image Captioning

1. Introduction

• Image captioning has typically used recurrent models, while fully attentive self-attention offers new modeling capabilities but requires architectures tailored to the task’s multimodal nature.Earlier recurrent approaches often added attention or graph-like structures to model relationships among image regions, words, and tags.Source: §1 · p. 1
• M2 Transformer encodes image-region relationships at multiple levels using persistent memory vectors, then uses learned gating to connect visual encoder and language decoder layers in a mesh.The gated connections let each decoder stage exploit both low- and high-level visual relationships.Source: §1 · pp. 1–2, Figure 1 · p. 1
• On COCO, M2 Transformer achieves a new state of the art on the Karpathy test set in single-model and ensemble configurations and ranks first among published algorithms on the online server.The authors also describe it as surpassing all previous proposals on the online COCO evaluation server.Source: §1 · p. 2
• The study compares fully attentive image-captioning architectures and tests M2 Transformer on novel-object captioning with the nocaps dataset.The authors also state that they will publicly release the source code and trained models to support reproducibility and further research.Source: §1 · p. 2

2. Related work

• Earlier systems filled caption templates from detector or attribute outputs, while later neural methods used RNN language models conditioned on CNN visual features [33] [9] [16].Source: §2 · p. 2
• Training evolved from time-wise cross-entropy to reinforcement learning with non-differentiable caption metrics, while image encoding advanced from grid attention to object-region and graph-based relationship modeling [33] [31] [25] [45] [26] [50] [4] [29] [27] [48] [46].Source: §2 · p. 2
• RNN models are described as limited by representation power and sequential processing; convolutional captioning and fully attentive paradigms followed, and recent work began applying Transformers to captioning [5] [39] [8] [36].Source: §2 · p. 2
• Prior captioning Transformers incorporated object-pair geometry [13] or external semantic tags, while another approach gated attended information and paired a Transformer-like encoder with an LSTM decoder [14].Source: §2 · p. 2
• This paper instead proposes a fully attentive model that changes both image encoding and language decoding through two attention operators and redesigned encoder–decoder connectivity.Source: §2 · p. 2

3.1. Memory-Augmented Encoder

• Self-attention encodes pairwise region relationships but cannot supply prior knowledge, so learnable key/value memory slots add input-independent knowledge; stacked layers refine multilevel region encodings.Source: §3.1 · pp. 3–4

3.2. Meshed Decoder

• Gated meshed cross-attention combines weighted contributions from every encoding layer, while masked self-attention and feed-forward blocks support autoregressive token generation.The gates modulate each layer’s contribution and its relative importance; the decoder conditions each step on previously generated words.Source: §3.2 · p. 4

3.3. Training details

• Following [31] [33] [4], training pretrains with word-level cross-entropy and fine-tunes by self-critical sequence training [33], using beam-search samples [4] and CIDEr-D rewards [4] [40].The reward baseline is the mean reward of sampled sequences rather than greedy decoding; final prediction selects the highest-probability sequence in the last beam.Source: §3.3 · p. 5

4.1. Datasets

• Evaluation uses COCO’s Karpathy split and online test server, plus nocaps validation and test images grouped as in-domain, near-domain, or out-of-domain.COCO has more than 120,000 images with five captions each; nocaps has 15,100 images with 11 captions each, and the model is trained on COCO for nocaps evaluation.Source: §4.1 · p. 5

4.2. Experimental settings

• The experiments use standard captioning metrics and Faster R-CNN region features, with one-hot word embeddings and sinusoidal positional encodings for COCO.For nocaps, words instead use GloVe embeddings, with fully connected layers mapping to and from the model dimension and output probabilities tied to the embeddings.Source: §4.2 · p. 5

4.3. Ablation study

• On COCO, three encoding and three decoding layers reach 123.6 CIDEr, and meshed connectivity raises performance from 129.2 to 131.2 CIDEr.The original six-layer Transformer reaches 121.8 CIDEr, while adding AoA to a three-layer Transformer yields 129.1 CIDEr; softmax gating and removing memory each reduce performance by around 1 CIDEr point. The full model improves by 7.6 points over the standard Transformer, and also outperforms AoA.Source: Table 1 · p. 6, §4.3 · pp. 6–7

4.4. Comparison with state of the art

• On the Karpathy test split, the single model leads in BLEU-4, METEOR, and CIDEr, while the four-model ensemble leads on all metrics with a 2.5-point CIDEr gain.The single model is competitive on BLEU-1 and SPICE and slightly behind AoANet on ROUGE; the ensemble’s CIDEr gain is over the current state of the art.Source: Table 2 · p. 6, Table 3 · p. 6, §4.4 · p. 7
• On the COCO online test server, the four-model ensemble surpasses the state of the art on all metrics, including a 1.4-point CIDEr lead over the best performer.The ensemble was trained on the Karpathy training split and compared with other ensemble configurations.Source: Table 4 · p. 7, §4.4 · p. 7

4.5. Describing novel objects

• On nocaps, M² surpasses Up-Down on both in-domain and out-of-domain images, and constrained beam search particularly improves out-of-domain captioning.The comparison uses GloVe embeddings and constrained beam search to handle out-of-vocabulary words and enforce detected object categories.Source: Table 5 · p. 8, §4.5 · p. 8

4.6. Qualitative results and visualization

• On average, M² generates more accurate, descriptive captions than the original Transformer, incorporating fine-grained details and object relations.Integrated Gradients visualizations show that the model grounds words to image regions, including object details and small detections.Source: §4.6 · p. 8, Figure 4 · p. 8

• The model achieves a new state of the art on COCO and ranks first on the online leaderboard.Source: §5 · p. 8
• M2 Transformer uses memory vectors to incorporate a priori knowledge into region encoding and connects encoding and decoding modules through a meshed pattern.The paper describes this connectivity pattern as unprecedented among fully-attentive architectures.Source: §5 · p. 8
• The authors validate the model’s components through ablation studies and evaluate its performance when describing novel objects.Source: §5 · p. 8

A. Supplementary material

• The supplement provides additional training and implementation details for the M2 Transformer.Source: A. Supplementary material · p. 11
• It presents further experimental results for the model.Source: A. Supplementary material · p. 11
• It also includes visualizations.Source: A. Supplementary material · p. 11

A.1. Additional implementation details

• At each decoding timestep, each attentive layer reuses stored keys and values and computes queries, keys, and values only for the previous word.This avoids recomputing intermediate results as the generated sequence grows; the naive approach repeats computations T − t times, with T equal to 20 in the experiments.Source: §A.1 · p. 11
• Approximately a factor of 3 reduction in training and inference times was observed on an NVIDIA 2080Ti GPU.The implementation uses PyTorch module buffers to hold previously computed results.Source: §A.1 · p. 11
• Captions are lowercased, stripped of punctuation, tokenized with spaCy, and words appearing fewer than 5 times in training and validation are removed from vocabularies.Each image uses at most 50 region feature vectors.Source: §A.1 · p. 11
• The model uses 8 attentive heads with query, key, and value size d/8 = 64 per head; attention weights use Glorot initialization, and feed-forward weights use [11].All biases are initialized to 0; memory key and value vectors use zero-mean normal distributions with variances 1/dk and 1/m, respectively.Source: §A.1 · p. 11

Memory vectors

• 40 memory vectors give the best BLEU, METEOR, ROUGE, and CIDEr results, while 80 give a slightly superior SPICE result.The main-paper experiments therefore use 40 memory vectors.Source: Table 6 · p. 11, §A.2 · p. 11

Encoder and decoder layers

• Three encoding and three decoding layers yield the best performance; non-symmetric encoder-decoder architectures provide no significant improvement.This confirms the initial findings on the base Transformer model.Source: Table 7 · p. 11, §A.2 · p. 11

SPICE F-scores

• M2 Transformer significantly improves SPICE F-scores for identifying objects, attributes, and relationships compared with Up-Down [4] and the three-layer base Transformer.The breakdown is reported on the “Karpathy” test split.Source: Table 8 · p. 11, §A.2 · p. 11

A.3. Qualitative results and visualization

• Figure 6 shows that the proposed model’s captions are, on average, more correct and detailed than captions from the original Transformer.Figure 5 also presents nocaps images, predicted captions from both models, and the detected object classes used as generation constraints.Source: §A.3 · p. 11, Figure 5 · p. 12
• Figures 7 and 8 visualize attentive states by averaging Integrated Gradients attribution scores across each region’s feature channels, renormalizing them, and contrast-stretching scores to 0–1.The attribution approach follows Sec. 4.6 of the main paper [38].Source: §A.3 · p. 11

A.4. Novel object captioning

• On nocaps images, the M2 Transformer better incorporates object-detector constraints than the baseline Transformer while maintaining fluency and properness.Figure 5 reports sample captions and the constraints provided by the detector.Source: §A.4 · p. 12
• The method uses the top three filtered Open Images detections as constraints and selects the highest-probability decoded caption satisfying them.It removes 39 classes, overlapping higher-order detections, and does not include plural forms or alternative phrases for object classes.Source: §A.4 · p. 12
• Figures 6–8 show additional caption comparisons with ground truths and attention states highlighting the maximally attributed image region for each generated word.Figure 6 compares the approach and original Transformer; Figures 7 and 8 visualize M2 Transformer attention.Source: Figure 6 · p. 13, Figure 7 · p. 14, Figure 8 · p. 15