Towards Local Visual Modeling for Image Captioning

• Region features can omit background context and object relationships, while flattened grid features can split objects across neighboring grids.The paper frames local grid modeling as a way to address these complementary shortcomings without processing both feature types.Source: §1 · pp. 2–3
• LSA enhances intra-layer interaction between each grid and its local neighbors to recover fine-grained object information.Its multi-branch perception module can be re-parameterized as a single branch during inference to reduce additional overhead.Source: §1 · pp. 3–4
• LSF aligns and fuses grid features from different encoder layers to provide cross-layer semantic complementarity.Source: §1 · pp. 3–4
• LSTNet outperforms previous state-of-the-art models on MS-COCO, with reported offline and online scores of 134.8 and 136.3 CIDEr, respectively.Its generalization is also verified on Flickr8k and Flickr30k.Source: §1 · pp. 3–4

2.1. Image Captioning

• Image-captioning models include CNN-RNN systems and Transformer-based systems, with prior work exploring attention, region-grid collaboration, and visual relationships.Source: §2.1 · p. 5

2.2. Region features & Grid features

• Captioning features progressed from CNN grids to detector regions and back to detector-map grids; LSTNet models neighboring grid relationships while retaining context beyond bounding boxes.Compared with DLCT's use of both region and grid features, LSTNet runs over three times faster during cross-entropy training.Source: §2.2 · pp. 6–7

2.3. Multi-head Self-Attention in Transformer

• Transformer multi-head self-attention models relationships and context across input positions by combining multiple attention heads.Source: §2.3 · p. 7

3.1. Overview

• Each encoder layer applies MSA, LSA, then FFN; LSF aggregates features from encoder layers for the decoder to predict caption words.Source: §3.1 · pp. 8–9

3.2. Locality-Sensitive Attention (LSA)

• LSA reshapes post-MSA grids and applies multi-scale 2D CNNs to produce a sigmoid attention map that reweights self-attention features.Each CNN uses identity, 1 × 1, and sequential 1 × 1 and 3 × 3 convolution branches; reparameterization combines branches into one during inference without performance loss.Source: §3.2 · pp. 10–11

3.3. Locality-Sensitive Fusion (LSF)

• LSF fuses encoder-layer semantics; spatial shifts align neighboring grids before an MLP interacts across spatial and channel domains.The top-layer feature is unshifted, and the fused features are combined with it through a residual connection.Source: §3.3 · pp. 11–13

3.4. Objectives

• Training first uses cross-entropy on ground-truth captions, then SCST optimizes CIDEr and BLEU-4 using their summed reward.Source: §3.4 · pp. 13–14

4.1. Datasets

• MS-COCO contains 123,287 images, and the offline split uses 5,000 images each for validation and testing, with the remainder for training.The official testing set is also evaluated online.Source: §4.1 · p. 14

4.2. Implementation Details

• Faster R-CNN C5 feature maps are average-pooled to 7 × 7; LSTNet uses d_model 512, 8 heads, FFN expansion rate 4, and beam size 5.Only simple augmentations, such as RandomCrop and RandomRotation, are used.Source: §4.2 · p. 14
• Training uses Adam with 18 epochs of cross-entropy pre-training followed by 42 epochs of SCST.The learning rates are scheduled during both stages.Source: §4.2 · pp. 14–15

4.3. Performance Comparison

• The evaluation compares methods using BLEU-N, METEOR, ROUGE-L, CIDEr, and SPICE.Online comparisons include c5 and c40 reference-caption settings.Source: §4.3 · p. 15, §4.3.1 · p. 15

4.3.1. Online Evaluation

• On the online COCO test server, LSTNet outperforms other state-of-the-art methods across all reported metrics.With ResNeXt-101, it also performs better than RSTNet with ResNeXt-152 and X-Transformer with SENet-154 on most metrics; results cover c5 and c40 settings.Source: §4.3.1 · p. 15

4.3.2. Offline Evaluation

• 134.8% CIDEr on the offline COCO Karpathy test split exceeds the strongest competitor DLCT by 1.0%.DLCT adopts both region and grid features; LSTNet outperforms state-of-the-art models on most metrics.Source: §4.3.2 · p. 15
• The authors attribute grid-feature advantages to retaining spatial context and avoiding errors from object-detector detections, while LSTNet addresses objects split across grids.They contrast this with region features, which may omit background context and object relationships.Source: §4.3.2 · pp. 15–16

4.3.3. Fair Comparisons with SOTA Methods

• With the same grid features, LSTNet still achieves superior performance on all metrics compared with the other methods.The comparison controls for differences in visual features.Source: §4.3.3 · p. 16, Table 3 · p. 16

4.4.1. Effect of Different Branches of LSA

• The complete three-branch LSA performs best, while adding branches generally improves results and reparameterization avoids added inference overhead.The authors associate multiple branches with multi-scale modeling of objects of different sizes.Source: §4.4.1 · p. 17, Table 4 · p. 20

4.4.2. Effect of Different Arrangements of LSA and SA

• Sequential SA-LSA performs better than sequential LSA-SA or parallel SA and LSA.The authors suggest LSA refines the coarse-grained features processed by SA.Source: §4.4.2 · p. 17, Table 5 · p. 20

4.4.3. Effect of Different Shift Distances of LSF

• Shifted LSF outperforms unshifted LSF, and a shift distance of 1 performs best; larger shifts are associated with lower performance.The authors attribute the decline beyond 1 to increased long-distance interaction and reduced local modeling.Source: §4.4.3 · pp. 17–18

4.4.4. Effect of λ in LSA

• A weighting factor λ = 0.2 performs well on most metrics, while too large a λ degrades performance.Source: §4.4.4 · p. 18

4.4.5. Effect of Decoupling LSA and LSF

• Removing both LSA and LSF lowers BLEU-4 by 1.4% and CIDEr by 3.5%; their individual CIDEr gains are 2.3% for LSA and 2.4% for LSF.The authors report that the modules promote each other.Source: §4.4.5 · pp. 18–19

4.4.6. Effect of Different Approaches to Fuse Features

• Feature fusion improves performance over no fusion, and LSF outperforms the other tested fusion modules.The authors attribute the benefit of fusion to complementary semantic information across layers.Source: §4.4.6 · p. 19

4.4.7. Effect of Different Grid Sizes

• Captioning performance gradually improves as grid size increases.The authors explain this trend by the finer-grained and richer semantic information in larger image features.Source: §4.4.7 · pp. 19–20

• Paired two-tailed t-tests compare LSTNet with a standard Transformer on conventional caption metrics and SPICE semantic subcategories.The analysis uses matched test-set data and evaluates significance at α = 0.05.Source: §4.5 · pp. 20–21
• All conventional metrics improve significantly at α = 0.05, and all SPICE subcategories improve; Attribute, Size, Color, and Object show significant gains.The Relation improvement is relatively insignificant.Source: §4.5 · p. 21
• Local visual modeling improves single-object semantic metrics significantly, but relation capture also requires global modeling.Source: §4.5 · pp. 21–22

4.6. Qualitative Analysis

• LSTNet captions are more accurate and distinguishable than Transformer captions on the same grid features.The authors attribute the difference to LSA and LSF helping recognize visual objects through local modeling.Source: §4.6 · p. 22
• Tables 11 and 12 report captioning-metric comparisons and SPICE subcategories, with paired two-tailed t-tests and a 0.05 significance threshold.The supplied captions do not state the table values or outcomes.Source: Table 11 · p. 23, Table 12 · p. 23
• In Fig. 7, LSTNet attends to the man and phone or the boy and teddy bear, while Transformer attends to less relevant areas and misses objects.The resulting LSTNet captions identify “A man holding a cell phone” and “a young boy holding a teddy bear” correctly.Source: §4.6 · pp. 22, 24

4.7. Attention Visualization

• LSTNet focuses on correct grids for informative words such as “chair” and “umbrella,” while Transformer attends to irrelevant regions and produces erroneous descriptions.The visualization averages attention weights from the eight heads in the last decoder layer.Source: §4.7 · p. 24
• Table 14 compares Flickr30k captioning methods across BLEU-N, METEOR, ROUGE-L, and CIDEr; daggered values denote ensemble results.The caption identifies the comparison scope but does not provide the results.Source: Table 14 · p. 25
• The local interaction and fusion provide the decoder with richer, finer-grained visual features.This is the stated basis for the grid-level visualization’s comparison.Source: §4.7 · p. 25

4.8. Generalization on the Flickr Datasets

• LSTNet outperforms previous state-of-the-art methods on both Flickr8k and Flickr30k.The performance comparisons are reported in Tables 13 and 14.Source: §4.8 · p. 25
• The authors report extensive experiments on Flickr8k and Flickr30k to evaluate LSTNet’s generalization.The passage describes the performance margin as significant but supplies no metric values.Source: §4.8 · p. 25
• The authors conclude that these results verify LSTNet’s strong generalization to other datasets.The stated scope of this conclusion is the Flickr8k and Flickr30k evaluations.Source: §4.8 · p. 25

5. Conclusion

• LSTNet uses local interaction and fusion with grid features to improve object recognition in image captioning.LSA models relationships within an encoder layer to recover object semantics and refine visual details; LSF fuses features from different layers through relationships between adjacent grids.Source: §5 · pp. 25–26
• LSTNet has performance advantages over previous state-of-the-art models on MS-COCO.The conclusion also reports that ablations and visualizations reveal the effectiveness and insights of its individual components.Source: §5 · pp. 25–26
• Experiments on Flickr8k and Flickr30k verify LSTNet’s generalization to other datasets.The conclusion reports this as an additional result beyond MS-COCO.Source: §5 · p. 26
• LSA and LSF add computation and parameters, but the conclusion describes these additions as negligible compared with vanilla Transformer self-attention.The added computation and parameters accompany improved object-location capability and captioning performance.Source: §5 · p. 26