Source-linked AI summary
Dual-Level Collaborative Transformer for Image Captioning
Yunpeng Luo, Jiayi Ji, Xiaoshuai Sun, Liujuan Cao, Yongjian Wu, Feiyue Huang, Chia-Wen Lin, Rongrong Ji
TL;DR
Region features advance image captioning but lack contextual coverage and fine-grained details, whereas grid features provide those properties. DLCT combines both through geometry-aware self-attention and locality-constrained cross-attention, achieving new state-of-the-art results on MS-COCO.
Problem
Region features lack contextual information and fine-grained details that traditional grid features provide, motivating complementary use of both feature types.
Method
DLCT processes region and grid features with geometry-aware Dual-Way Self-Attention and uses Locality-Constrained Cross Attention guided by a geometric alignment graph.
Results
133.8% CIDEr scores on the Karpathy test set and 135.4% CIDEr scores on the online test achieve new state-of-the-art image-captioning performance.
Takeaways & Limitations
DLCT realizes complementary interaction between region and grid features while addressing semantic noise from their direct fusion.
Abstract
from arXiv · showhide
Descriptive region features extracted by object detection networks have played an important role in the recent advancements of image captioning. However, they are still criticized for the lack of contextual information and fine-grained details, which in contrast are the merits of traditional grid features. In this paper, we introduce a novel Dual-Level Collaborative Transformer (DLCT) network to realize the complementary advantages of the two features. Concretely, in DLCT, these two features are first processed by a novelDual-way Self Attenion (DWSA) to mine their intrinsic properties, where a Comprehensive Relation Attention component is also introduced to embed the geometric information. In addition, we propose a Locality-Constrained Cross Attention module to address the semantic noises caused by the direct fusion of these two features, where a geometric alignment graph is constructed to accurately align and reinforce region and grid features. To validate our model, we conduct extensive experiments on the highly competitive MS-COCO dataset, and achieve new state-of-the-art performance on both local and online test sets, i.e., 133.8% CIDEr-D on Karpathy split and 135.4% CIDEr on the official split. Code is available at https://github.com/luo3300612/image-captioning-DLCT.
Introduction
Image captioning benefits from both region and grid features, but combining them directly can introduce semantic noise. DLCT addresses this through geometry-aware processing and locality-constrained cross-feature interaction, achieving state-of-the-art MS-COCO results.
- Motivation: Region features provide object-level information but lack contextual coverage and fine-grained details that grid features retain.Detected regions may miss global scenes, while representing each region with one vector loses object details; grid features cover the image more completely.
- Motivation: Directly combining region and grid features can cause semantically incorrect attention interactions and may perform worse than using one feature source.Similar appearances can lead grids to attend to incorrect regions, hindering feature complementarity and degrading performance.
- Method: DLCT uses Dual-Way Self-Attention and Comprehensive Relation Attention to process both feature sources and embed absolute and relative geometric information.The two modules explore intrinsic feature properties while modeling geometric relationships.
- Method: Locality-Constrained Cross Attention uses a geometric alignment graph to align the two feature sources and reinforce each through cross-attention fusion.Region features transfer objectness to grids, while grid features supplement fine-grained details for regions.
- Results: 133.8% CIDEr scores on the Karpathy test set and 135.4% CIDEr scores on the online test establish new state-of-the-art image-captioning performance.The results were obtained through extensive experiments on MS-COCO.
Related Work
Image captioning research has used grid features, region features, and multimodal relation modeling, with prior approaches differing in how they represent visual relationships and location information.
- Visual Features: Earlier image-captioning methods used fixed-size CNN grid patches, while later methods introduced Faster R-CNN region-level features.Grid features represent image patches; region features provide object-level visual representations.
- Multimodal Modeling: HAN combines text, grid, and region features but does not integrate location information, coarsely models appearance relationships, and ignores semantic-noise filtering.Its relation module exploits relationships among diverse features but leaves these limitations unresolved.
- Location Modeling: GCN-LSTM and Object Relation Transformer model relative location relationships between regions.Their location modeling is based on bounding-box relations represented relatively.
Dual-Level Collaborative Transformer
The Dual-Level Collaborative Transformer uses both grid and region features to achieve their complementarity in image captioning.
- Model Overview: DLCT is an image-captioning model that uses grid and region features together to achieve feature complementarity.Its overall architecture is illustrated in Figure 2.
Integrating Position Information
Comprehensive Relation Attention combines absolute and relative positional information for grid and region features, using their geometric structure to guide attention.
- Integrating Position Information: Comprehensive Relation Attention integrates absolute and relative location information to model visual and positional relationships.
- Integrating Position Information: Absolute positional encoding represents where each grid or region feature is located, helping distinguish similar appearances at different positions.Grid positions use sine-cosine embeddings, while region positions embed four-dimensional bounding boxes.
- Integrating Position Information: Relative location information represents geometric relationships between visual features using box centers, widths, and heights.A grid is treated as a special case of a bounding box for this representation.
- Integrating Position Information: CRA modifies attention queries and keys with absolute positions, then adjusts attention weights using relative location information before normalization.
- Integrating Position Information: The geometric alignment graph connects intersecting regions and grids while retaining self-connected edges for every node.These connections restrict attention to geometrically aligned visual features.
Dual-Level Collaborative Encoder
The encoder separately models relationships within grid and region features, then uses locality-constrained cross-attention to align and reinforce them across levels.
- Dual-Level Collaborative Encoder: Given an image, the encoder extracts grid and region features and processes them with Dual-Way Self Attention and Locality-Constrained Cross Attention.
- Dual-Level Collaborative Encoder: Dual-Way Self Attention uses two independent self-attention modules to model intra-level relationships and enrich grid and region representations.Each feature type also uses an independent position-wise feedforward network.
- Dual-Level Collaborative Encoder: Locality-Constrained Cross Attention constructs a geometric alignment graph whose edges connect grid and region nodes when their bounding boxes intersect.The graph supports attention between source and target feature fields while filtering semantically unrelated features.
- Dual-Level Collaborative Encoder: Grid features attend to regions for high-level object information, while regions attend to grids for detailed and contextual information.
- Dual-Level Collaborative Encoder: Self-connected edges preserve nodes without aligned counterparts and allow the attention module not to attend to other features.LCCA outputs feed into DWSA, after which grid and region features are concatenated for decoder layers.
Objectives
The training objective uses cross-entropy pre-training followed by Self-Critical Sequence Training to optimize the non-differentiable CIDEr-D score.
- The model first optimizes cross-entropy loss against the ground-truth caption sequence.
- Self-Critical Sequence Training then optimizes the non-differentiable CIDEr-D score.
- The SCST formulation uses beam size k, CIDEr-D reward function r, and baseline b.
Experiments
Experiments use COCO with both offline Karpathy-split evaluation and online official-test evaluation, using Faster R-CNN-derived grid and region features.
- COCO contains 123,287 images, each annotated with five captions.
- The Karpathy split assigns 113,287, 5,000, and 5,000 images to training, validation, and testing, respectively.
- The study also uploads captions from COCO’s official testing set for online evaluation.
- Grid features are average-pooled to a 7×7 grid, while region features are extracted as 2048-d vectors.
- Training uses a 4-epoch warm-up, an 18-epoch XE pre-training stage, and subsequent CIDEr-reward optimization.
Performance Comparison
DLCT is evaluated against established image-captioning systems on offline and online COCO benchmarks, where it reports leading performance across the main comparisons.
- Offline evaluation: The offline comparison covers SCST, Up-Down, HAN, GCN-LSTM, SGAE, ORT, SRT, AoA, HIP, M2, and X-Transformer.
- Offline evaluation: DLCT surpasses X-Transformer by 1% CIDEr on the offline test split and leads most reported metrics for its single model.
- Online evaluation: The online leaderboard reports published state-of-the-art models with their backbone configurations.
Ablation Study
Ablations examine feature combinations, Comprehensive Relation Attention, Locality-Constrained Cross Attention, and qualitative attention behavior to assess DLCT’s design choices.
- Feature settings: Using both grid and region features with the proposed approach achieves substantially better results than either feature alone or their concatenation in Standard Transformer.
- CRA: Comprehensive Relation Attention improves models using either grid or region features and also benefits cooperation with LCCA.
- Qualitative analysis: DLCT attends to corresponding grids for words such as “blue,” “yellow,” and “tracks,” providing detailed and contextual visual information.
- LCCA: Removing LCCA or replacing its alignment graph with a complete bipartite graph produces worse performance.
Qualitative results and visualization
Qualitative examples and attention visualizations indicate that DLCT captures both contextual and fine-grained visual information, supporting more accurate and descriptive captions.
- DLCT generally captures detailed and contextual information to generate more accurate and descriptive captions.
- DLCT attends to corresponding grids for color words such as “blue” and “yellow,” beyond attending to corresponding regions.
- For “tracks,” grid attention provides a more fine-grained semantic segmentation, demonstrating the advantage of incorporating grid features.
Conclusion
DLCT combines region and grid features through geometry-aware intra-level and locality-constrained cross-level fusion. Extensive results show state-of-the-art performance on offline and online test splits.
- DLCT integrates region and grid appearance and geometry features through Comprehensive Relation Attention and Dual-Way Self Attention.
- A geometric alignment graph enables Locality-Constrained Cross Attention to reinforce both feature types and address semantic noise from direct fusion.
- DLCT achieves new state-of-the-art performance on both offline and online test splits.
- The authors plan to extend collaborative features to other multimedia areas requiring detailed and contextual information.