Source-linked AI summary
Length-Controllable Image Captioning
Chaorui Deng, Ning Ding, Mingkui Tan, Qi Wu
TL;DR
Image captioning methods generally do not let users choose whether to describe an image roughly or in detail. The paper adds length-level embeddings to captioning models and proposes non-autoregressive LaBERT. Its experiments report competitive or better performance across length levels, with LaBERT improving controllability and diversity over autoregressive baselines and decoding efficiency for long captions.
Problem
Most image captioning methods do not let users control whether an image is described roughly or in detail, despite caption length reflecting the information carried and a fidelity-efficiency trade-off.
Source: §1 · p. 2Method
The paper learns an embedding for each caption-length range and uses it in existing models, then proposes non-autoregressive LaBERT with length-level information and iterative refinement.
Source: §1 · pp. 2–3Results
The length-aware models report competitive or better performance across length levels, while LaBERT improves control precision and diversity over autoregressive baselines and decoding efficiency for long captions.
Source: §1 · p. 3Takeaways & Limitations
Length-level embeddings can add length control to existing captioning approaches, while LaBERT offers a non-autoregressive option for efficient long-caption decoding.
Source: §1 · p. 3, §5 · p. 14
Abstract
from arXiv · showhide
The last decade has witnessed remarkable progress in the image captioning task; however, most existing methods cannot control their captions, \emph{e.g.}, choosing to describe the image either roughly or in detail. In this paper, we propose to use a simple length level embedding to endow them with this ability. Moreover, due to their autoregressive nature, the computational complexity of existing models increases linearly as the length of the generated captions grows. Thus, we further devise a non-autoregressive image captioning approach that can generate captions in a length-irrelevant complexity. We verify the merit of the proposed length level embedding on three models: two state-of-the-art (SOTA) autoregressive models with different types of decoder, as well as our proposed non-autoregressive model, to show its generalization ability. In the experiments, our length-controllable image captioning models not only achieve SOTA performance on the challenging MS COCO dataset but also generate length-controllable and diverse image captions. Specifically, our non-autoregressive model outperforms the autoregressive baselines in terms of controllability and diversity, and also significantly improves the decoding efficiency for long captions. Our code and models are released at \textcolor{magenta}{\texttt{https://github.com/bearcatt/LaBERT}}.
1 Introduction
The paper addresses the lack of control over whether image captions describe an image briefly or in detail. It introduces length-level embeddings for existing captioning models and LaBERT, a non-autoregressive model; experiments report competitive or better performance alongside improved control precision, diversity, and long-caption decoding efficiency.
1 Introduction
- Existing methods generally lack control over how roughly or in detail they describe an image, although caption length reflects information content and entails a fidelity-efficiency trade-off.The paper notes that longer captions can carry more image information, while shorter captions can be delivered more efficiently with some information loss.Source: §1 · p. 2
- Length-level embeddings let existing captioning approaches generate captions within corresponding length ranges, and the paper evaluates the signal across LSTM- and Transformer-based autoregressive models and a non-autoregressive model.The embedding for each level is learned from training data within its length range.Source: §1 · pp. 2–3
- LaBERT combines length-level information with non-autoregressive decoding to generate captions in a fixed number of refinement steps regardless of caption length.The paper presents this as addressing the linear growth in autoregressive decoding complexity as caption length increases.Source: §1 · p. 3
- LaBERT achieves competitive or better performance across length levels while improving control precision, caption diversity, and long-caption decoding efficiency over autoregressive baselines.The length-aware AoANet and VLP also outperform their original versions in CIDEr-D on the normal length level and achieve remarkable SPICE scores on longer levels.Source: §1 · p. 3
2 Background and Related Works
The background describes autoregressive captioning as sequential token generation, reviews prior approaches to diversity and controllability, and outlines non-autoregressive generation and iterative refinement.
2.1 Autoregressive Image Captioning (AIC)
- Autoregressive image captioning factors a caption into left-to-right token probabilities conditioned on earlier tokens and the image, using teacher forcing in training and sequential decoding at inference.Inference starts with [BOS] and continues until [EOS] is predicted.Source: §2.1 · p. 4
2.2 Diverse and Controllable Image captioning
- Prior controllable or diverse captioning approaches use scene graphs, region-specific captions, GANs, or style data, with cited constraints including extra supervision, reduced accuracy, or scarce annotations.Earlier length-control strategies for text generation also face trade-offs in sentence completion, diversity, flexibility, or model applicability.Source: §2.2 · pp. 4–5
2.3 Non-autoregressive Text Generation
- Non-autoregressive text generation predicts a sequence in one decoder pass, while iterative-refinement methods use multiple passes to address token-dependency challenges.Approaches determine output length either with a length predictor or insertion/deletion modules.Source: §2.3 · p. 5
3 Method
The method adds a length-level signal to image-caption tokens, then uses LaBERT to generate captions within a chosen length range through parallel iterative refinement. The design applies length control to existing autoregressive models and uses a non-autoregressive model to reduce decoding complexity for long captions.
3.1 Acquisition of Length Information
- A length-level embedding is added to word embeddings, and optionally positional embeddings, to make token representations length-aware.Captions are assigned to a length level by their length; during inference, the chosen level acts as a control signal for the corresponding length range.Source: §3.1 · p. 6
- Length-level boundaries are chosen to provide enough training data per level while keeping ranges broad enough to retain caption flexibility.The experiments explore plans with four and five length levels.Source: §3.1 · p. 6
3.2 Non-autoregressive Length-controllable Decoding
- LaBERT combines image-region and caption-word representations in a shared space, processes them through Transformer blocks, and predicts masked tokens from their final hidden states.For image regions, the representation combines visual and location embeddings with a learnable embedding that distinguishes image regions from text tokens.Source: Figure 2 · p. 7, §3.2 · pp. 7–8
3.1 Acquisition of Length Information
- The length-level embedding can be integrated into both LSTM-based AoANet and Transformer-based VLP by adding it to their input word embeddings.The paper describes these integrations without other model modifications.Source: §3.1 · p. 7
Training
- During training, LaBERT randomly masks caption tokens and predicts them from image-region and unmasked-token embeddings, allowing masked-position predictions to be calculated in parallel.The training objective is cross-entropy over masked positions.Source: Training · p. 8
Inference
- At inference, LaBERT initializes a caption at the upper length bound, predicts masked positions in parallel, and iteratively remasks lower-confidence tokens for refinement.It decays the [EOS] probability after the lower length bound to encourage longer captions, and the number of masks decreases across steps.Source: Inference · pp. 8–9
- LaBERT reduces decoding complexity from Θ(Lhigh) in autoregressive methods to Θ(T), where T may be smaller than Lhigh.Its global update rule also allows early-step mistakes to be revised, and the model can find an end position within the selected range without a separate length predictor.Source: Inference · p. 9
4 Experiments
Experiments evaluate length-aware captioning on MS COCO using autoregressive AoANet and VLP and non-autoregressive LaBERT, assessing caption quality, controllability, diversity, and decoding efficiency.
4 Experiments
- The experiments assess the length-level embedding on AoANet, VLP, and LaBERT for caption quality, controllability, diversity, and decoding efficiency.Source: §4 · pp. 9–10
4.1 Dataset and Metrics
- On MS COCO Karpathy’s split, 113,287/5,000/5,000 images are used for training, validation, and testing; SPICE is robust to candidate length and prioritized for long captions.The evaluation also uses BLEU, ROUGE, METEOR, and CIDEr-D.Source: §4.1 · p. 10
4.2 Implementation Details
- LaBERT starts from BERT-base and uses 100 Faster R-CNN region proposals; training runs for 100,000 iterations with batch size 256.For levels 2 and 3, the [EOS] decay factor γ is set to 0.88 and 0.95, respectively.Source: §4.2 · p. 10
4.3 Performance on AoANet and VLP
- On 10–14-token captions, 4-level VLP improves CIDEr-D by 1.8% over original VLP; 5-level AoANet reaches 23.0 SPICE, 1.7% above its original result.Table 1 reports percentage scores for length-aware AoANet and VLP on the MS COCO Karpathy test split.Source: Table 1 · p. 11, §4.3 · p. 11
4.4 Performance on LaBERT
- Against single-level LaBERT, 4-level LaBERT performs better on every metric at level 2 and scores higher on SPICE at levels 3 and 4; it is comparable to autoregressive models at matched complexity.The comparison uses roughly matched decoding complexity.Source: Table 2 · p. 12, §4.4 · p. 12
4.5 Performance Analysis of LaBERT
- Reducing LaBERT refinement from 25 to 15 steps yields 1.67× speedup with 0.2% SPICE degradation; 10–20 steps remain competitive with 25-step LaBERT and 4-level VLP.Table 3 defines speedup relative to an autoregressive baseline.Source: Table 3 · p. 13, §4.5 · p. 13
- Removing LaBERT’s global update rule or [EOS] decay reduces second-level CIDEr-D by 1.6% and 2.2%, respectively.Source: §4.5 · p. 13
4.6 Controllability and Diversity Analysis
- LaBERT exceeds 95% control precision across levels; length-aware models distribute captions uniformly, and LaBERT outperforms autoregressive methods on all three diversity metrics.The compared diversity metrics are Div-1, Div-2, and SelfCIDEr.Source: §4.6 · pp. 13–14, Figure 4 · p. 14, Table 4 · p. 14
5 Conclusion
The conclusion presents length-level embeddings as a way to control caption length in existing image-captioning methods. It also proposes LaBERT, a non-autoregressive model designed for length-irrelevant decoding complexity.
5 Conclusion
- LaBERT generates image captions with length-irrelevant complexity to improve decoding efficiency for long captions.The conclusion reports that experiments demonstrate the method's effectiveness.Source: §5 · p. 14
- Adding a length-level embedding to input-token word embeddings gives existing image-captioning methods control over prediction length.Source: §5 · p. 14
- The paper proposes length-level embeddings for length-controllable image captioning.Source: §5 · p. 14