Source-linked AI summary
Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions
Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
TL;DR
Captioning models have limited external controllability and explainability, despite images admitting descriptions tailored to different goals and contexts. Show, Control and Tell conditions recurrent caption generation on region sequences or sets, explicitly grounding noun chunks to those regions. On Flickr30k Entities and COCO Entities, it achieves state-of-the-art controllable captioning results for caption quality and diversity.
Problem
Existing captioning architectures are difficult to control and explain externally, limiting their use when descriptions must follow task- or context-specific region priorities.
Method
Show, Control and Tell uses a recurrent architecture that predicts noun chunks conditioned on a sequence or set of image regions and explicitly grounds those chunks.
Results
The method achieves state-of-the-art controllable image captioning results on Flickr30k Entities and COCO Entities in caption quality and diversity.
Takeaways & Limitations
The framework supports diverse descriptions of the same image while following externally supplied regional control and grounding generated chunks to image regions.
Takeaways & Limitations
The annotation pipeline can produce region sets with multiple elements or empty noun-chunk region sets, which are filled during training or dropped during validation and testing.
Abstract
from arXiv · showhide
Current captioning approaches can describe images using black-box architectures whose behavior is hardly controllable and explainable from the exterior. As an image can be described in infinite ways depending on the goal and the context at hand, a higher degree of controllability is needed to apply captioning algorithms in complex scenarios. In this paper, we introduce a novel framework for image captioning which can generate diverse descriptions by allowing both grounding and controllability. Given a control signal in the form of a sequence or set of image regions, we generate the corresponding caption through a recurrent architecture which predicts textual chunks explicitly grounded on regions, following the constraints of the given control. Experiments are conducted on Flickr30k Entities and on COCO Entities, an extended version of COCO in which we add grounding annotations collected in a semi-automatic manner. Results demonstrate that our method achieves state of the art performances on controllable image captioning, in terms of caption quality and diversity. Code and annotations are publicly available at: https://github.com/aimagelab/show-control-and-tell.
1. Introduction
Existing captioning models can describe images, but their region selection and generation are difficult to control or explain externally. Show, Control and Tell addresses this gap by conditioning captions on image regions and grounding generated noun chunks to them.
- Image captioning models map images to language distributions, commonly using greedy decoding, beam search, or related sampling strategies.
- Attentive models select image regions during generation, but their region choices are implicit and cannot be externally supervised.
- This lack of control limits captioning in complex settings where systems must prioritize task-relevant regions and support interpretable behavior.
- Show, Control and Tell generates diverse captions conditioned on either a sequence or set of image regions, allowing different regions and orders to guide description.
- Its recurrent architecture decomposes sentences into noun chunks and explicitly grounds generated textual chunks on image regions.
- The framework is evaluated on Flickr30k Entities and COCO with grounding annotations, achieving state-of-the-art controllable captioning results in diversity and caption quality.
2. Related work
Prior work improved captioning through recurrent language models, visual attention, grounding, and diversity-oriented generation. These approaches generally lacked the combination of explicit region grounding and external controllability targeted by this paper.
- Image captioning systems commonly combine recurrent language models with CNN-based global or attention-derived visual representations.
- Attentive models provide an initial correspondence between words and image regions, but their grounding signals are often blurry and not semantically significant.
- DenseCap and Neural Baby Talk pursue more principled grounding, while this work decomposes captions into noun chunks and explicitly grounds each chunk to regions.
- Diversity methods can sample multiple captions or use GANs, but most improve diversity without controllability and may sacrifice accuracy.
3. Method
The method represents captions hierarchically as noun chunks grounded in image regions, then generates words and chunk transitions under a sequence or set of region-based controls.
- Sentence and region representation: Dependency trees group nouns with modifiers into noun chunks that can be mapped to corresponding image regions.Chunks may associate with multiple regions when more than one mapping exists.
- Controlled generation: The model generates captions that describe an ordered sequence of region sets while maintaining language fluency.The control input may contain multiple regions for a chunk and is handled generally as a sequence of region sets.
- Controlled generation: The recurrent architecture jointly predicts the next word and a boolean gate that switches between region sets.At test time, words are sampled from the word distribution, while the gate determines the next image region set.
- Chunk-level control: A chunk-shifting mechanism compares a latent chunk-end sentinel with current regions to decide when generation advances to the next region set.The LSTM-derived chunk sentinel represents memory at the end of a chunk, and the gate is sampled at test time.
- Word-level grounding: Adaptive attention combines current regions with a visual sentinel so the model can alternate between visual and textual words.The visual sentinel provides a fallback when the model does not attend a region, while the selected region set follows the chunk-shifting mechanism.
- Training objective: Training first uses cross-entropy for both output distributions and then reinforces caption quality and control alignment with a joint reward.The reward combines CIDEr-D with an alignment score based on Needleman-Wunsch matching.
4. Experiments
Experiments evaluate controllability through region sequences and sets, using grounding-aware datasets and metrics alongside standard captioning measures. The proposed method outperforms baselines on caption quality, alignment, IoU, and diversity across COCO Entities and Flickr30k Entities.
- Datasets: Flickr30k Entities provides manually linked caption chunks and image regions, while COCO receives semi-automatic grounding annotations.Flickr30k Entities contains 31,000 images, five sentences per image, and 276,000 manually annotated bounding boxes.
- Experimental setting: Sequence-controlled evaluation compares captions sharing the same region-sequence and image input, using standard captioning metrics and the NW alignment score.The experimental setting includes BLEU-4, METEOR, ROUGE, CIDEr, and SPICE, in addition to controllability metrics.
- Experimental setting: Set-controlled evaluation compares captions sharing the same region-set and image input, measuring control coverage with a soft noun-based IoU.The IoU uses an optimal assignment between ground-truth and predicted noun sets; intersection and set cardinality determine the final score.
- Quantitative results: On COCO Entities sequence control, the method achieves state-of-the-art results across automatic metrics, outperforming baselines in caption quality and control alignment.With CIDEr and NW optimization, the final NW score reaches 0.649 and exceeds the Controllable Up-Down baseline by 0.25.
- Quantitative results: On Flickr30k Entities sequence control, the method significantly outperforms compared approaches in both caption quality and alignment with the control signal.All controllable methods use CIDEr+NW optimization for this comparison.
- Quantitative results: For set control, the method outperforms baselines in IoU and captioning metrics, and it generates significantly diverse captions on the full COCO test split.The set-control results cover COCO Entities and Flickr30k Entities, while diversity is evaluated separately on COCO.
5. Conclusion
Show, Control and Tell generates controllable and grounded captions through regions by decomposing sentences into noun chunks and following a control signal. Experiments on Flickr30k and COCO Entities validate its effectiveness for controllability and diversity.
- The framework grounds noun chunks to image regions according to a control signal.
- Experiments on Flickr30k and COCO Entities validate the approach for controllability and diversity.
A. Sorting network
The sorting network converts an unsorted set of region sets into an ordered sequence for caption generation. It uses differentiable Sinkhorn normalization during learning and the Hungarian algorithm at test time, outperforming the compared ranking baselines.
- Sorting network: The network maps a set of region sets to a sorted version by producing a soft permutation matrix with the Sinkhorn operator.It concatenates per-region-set feature vectors into an N × N matrix before applying the operator.
- Sorting network: At test time, 20 normalization steps produce a soft permutation matrix that is converted into a discrete permutation with the Hungarian algorithm.
- Sorting network: The network is trained by minimizing mean square error between the scrambled input and its reconstructed sorted sequence.The reconstruction applies the transposed soft permutation matrix to the sorted ground-truth sequence.
- Experimental evaluation: Accuracy measures completely correct rankings, while Kendall’s Tau measures correlation between predicted and ground-truth rankings.The evaluation compares predefined local and global rankings, a Sinkhorn network, and an SVM Rank model.
- Experimental evaluation: The Sinkhorn network outperforms the other baselines and generates accurate rankings.
B. Training details
The captioning and sorting networks are trained with Adam using scheduled learning rates, while reinforcement learning for captioning uses a fixed lower rate. Training uses batches of 100, and decoding uses beam search.
- Training details: Adam training starts at 5 × 10^-4 and decreases the learning rate by 0.8 every epoch for both networks.
- Training details: Reinforcement learning for the captioning model uses a fixed learning rate of 5 × 10^-5.
- Training details: All experiments use a batch size of 100, and caption decoding uses beam search.
C. The COCO Entities dataset
COCO Entities extends COCO with semi-automatically collected grounding annotations linking noun chunks to image regions. The examples illustrate successful associations, multiple-region associations, and chunks without matching detections.
- The COCO Entities dataset: COCO Entities uses colors to show correspondences between textual chunks and image regions.Gray marks noun chunks without a visual grounding because of missing detections or noun-class association errors.
- The COCO Entities dataset: A noun chunk may correspond to multiple image regions, although visualizations display a single region per chunk.In such cases, the underlying region set contains more than one element.
- The COCO Entities dataset: For noun chunks without detections, training uses the image’s most probable detections and adaptive attention to learn the association.The authors report that this increases final network accuracy relative to feeding empty region sets.
- The COCO Entities dataset: Captions with missing noun-chunk associations are excluded from validation and testing.
D. Additional experimental results
Additional experiments evaluate controllability through region sequences and sets across Flickr30K and COCO Entities, with CIDEr+NW fine-tuning and comparisons against controllable baselines. The proposed model outperforms the baselines in both sequence- and set-controlled scenarios.
- Experimental setup: Tables 8–10 extend the evaluation to region-sequence and region-set controllability on Flickr30K and COCO Entities across training strategies.Table 8 covers sequence control on Flickr30K, while Tables 9 and 10 cover set control on COCO Entities and Flickr30K.
- Training strategies: CIDEr+NW fine-tuning is effective across all evaluated settings.
- Quantitative results: Our model outperforms the baselines by a clear margin under both sequence control and scrambled region-set control.The comparison holds across the carefully selected baselines.
- Baseline comparison: The Controllable LSTM baseline performs significantly below Controllable Up-Down, highlighting the importance of attention and control-signal representation.
E. Additional qualitative results
Additional qualitative results on COCO Entities show that the method generates multiple captions for the same image while following different region controls. The examples illustrate changes in described objects, relations, and ordering.
- Controllability: The method generates multiple captions for the same image when different control inputs are provided.
- Controllability: The examples show accurate adherence to control inputs across scenes involving vehicles, animals, skiing, and kite flying.Paired captions vary in described details such as a cat’s position, a dog’s frisbee, and the people flying a kite.