Source-linked AI summary
Neural Baby Talk
Jiasen Lu, Jianwei Yang, Dhruv Batra, Devi Parikh
TL;DR
Image captioning models can generate fluent but weakly grounded descriptions, while detector-based slot filling is grounded but less natural. Neural Baby Talk combines both by generating region-linked templates and filling them with detected concepts; it achieves state-of-the-art results across the reported captioning settings.
Problem
Modern captioning can lack visual grounding, producing generic or inaccurate descriptions instead of associating named concepts with image pixels.
Method
Neural Baby Talk generates a hybrid template containing text words and region-linked visual slots, then fills those slots with concepts recognized by object detectors in an end-to-end architecture.
Results
The method achieves state-of-the-art performance on COCO and Flickr30k standard captioning and significantly outperforms existing methods on robust and novel-object captioning tasks.
Takeaways & Limitations
Grounding caption words in detected entities supports natural-language descriptions across standard, novel-object, and changed scene-composition settings.
Abstract
from arXiv · showhide
We introduce a novel framework for image captioning that can produce natural language explicitly grounded in entities that object detectors find in the image. Our approach reconciles classical slot filling approaches (that are generally better grounded in images) with modern neural captioning approaches (that are generally more natural sounding and accurate). Our approach first generates a sentence `template' with slot locations explicitly tied to specific image regions. These slots are then filled in by visual concepts identified in the regions by object detectors. The entire architecture (sentence template generation and slot filling with object detectors) is end-to-end differentiable. We verify the effectiveness of our proposed model on different image captioning tasks. On standard image captioning and novel object captioning, our model reaches state-of-the-art on both COCO and Flickr30k datasets. We also demonstrate that our model has unique advantages when the train and test distributions of scene compositions -- and hence language priors of associated captions -- are different. Code has been made available at: https://github.com/jiasenlu/NeuralBabyTalk
1. Introduction
Neural Baby Talk addresses weak visual grounding in modern captioning by combining natural-language generation with object-detector-grounded slots. It generates a hybrid sentence template tied to image regions, then fills those slots with detected concepts.
- Motivation: Modern neural captioning models can produce plausible but inaccurate descriptions because they often lack visual grounding and copy training captions.Their attention regions may not correspond to semantically meaningful entities, and language priors can favor generic scene descriptions.
- Motivation: Earlier slot-filling methods ground captions in detected objects and attributes but produce unnatural language.This motivates reconciling detector-based grounding with neural captioning’s fluent language generation.
- Approach: Neural Baby Talk generates a free-form sentence template whose slots are explicitly tied to image regions, then fills them using object detectors.The approach is end-to-end trained and allows different detection backends to produce varied captions for the same image.
- Approach: The decoder chooses at each time step between textual vocabulary words and visual words that reserve slots for region-specific concepts.For example, textual words express relations such as “sitting,” while visual slots can later become “puppy” or “table.”
- Evaluation: The framework is evaluated on standard, robust-composition, and novel-object captioning tasks, with state-of-the-art performance reported on COCO and Flickr30k.The robust task tests images containing known objects in combinations not observed during training.
2. Related Work
Prior image-captioning methods either use templated slot filling for grounding or neural generation for fluent language, while Neural Baby Talk grounds words in object detections. Its detector interface also supports novel-object captioning.
- Grounded and templated captioning: Earlier slot-filling systems detect objects, attributes, and relations before inserting them into templates, but their captions are often relevant rather than natural sounding.Some alternatives use structured scene meanings or syntactically richer templates.
- Neural captioning: Neural captioning models use CNN-RNN architectures and attention, but attention regions are often blurry and do not correspond to individual object instances.Neural systems may therefore rely on language correlations rather than precise entity grounding.
- Related grounding tasks: Dense Captioning grounds descriptions in specific regions, whereas Neural Baby Talk generates a caption for the entire image with words grounded in concrete entities.This distinguishes whole-image captioning from region-level description generation.
- Novel-object captioning: Neural Baby Talk can incorporate different object detectors, including detectors for objects never seen in the captioning training data.It outperforms prior novel-object captioning approaches by 14.6% on averaged F1 score.
3. Method
The method generates captions as hybrid templates containing textual words and region-linked slots, then fills those slots with detector-based visual words. Its neural decoder supports region grounding, textual generation, and caption refinement within an end-to-end training framework.
- 3. Method: The model decomposes caption generation into textual words and visual words, with each visual word grounded in a specific image region.A visual sentinel provides dummy grounding for textual words that are not tied to specific regions.
- 3. Method: It first generates a sentence template consisting of textual words interspersed with region slots, then fills the slots using object-detection information.The generated template is not restricted to a pre-specified template bank and may take exponentially many forms.
- 3.1. “Slotted” Caption Template Generation: An RNN decoder uses image features and a pointer network over detector regions to generate slots associated with candidate grounding regions.At training time it uses ground-truth tokens, while at test time it samples the previous token.
- 3.1. “Slotted” Caption Template Generation: The model includes a visual sentinel so the decoder can generate textual words from the language model without assigning them to a detected region.The sentinel represents information the decoder already knows about the image.
- 3.2. Caption Refinement: Filling in The Slots: Slot filling refines detector categories by predicting plurality and a possible fine-grained class, producing forms such as “puppies” from a category and region representation.The refinement uses two single-layer MLPs with ReLU activation and learned word embeddings for fine-grained categories.
- 3.3. Training Objective: Training minimizes a cross-entropy objective combining textual-word probability, caption refinement, and averaged target-region probability.Visual words are dynamically identified by lemmatizing caption words and matching them against a visual-word vocabulary; the objective can use strong or weak grounding supervision.
4. Experimental Results
Experiments evaluate Neural Baby Talk on standard, robust, and novel-object captioning tasks using COCO and Flickr30k, with comparisons to prior methods and qualitative grounding analysis.
- Qualitative Analysis: Figures visualize word-to-region grounding, with successful examples in the first three columns and grounding failures in the last column.For robust captioning, the examples include novel compositions such as cat-remote, man-bird, and dog-skateboard; one failure omits orange.
- Standard Image Captioning: Flickr30k results achieve state of the art on all automatic metrics, including gains of 2.0 BLEU4 and 4.4 CIDEr over Adaptive.With ground-truth proposals, NBToracle improves SPICE by 5.1, indicating further benefit from improved object detectors.
- Standard Image Captioning: On COCO, Neural Baby Talk outperforms the compared state-of-the-art methods on 4 out of 5 automatic evaluation metrics.The model also learns to ground visual words correctly under weak supervision and remains robust to erroneous detections.
- Robust Image Captioning: The robust-COCO split changes train-test object co-occurrence distributions to evaluate captioning under novel scene compositions.The split contains 110,234/3,915/9,138 train/val/test images.
- Robust Image Captioning: On robust-COCO, Neural Baby Talk outperforms previous methods on all metrics, exceeding Up-Down by 2.7 on the proposed metric.NBToracle improves the proposed metric by 3.3, while all models perform 2∼3 points worse than on Karpathy’s split in general.
- Novel Object Captioning: For novel object captioning, NBT∗+G captions novel concepts with minimal model changes, while constrained beam search improves performance across evaluation metrics.NBT†+T2 outperforms C-LSTM by 14.6% on average F1 scores, although greedy decoding is less likely to select small objects such as bottle and racket.
5. Conclusion
Neural Baby Talk combines natural-language generation with explicit grounding in detector-identified entities. Its evaluations cover standard, robust, and novel-object captioning tasks.
- Conclusion: Neural Baby Talk generates hybrid templates with image-region slots and fills those slots using categories recognized by object detectors.The framework is presented as a two-stage approach for producing natural language grounded in image entities.
- Conclusion: Experimental results on standard, robust, and novel object captioning tasks validate the effectiveness of the proposed approach.The conclusion states that the framework produces natural language explicitly grounded in entities found by object detectors.
6. Appendix: COCO Fine-grained Categories
The appendix describes a category-to-word mapping used to derive fine-grained COCO labels for visual words.
- COCO Fine-grained Categories: COCO category mappings associate object categories such as <Person> with potential fine-grained labels such as child, man, and baker.Caption entity words are lemmatized, and similar words are retrieved for each category class.