Source-linked AI summary
Rich Image Captioning in the Wild
Kenneth Tran, Xiaodong He, Lei Zhang, Jian Sun, Cornelia Carapcea, Chris Thrasher, Chris Buehler, Chris Sienkiewicz
TL;DR
Open-domain image captioning remains uncertain beyond controlled benchmarks, and automatic metrics may not reflect human judgments. The paper combines broad visual-concept detection, entity recognition, and confidence estimation in a captioning system. Human evaluations show significant improvements over a previous state-of-the-art system in-domain and out-of-domain, including a 94.9% relative improvement in Instagram human satisfaction.
Problem
Prior captioning systems were mainly evaluated on controlled benchmarks, leaving open-domain performance unclear, while automatic metrics can diverge from human judgments.
Source: §1 · pp. 1–2Method
Results
Human evaluation found significant outperformance over the previous state of the art on MS COCO and out-of-domain datasets, with a 94.9% relative improvement in Instagram human satisfaction.
Source: §1 · p. 2Takeaways & Limitations
Entity-integrated captions can provide richer information even when both the baseline and system outputs receive the same Excellent rating.
Source: §3 · p. 5
Abstract
from arXiv · showhide
We present an image caption system that addresses new challenges of automatically describing images in the wild. The challenges include high quality caption quality with respect to human judgments, out-of-domain data handling, and low latency required in many applications. Built on top of a state-of-the-art framework, we developed a deep vision model that detects a broad range of visual concepts, an entity recognition model that identifies celebrities and landmarks, and a confidence model for the caption output. Experimental results show that our caption engine outperforms previous state-of-the-art systems significantly on both in-domain dataset (i.e. MS COCO) and out of-domain datasets.
1. Introduction
The paper targets open-domain captioning, where benchmark results leave performance on in-the-wild images unclear and generic captions may omit important entities. It combines broader visual recognition, entity recognition, and confidence-based back-off, then reports human-evaluation improvements across in-domain and out-of-domain datasets.
1. Introduction
- Most prior systems were evaluated on controlled academic benchmarks, leaving their performance on open-domain images unclear.The benchmark test images had distributions similar to training examples.Source: §1 · p. 1
- Captioning systems often describe generic visual content without identifying celebrities or landmarks, even when those entities are key information in an image.The example caption names Sasha Obama, Malia Obama, Michelle Obama, and the Forbidden City.Source: §1 · pp. 1–2
- Automatic caption metrics can diverge from human judgments, with potentially weaker correlation when captions include entity information.The paper therefore reports crowd-sourced human evaluations.Source: §1 · p. 2
- The system extends a state-of-the-art compositional framework with broader visual-concept detection, celebrity and landmark recognition, and confidence-based back-off captions.It is designed to generate captions end-to-end within one second on a 4-core CPU.Source: §1 · p. 2
- Human satisfaction improved by 94.9% relatively on Instagram, and the system significantly outperformed the previous state of the art on MS COCO and out-of-domain datasets.The out-of-domain evaluations included Adobe-MIT FiveK and randomly sampled Instagram images.Source: §1 · p. 2
2. Model architecture
The architecture decomposes captioning into separately trained components integrated in a pipeline. It combines residual-network visual concepts, multimodal caption ranking, celebrity and landmark recognition, and confidence estimation.
2. Model architecture
- The system separates visual-concept detection, caption generation and ranking, entity recognition, and confidence estimation into components integrated in the caption pipeline.The components are trained separately before integration.Source: §2 · p. 2
2.1. Vision model using deep residual network
- Two vision classifiers cover 700 concepts from MS COCO training data and 1.5K visual objects from commercial image-search data, using a 50-layer ResNet initialization.At test time, the ResNet applies convolution layers once and uses average pooling; its typical runtime is 200 ms on a single-core desktop CPU.Source: §2.1 · pp. 2–3
2.2. Language and semantic ranking model
- MELM generates candidate captions, while DMSM maps images and text into a shared semantic space and supplies the main ranking signal.Ranking also uses language-model score, caption length, and detected-word coverage; MELM and DMSM are trained on MS COCO.Source: §2.2 · p. 3
2.3. Celebrity and landmark recognition
- Celebrity recognition uses a large-class CNN initialized from a model trained on 500 celebrities, with a high softmax threshold to favor precision.For landmarks, iterative data cleaning and visual-model training reduced an initial set of about 10K to about 5K landmarks.Source: §2.3 · p. 4
2.4. Confidence estimation
- The confidence model estimates caption quality using DMSM vision and caption vectors alongside language-model, caption-length, tag-coverage, and DMSM-score features.It is a logistic regression model trained on 2.5K image-caption pairs with human quality labels.Source: §2.4 · p. 4
3. Evaluation
Human evaluations compare the system with Fang et al. across COCO, MIT, and Instagram images, including challenging Instagram content. The reported results show improvements across these settings, with confidence thresholding and entity recognition contributing to the Instagram result.
Evaluation setup
- Three judges rated each image-caption pair on a four-point scale, and the evaluation included random samples from COCO, MIT, and Instagram.The reported sample sizes were 1,000 COCO images, 1,000 MIT images, and 1,380 Instagram images; the Instagram set was sampled from about 10,000 scraped images.Source: §3 · pp. 4–5, Table 1 · p. 5, Table 2 · p. 5
COCO evaluation
- On COCO, Basic reduces combined Bad and Embarrassing rates by 6.0% and raises Excellent ratings by more than 10% versus Fang et al.Adding the confidence classifier further reduces Bad and Embarrassing rates.Source: Table 1 · p. 5, §3 · p. 5
MIT evaluation
- On out-of-domain MIT, adding the confidence classifier improves Satisfaction by more than 10%.Satisfaction combines Excellent and Good ratings.Source: Table 2 · p. 5, §3 · p. 5
Instagram evaluation
- 49.5% Satisfaction for Full versus 25.4% for Fang et al. on Instagram, a 94.9% relative improvement.Confidence thresholding raises Satisfaction to 47.9%; Instagram images include filtered images and handcrafted abstract pictures that are difficult for the system to process.Source: Table 3 · p. 5, §3 · p. 5
Confidence analysis
- On Instagram under Basic, confidence scores generally align with human judgments across Excellent, Good, Bad, and Embarrassing captions.The paper suggests that confidence scores could support more sophisticated handling of difficult images.Source: §3 · p. 5
Qualitative evaluation
- For one Instagram example, both captions received Excellent ratings, although naming Ian Somerhalder provided richer information than “a man wearing a suit and tie.”The baseline caption is shown above and the system’s caption below in the qualitative examples.Source: §3 · p. 5
4. Conclusion
The paper presents a captioning system that achieves state-of-the-art results in human evaluation. It also reports public deployment to support reproducibility and further research.
Conclusion
- The paper presents an image caption system described as state of the art with respect to human evaluation.Figure 6 shows randomly sampled Instagram test images, with Fang2015 captions above and the system’s captions below.Source: §4 · p. 5, Figure 6 · p. 6
- Table 4 reports the mean and standard deviation of confidence scores by caption-quality category on Instagram under Basic.Source: Table 4 · p. 7
- The system was deployed and made publicly accessible to encourage reproducibility and facilitate further research.Source: §4 · p. 7