Source-linked AI summary
Engaging Image Captioning Via Personality
Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, Jason Weston
TL;DR
Standard image captioning focuses on factual, neutral descriptions that are not engaging to humans, motivating a task centered on personality-conditioned communication. The paper collects Personality-Captions and develops retrieval and generative models using strong image and language representations; its best retrieval model approaches human engagingness, while generative models remain weaker on the new task.
Problem
Standard captioning verifies image understanding but produces factual, neutral descriptions that humans do not find engaging as communication.
Method
The paper collects personality-conditioned captions and adapts retrieval and generative image-captioning models using image and language representations.
Results
The best retrieval model nearly matches human engagingness, while generative models perform worse on Personality-Captions.
Takeaways & Limitations
Personality-Captions supports evaluating and modeling image-grounded captions for human engagement, with retrieval currently stronger than generation on this task.
Takeaways & Limitations
Personality-Captions has more diverse outputs than COCO, and automatic measures such as BLEU correlate poorly with human judgments.
Abstract
from arXiv · showhide
Standard image captioning tasks such as COCO and Flickr30k are factual, neutral in tone and (to a human) state the obvious (e.g., "a man playing a guitar"). While such tasks are useful to verify that a machine understands the content of an image, they are not engaging to humans as captions. With this in mind we define a new task, Personality-Captions, where the goal is to be as engaging to humans as possible by incorporating controllable style and personality traits. We collect and release a large dataset of 201,858 of such captions conditioned over 215 possible traits. We build models that combine existing work from (i) sentence representations (Mazare et al., 2018) with Transformers trained on 1.7 billion dialogue examples; and (ii) image representations (Mahajan et al., 2018) with ResNets trained on 3.5 billion social media images. We obtain state-of-the-art performance on Flickr30k and COCO, and strong performance on our new task. Finally, online evaluations validate that our task and models are engaging to humans, with our best model close to human performance.
1. Introduction
The paper argues that image captioning should move beyond factual scene description toward communication that captures human interest, emotion, and personality. It introduces Personality-Captions and models designed to generate engaging, trait-conditioned captions.
- Motivation: Standard captioning describes image content factually and neutrally, but does not target the engaging communication humans expect.This limits its usefulness as an end application when viewers can already see the image.
- Contribution: The PERSONALITY-CAPTIONS dataset contains 241,858 captions conditioned on 215 personality traits.The dataset is introduced to cover a broad range of human personalities.
- Approach: The authors develop retrieval and generative models that combine image and language representations to produce engaging captions.They use image features trained on 3.5 billion social media images and Transformer text representations trained on 1.7 billion dialogue examples.
- Results: The generative model reaches state-of-the-art COCO caption generation, while TransResNet achieves the highest known Flickr30k R@1 score.The same architectures are adapted to Personality-Captions by conditioning the input image on personality traits.
- Results: Human annotators preferred the retrieval model's captions over human captions 49.5% of the time, while generative models remained challenging on the new task.The reported preference rate was close to human performance.
2. Related Work
Related work largely evaluates image understanding through factual captioning, while other research explores personalization, humor, style, and human communication. The paper positions Personality-Captions as a general personality-based approach and motivates human evaluation because automatic overlap metrics are weak for diverse outputs.
- Image Captioning: COCO and Flickr30k measure image-content understanding but do not address more natural human communication.The paper compares against both generative and retrieval-based captioning systems on these datasets.
- Engaging Captions: Prior work personalizes captions to readers using features such as location, age, or vocabulary, whereas this paper focuses on personality traits.The paper explicitly does not address reader personalization.
- Engaging Captions: Other approaches target humor through wordplay or humor-web data, but the paper aims to model a general set of personality traits.This distinguishes the task from specialized amusing-caption generation.
- Human Communication: Image-grounded communication tasks have intrinsically diverse outputs, making simple word-overlap metrics weak and motivating human evaluation.The paper therefore evaluates engagingness with human judgments.
- Modeling: The modeling approach builds on advances in image encoding and attention-based Transformer text representations.The paper applies strong image features and Transformer-based representations to captioning.
3. Personality-Captions
Personality-Captions is collected as image, personality-trait, and caption triples designed to elicit engaging comments rather than factual descriptions. The dataset uses crowdsourced trait-conditioned writing and quality controls, and is presented alongside comparisons with existing captioning resources.
- Dataset: PERSONALITY-CAPTIONS is a publicly available collection of image, personality-trait, and caption triples gathered with crowdworkers.The dataset is released for training models on personality-conditioned captioning.
- Personality Traits: The collection uses single descriptive personality labels because crowdsourced annotators may not know multidimensional personality models.The paper contrasts this design with low-dimensional frameworks such as the Big-Five.
- Data Collection: For each selected image, the dataset assigns a randomly chosen personality trait and collects captions from multiple crowdworkers.Training, validation, and test sets are sampled from YFCC100M images; test examples have five captions per image.
- Annotation Task: Annotators write an engaging utterance for an image in the context of the assigned trait, rather than a factual caption.The instructions avoid the word “caption” to encourage natural personality-based comments.
- Quality Control: Captions must contain at least three words, must not use the trait word, and treat the trait as describing the author rather than the image.Crowdworkers were manually monitored and poor performers were removed.
- Dataset Comparison: Table 1 compares PERSONALITY-CAPTIONS with FlickrStyle10k, COCO, and Flickr30k as reference datasets.The cited text states that FlickrStyle10k is significantly smaller in images, examples, and personalities.
4. Models
The paper develops retrieval and generative captioning models conditioned on images and personality traits, using pretrained image encoders and Transformer-based text representations. Its TransResNet retrieval model projects image, personality, and caption representations into a shared space for candidate selection.
- Model classes: The paper compares retrieval models, which select training captions, with generative models, which produce novel captions word by word from an image and personality trait.Both model classes require an image encoder.
- Image encoders: Both model types use pretrained image features, comparing ResNet152 trained on ImageNet with ResNeXt-IG-3.5B trained on 3.5 billion Instagram pictures.The encoders produce 2048-dimensional image vectors; attention models can retain 7 × 7 × 2048 spatial features.
- Generative models: Generative captioning models include SHOWTELL, SHOWATTTELL, and UPDOWN, with attention-based variants retaining spatial image features for decoding.The models use LSTM-based caption decoders, with UPDOWN using separate LSTMs for attention and word generation.
- TransResNet: TransResNet projects image, personality, and caption representations into a shared space and scores candidates using the dot product (rI + rP) · rC.Image and personality representations are summed, while captions are encoded with a Transformer or bag-of-words encoder.
- TransResNet: At inference, TransResNet selects the candidate caption with the maximum score, while training maximizes the log-likelihood of correct responses using in-batch captions as negatives.Training uses mini-batches of 500 examples.
5. Experiments
The experiments evaluate the proposed architectures on traditional captioning and Personality-Captions using automatic metrics and human judgments of engagingness and fit. Results show strong performance from ResNeXt-IG-3.5B features, personality conditioning, and retrieval-based modeling, with the best model approaching human engagingness.
- Experimental setup: The experiments assess factual captioning on COCO and Flickr30k, then evaluate engaging captions conditioned on personality using automatic metrics and human judgments.Human evaluations measure engagingness and whether captions fit the image and personality trait.
- Traditional captions: On COCO, UPDOWN with ResNeXt-IG-3.5B features outperforms or matches state-of-the-art single-model performance across most metrics, especially CIDEr.ResNeXt-IG-3.5B features consistently outperform ResNet152 features in the generative comparisons.
- Traditional captions: 68.4 R@1 on Flickr30k versus 56.8 R@1 for the previous state of the art makes the best retrieval model state-of-the-art by a large margin.The same experiments find improvements from stronger image features, Transformer text encoding, and text pretraining.
- Personality-Captions: 16.5 CIDEr is achieved by the best generative model when personality embeddings are added, demonstrating the importance of personality modeling for the new task.Without personality conditioning, the generative models follow the standard-captioning setup.
- Personality-Captions: 77.5% R@1 is obtained by the best ResNeXt-IG-3.5B retrieval model, compared with 53.9% for its best unconditioned variant.The evaluation ranks 500 candidates per sample, comprising 495 random test-set candidates plus the true labels.
- Human evaluation: Personality-conditioned human captions beat neutral captions with a 64.5% win rate, while TransResNet with ResNeXt-IG-3.5B nearly matches human engagingness at 49.5%.TransResNet also beats the best generative model directly with an 80.1% win rate; human captions fit images slightly better, while TransResNet fits personality better.
6. Conclusion
The paper combines image understanding with personality-conditioned captioning to make captions engaging for humans. Its best system approaches human performance, while generative models remain an open problem.
- The paper develops models that jointly understand image content and produce engaging captions for humans.
- It uses generative and retrieval architectures built from advances in image and sentence encoding.
- The models are conditioned on controllable personality traits using the PERSONALITY-CAPTIONS dataset.
- The best system produces captions close to human performance in engagement and relevance.
- Generative models perform less well on this task, leaving their improvement as an open problem.
A. Impact of Pretrained Word Embeddings and Text Encoders
The retrieval model uses Transformer-based text encoding with either pretrained word embeddings or a pretrained encoder. Ablations show that pretraining matters for retrieval, while word pretraining gives a smaller generative-model gain.
- TransResNet encodes captions with a Transformer architecture that can be pretrained in two ways.
- The first option pretrains word embeddings using FastText word vectors.
- The second option pretrains the entire encoder on a similar task following prior work.
- Tables 9, 10, and 12 report ablations showing the importance of pretraining for retrieval models.
- 0.8 BLEU can be gained by applying the same word pretraining to generative models.
B. Engaging Captions, with no personality conditioning
The study compares captions written with and without personality conditioning and examines their diversity. Unconditioned engaging captions receive a slight overall preference, while personality conditioning broadens the observed personality range.
- Engaging-only Captions: Human annotators slightly preferred unconditioned engaging captions, selecting them approximately 54% of the time.
- Engaging-only Captions: The comparison tests captions written without a personality cue against captions conditioned on a personality trait.
- Diversity of captions: Personality-conditioned captions were more diverse in personality traits than captions produced without personality conditioning.
- Diversity of captions: Diversity was measured by assigning predicted personality traits with a classifier and comparing the total trait sets across annotation types.
C. Comparing Generative and Retrieval Models on COCO
The paper evaluates generative and retrieval captioning models with automatic metrics and human engagingness comparisons. The supplied passages emphasize that automatic metrics may not reflect human judgments and provide tables, examples, and evaluation procedures for comparison.
- Automatic metrics are difficult to compare across retrieval and generative models because each model type is optimized for different metrics.
- Human evaluations are used as the ultimate test of generative and retrieval models on PERSONALITY-CAPTIONS.
- The COCO comparison reports BLEU, CIDEr, SPICE, and ROUGE-L scores for the best TransResNet model.
- The tables include retrieval results on Flickr30k and PERSONALITY-CAPTIONS, generative pretraining comparisons, pairwise engagingness win rates, and caption-diversity measurements.
- Table 15 compares generative and retrieval model performance on COCO using the test split and ResNeXt-IG-3.5B image features.
- Example tables show human and TransResNet captions, personality-conditioned evaluation variants, and additional TransResNet predictions.