PaperlayerView exact revision on arXiv

Source-linked AI summary

SentiCap: Generating Image Descriptions with Sentiments

Alexander Mathews, Lexing Xie, Xuming He

arXiv:1510.01431v2cs.CVcs.CL

TL;DR

Existing image-captioning systems largely produce factual descriptions, leaving emotional style underrepresented despite its prevalence in everyday communication. SentiCap generates positive or negative captions with a switching RNN and word-level supervision, using limited sentimental data. Crowd evaluations found strong sentiment alignment and comparable descriptiveness, including over 90% emotional-caption coverage across images.

  • Problem

    Current image-captioning systems omit stylized, non-factual language such as emotion, despite emotions being common in everyday communication.

  • Method

    SentiCap combines parallel factual and sentiment CNN+RNN streams with a switching gate and word-level regularization, trained using 2000+ sentimental captions alongside neutral captions.

  • Results

    Over 90% of images received an emotional caption, and crowd workers rated the vast majority of generated captions as having the appropriate sentiment.

  • Takeaways & Limitations

    SentiCap generates emotional image captions that retain descriptive quality while expressing positive or negative sentiment.

Abstract

from arXiv · show

The recent progress on image recognition and language modeling is making automatic description of image content a reality. However, stylized, non-factual aspects of the written description are missing from the current systems. One such style is descriptions with emotions, which is commonplace in everyday communication, and influences decision-making and interpersonal relationships. We design a system to describe an image with emotions, and present a model that automatically generates captions with positive or negative sentiments. We propose a novel switching recurrent neural network with word-level regularization, which is able to produce emotional image captions using only 2000+ training sentences containing sentiments. We evaluate the captions with different automatic and crowd-sourcing metrics. Our model compares favourably in common quality metrics for image captioning. In 84.6% of cases the generated positive captions were judged as being at least as descriptive as the factual captions. Of these positive captions 88% were confirmed by the crowd-sourced workers as having the appropriate sentiment.

1 Introduction

Image captioning has advanced toward factual descriptions, but current systems omit emotional and other stylistic language. SentiCap addresses this gap by generating positive or negative image captions with a switching RNN.

  • Automatic image description combines visual recognition and language generation, while existing systems primarily describe image content factually.The paper frames emotion as one richer variation in language style that is common in everyday communication.
  • Figure 1 contrasts factual captions with positive and negative emotional captions for the same images.The examples are drawn from MSCOCO and this work, with positive captions shown in green and negative captions in red.
  • SentiCap uses two parallel RNN streams and word-level regularization to generate captions with sentiments from limited sentimental training data.One stream models background language, while the other specializes in sentiment descriptions; training uses 2000+ sentimental captions and 413K neutral captions.

2 Related Work

Prior work advanced factual image captioning, language modeling, and analysis of visual emotion, but did not compose image descriptions with user-specified sentiments. SentiCap builds on these strands while focusing on sentence generation rather than hidden-variable sentiment modeling.

  • Image captioning progressed from retrieval and sentence-fragment assembly toward models representing richer language structure.Examples include parse-tree mapping and joint spaces for visual and compositional semantics.
  • CNN image representations paired with RNN or maximum-entropy language models became a standard approach for generating image captions.The cited systems seed or condition language generation on visual features.
  • Earlier research modeled image aesthetics, emotion, and frequently used adjective-noun pairs, but did not compose descriptions with desired sentiments.SentiCap uses Visual SentiBank to construct a sentiment vocabulary.
  • Some sentiment-text models used latent variables, whereas SentiCap generates sentences without explicitly modeling sentiment through hidden variables.The distinction concerns the paper’s focus on generation rather than latent-variable sentiment modeling.

3 Describing an Image with Sentiments

SentiCap models sentiment-aware caption generation with two parallel CNN+RNN streams and a learned switching mechanism. Word-level sentiment supervision regularizes learning so the sentiment stream can be trained from limited sentiment-caption data while leveraging factual-caption data.

  • 3.1 Switching RNNs for Sentiment Captions: SentiCap generates captions by combining a factual background language model with a sentiment-specialized model in a switching multi-modal language model.A binary sentiment variable selects between the two underlying generation mechanisms at each word position.
  • 3.1 Switching RNNs for Sentiment Captions: The switching RNN uses two parallel CNN+RNNs, with the sentiment variable functioning as a gate between the background and sentiment streams.The switch probability is produced from the hidden states of both RNNs by a single-layer network.
  • 3.1 Switching RNNs for Sentiment Captions: Each stream is an LSTM-based conditional caption model whose initial word embedding incorporates a learned projection of CNN image features.The LSTM hidden state summarizes the image and preceding words, and the corresponding stream determines output-word probabilities.
  • 3.2 Learning the Switching RNN Model: Training proceeds in two stages: a background CNN+RNN is learned from a large factual-caption dataset, then the switching model is learned from a small sentiment-caption dataset.The sentiment model is initialized from the background model, while the background parameters remain fixed during sentiment-stage optimization.
  • 3.2 Learning the Switching RNN Model: The sentiment-stage objective combines weighted caption log likelihood, cross-entropy between predicted and word-level sentiment labels, and parameter regularization.Sentiment-word indicators increase the likelihood weighting, while regularization balances the new model against the base RNN.
  • 3.2 Learning the Switching RNN Model: Positive and negative sentiment models are trained separately, with sentiment words identified through ANPs and default L2 regularization applied during optimization.The training procedure uses polarity-specific sentiment data and searches λθ, λη, and λγ on a validation set.

4 An Image Caption Dataset with Sentiments

The authors construct a dataset of image captions that describe pictured content while conveying a target positive or negative sentiment. The collection uses crowd-sourced rewriting and sentiment judgments, with broad agreement that the captions match their intended emotional polarity.

  • 4 An Image Caption Dataset with Sentiments: The dataset pairs images with captions that both describe the image and convey a desired sentiment.The collection is designed to support learning associations between image content and sentiment-bearing language.
  • 4 An Image Caption Dataset with Sentiments: The collection targets emotional captions from objective viewers who do not know the photograph’s external backstory.Crowd workers rewrite captions to add sentiment while retaining objective image description.
  • 4 An Image Caption Dataset with Sentiments: 1,027 positive and 436 negative ANPs form the sentiment vocabulary used to guide caption collection.The ANPs are expanded from Visual SentiBank with entries from YFCC100M captions and filtered for frequency and clear polarity.
  • 4 An Image Caption Dataset with Sentiments: Negative captions received a significant 0.73 decrease in the reported evaluation measure, likely reflecting the diversity of negativity.The passage contrasts this decrease with the positive-caption results and attributes it cautiously to the varied notion of negativity.
  • 4 An Image Caption Dataset with Sentiments: 94% of POS captions and 82% of NEG captions were unanimously judged to have sentiment appropriate to their images.These correspond to 315/335 positive captions and 250/305 negative captions; most remaining captions still received majority agreement.

5 Experiments

Experiments evaluate SentiCap with automatic metrics and crowd-sourced judgments, showing sentiment control, descriptiveness, novelty, and occasional semantic or sentiment errors.

  • Evaluation setup: SentiCap was evaluated with BLEU, ROUGEL, METEOR, and CIDEr automatic metrics alongside Amazon Mechanical Turk judgments.Crowd workers rated descriptiveness and selected the more positive or negative caption, with each pair rated by three workers.
  • Sentiment evaluation: 88.4% of positive SentiCap captions were judged more positive than CNN+RNN, while 72.5% of negative captions were judged more negative.These sentiment gains were reported against the factual CNN+RNN baseline.
  • Descriptiveness: 84.6% of positive SentiCap captions were judged descriptive or more descriptive than CNN+RNN despite a small average descriptiveness reduction.The reported decrease was attributed to a minority of failure cases.
  • Interest judgments: Positive SentiCap captions were judged more interesting than factual captions in 66.4% of cases, whereas negative captions were judged less interesting in 63.2%.The paper relates the negative result to negativity conflicting with the positive notion of interest.
  • Novelty: 95.7% of SentiCap captions were novel, compared with 38.2% for CNN+RNN trained only on MSCOCO.A caption is novel when it has no match in either the MSCOCO training set or sentiment caption dataset.
  • Qualitative results: Generated examples often combine sentiment phrases with surrounding visual content, but some contain semantic or sentiment errors.One example confuses a child on a playground with a man on a skateboard, while another juxtaposes incompatible positive and negative adjective-noun pairs.

6 Conclusion

The paper proposes SentiCap, a switching RNN for emotional image captions, using specialized word-level supervision to learn from limited sentiment data. Evaluations show emotional captions for over 90% of images, with most judged sentiment-appropriate.

  • Contribution: SentiCap is a switching RNN model for generating image captions with sentiments.The conclusion identifies this as the paper's central proposed model.
  • Contribution: Specialized word-level supervision helps the model use a small amount of sentiment-caption training data.The method is designed for positive and negative emotional captions.
  • Conclusion: SentiCap generates an emotional caption for over 90% of images, and the vast majority are rated as having the appropriate sentiment.These findings are based on automatic and crowd-sourced evaluations.
  • Future work: Future work includes unified positive-and-negative modeling, broader linguistic styles, and richer emotions such as pride, shame, and anger.These directions extend sentiment generation beyond the current scope.

7 Appendix

The appendix documents model details, sentiment categories, dataset construction, caption rewriting, and crowd-sourced validation and evaluation procedures.

  • The appendix provides additional details on the model, data collection, sentiment perspectives, ANP selection, sentence rewriting, validation, and generated-sentence evaluation.It covers Sections 7.1–7.6 and is intended to support reproducibility and clarify data collection.
  • 7.1 The LSTM unit: The LSTM units are functionally the same as Vinyals et al. (2015), omit contextual input concatenation, and exclude output and embedding layers from Figure 5.The figure depicts only the LSTM unit, while Equation 2 gives its complete definition.
  • 7.2 Sentimental descriptions in the first, second, and third person: The work focuses on third-person sentiment because first- and second-person sentiment depends on personal or communication context unavailable to the computer.Third-person sentiment is expressed by an objective viewer who knows the visual content but not the photo’s backstory.
  • 7.3 Customizing Visual Sentibank for captions: The sentiment vocabulary combines Visual SentiBank methodology with YFCC100M captions, retaining frequent ANPs with clear positive or negative sentiment.ANPs are scored using the average of SentiWordNet and SentiStrength, with an absolute sentiment score of 0.1 or above.
  • 7.4 AMT interface for collecting image captions with sentiment: Caption collection used three design iterations, ending with rewriting MSCOCO captions using candidate ANPs and obtaining three positive and three negative descriptions per image.The earlier approaches produced overly long imaginative captions or rigid, linguistically awkward sentences.
  • 7.5 AMT interface validating image captions with sentiment: The validation interface rated descriptiveness and sentiment-caption suitability, while the comparative evaluation rated positivity, interestingness, and descriptiveness across four methods.The validation task presented three MSCOCO captions and three sentiment sentences; comparative tasks used pairwise ratings and a four-point descriptiveness scale.
  • 7.6 AMT interface for rating captions with a sentiment: Initial AMT ratings were unreliable, so the study used workers with at least 95% approval and filtered out workers failing 60% accuracy on interspersed sentiment checks.The authors suspected bots or random selection in the initial rating process.
Loading 1510.01431v2…