PaperlayerView exact revision on arXiv

Source-linked AI summary

SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text

Alexander Mathews, Lexing Xie, Xuming He

arXiv:1805.07030v1cs.CV

TL;DR

Image captioning systems either require styled captions paired with images or can produce captions with low image relevance. SemStyle separates image semantics from linguistic style, learning from unpaired styled text; 58.8% of its captions were judged distinctively styled while descriptiveness was comparable to a descriptive baseline.

  • Problem

    Styled image captioning lacks approaches that both preserve image relevance and learn from large collections of styled text without image-caption alignment.

    Source: §1 · p. 1

  • Method

    SemStyle maps images to concise semantic terms, then uses a language generator trained on styled text to decode them into captions in a target style.

    Source: §1 · p. 1

  • Results

    58.8% of SemStyle captions were judged distinctively styled, with descriptiveness of 2.97/4 versus 2.95 for the descriptive baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that SemStyle learns visually grounded style generation from text without paired images.

    Source: §7 · p. 8

    The StyleNet test set was unavailable, preventing comparisons on that test set.

    Source: §5.1 · p. 5

Abstract

from arXiv · show

Linguistic style is an essential part of written communication, with the power to affect both clarity and attractiveness. With recent advances in vision and language, we can start to tackle the problem of generating image captions that are both visually grounded and appropriately styled. Existing approaches either require styled training captions aligned to images or generate captions with low relevance. We develop a model that learns to generate visually relevant styled captions from a large corpus of styled text without aligned images. The core idea of this model, called SemStyle, is to separate semantics and style. One key component is a novel and concise semantic term representation generated using natural language processing techniques and frame semantics. In addition, we develop a unified language model that decodes sentences with diverse word choices and syntax for different styles. Evaluations, both automatic and manual, show captions from SemStyle preserve image semantics, are descriptive, and are style shifted. More broadly, this work provides possibilities to learn richer image descriptions from the plethora of linguistic data available on the web.

1. Introduction

SemStyle addresses the challenge of generating image captions that are both visually relevant and written in a chosen style, without requiring styled text paired with images. It separates image semantics from style and reports captions that are stylistically distinct while remaining descriptive.

1. Introduction

  • The open challenge is to write visually relevant captions in a given style, while existing systems require aligned styled captions or can produce weakly relevant text.The intended captions should communicate image content clearly, attractively, and in an emotionally appropriate way.Source: §1 · p. 1
  • SemStyle learns from large collections of stylised text without paired images by separating semantic relevance from style.Its semantic terms are normalized words with part-of-speech tags and verbs generalized using FrameNet; separate term and language generators map images to styled sentences.Source: §1 · p. 1
  • 58.8% of SemStyle captions were judged distinctively styled, while descriptiveness averaged 2.97/4 versus 2.95 for the descriptive baseline.The paper also reports that visual semantics were retained by the SPICE metric [1], and 41.9% of captions were judged to tell a story about the image.Source: §1 · p. 2
  • The contributions include a concise semantic representation, flexible term-based caption generation, training on styled and descriptive corpora, and competitive human and automatic evaluations.The authors also report two novel automated metrics for style.Source: §1 · p. 2

2. Related work

Related work spans stylistic text generation and image captioning, but the paper identifies a gap in systems that combine image grounding, flexible language, and learning from text without paired images.

2. Related work

  • Prior stylistic text generation is limited by small parallel corpora or uncontrolled semantic content, leaving outputs potentially irrelevant to the subject [20] [14].The paper positions its work as addressing both constraints for styled image captioning.Source: §2 · p. 2
  • Standard CNN-RNN captioners need hundreds of thousands of aligned image-caption pairs and reflect the descriptive style of their training data [12] [23] [27] [36] [57] [31] [63] [43] [52].Related work also explores unseen objects [35] [2] [55], multi-task language learning [33], and additional text for video-caption grammar.Source: §2 · p. 2
  • Neural-storyteller can produce representative style but only loose image relevance, while SentiCap [38] and StyleNet require styled image-caption data.SentiCap also requires word-level sentiment annotations; StyleNet learns from styled image-caption pairs.Source: §2 · p. 2

3. Our Approach

SemStyle first maps an image to ordered semantic terms, then decodes those terms into a sentence in a target style. Its design lets the term generator learn from image captions and the language generator learn from styled text.

3. Our Approach

  • The two-stage architecture separates image-to-term mapping from term-to-caption generation, enabling each stage to use a different training source.The term generator can use standard image-caption datasets, while the language generator can train on styled text such as romantic novels.Source: §3 · pp. 2–3, Figure 2 · p. 3

3.1. Semantic term representation

  • Semantic terms strip stylistic function words, normalize most retained words with part-of-speech tags, and generalize verbs to FrameNet frames.Frames occurring fewer than 200 times in MSCOCO are mapped to an in-vocabulary ancestor or filtered out; terms retain the original sentence order, which the authors say helps performance.Source: §3.1 · p. 3

3.2. Generating semantic terms from images

  • A CNN-RNN term generator maps image features to an ordered sequence of semantic terms, using a shorter sequence than a full sentence.At each step it predicts the next term from the image and term history, and decoding stops at the end token or a maximum length.Source: §3.2 · p. 4

3.3. Generating styled descriptions

  • The language generator encodes semantic terms with a bidirectional GRU and decodes a target-style sentence with a unidirectional GRU and attention.Attention weights encoder states for each output step; their weighted sum forms a context vector used with the decoder state to produce output-word probabilities.Source: §3.3 · p. 4

4. Learning with Unpaired Styled Texts

SemStyle learns from factual image-caption data and unaligned styled text through separate term and language generators.

4.1. Training the term generator

  • The term generator learns from factual image captions to predict semantic sequences derived from their descriptive references using mean categorical cross-entropy.Ground-truth terms are fed as inputs during training; schedule sampling did not improve performance.Source: §4.1 · pp. 4–5

4.2. Training the language generator

  • The language generator maps semantic sequences from descriptive or styled sentences to sentences in a specified style.A target-style term is appended to each input sequence during training and testing.Source: §4.2 · p. 5
  • Training the language generator on both descriptive and styled text supplies descriptive terms that are infrequent in styled sentences.Training requires text, which the authors say makes the approach adaptable to different datasets.Source: §4.2 · p. 5

5. Evaluation settings

The evaluation uses descriptive image captions and unaligned romance-novel text, compares SemStyle with baselines and variants, and assesses relevance and style automatically and with crowd judgments.

5. Evaluation settings

  • Both generators use separate 512-dimensional GRUs; the term vocabulary has 10,000 entries, and each language vocabulary has 20,000.Language-generator batches contain 64 styled sentences and 64 image captions; both generators use greedy decoding at test time.Source: §5 · p. 5

5.1. Datasets

  • MSCOCO captions and romance-novel text provide descriptive and styled training data, while the unavailable StyleNet test set precludes direct comparisons.The styled corpus is filtered and down-sampled before training.Source: §5.1 · p. 5

5.2. Compared approaches

  • The evaluation compares six state-of-the-art baselines and seven SemStyle variants covering term representation, ordering, training corpus, and target style.Baselines include CNN+RNN-coco, TermRetrieval, StyleNet, neural-storyteller, and JointEmbedding.Source: §5.2 · pp. 5–6

5.3. Evaluation metrics

  • Evaluation combines SPICE and three proposed style metrics with separate crowd ratings of descriptiveness and story-likeness.The crowd tasks rate image descriptiveness and whether a caption seems likely to belong to a story related to the image.Source: §5.3 · p. 6

6. Results

SemStyle generates styled captions while retaining image relevance: automatic and human evaluations find semantic scores competitive with descriptive systems, and crowd judgments find more story-like output than a descriptive baseline. Results also show that term representation and joint training affect style and relevance, while examples expose errors in term identification and word choice.

Evaluating relevance

  • SemStyle-coco matches CNN+RNN-coco on BLEU-4 at 0.238 and is comparable on SPICE, 0.157 versus 0.154; 99.7% of its captions are classified as descriptive.These comparisons evaluate descriptive captions against factual MSCOCO captions.Source: §6 · p. 6, Table 2 · p. 7

Evaluating style

  • SemStyle produces styled captions in 58.9% of cases with SPICE 0.144; styled baselines have higher CLF scores but much lower SPICE.TermRetrieval has SPICE 0.088, neural-storyteller 0.057, and JointEmbedding 0.046.Source: §6 · p. 6, Table 3 · p. 7

Coverage of semantic terms

  • The language generator represents 94% of generated non-FrameNet terms and 96% of verb frames; the full pipeline’s term-space precision and recall are 0.626 and 0.517.These precision and recall scores exceed styled-caption baselines and are close to SemStyle-coco’s 0.636 and 0.531.Source: §6 · p. 7

Human evaluations

  • Human raters gave SemStyle mean relevance 2.97 versus CNN+RNN-coco’s 2.95, and judged 43.8% versus 43.4% of captions clear and accurate.Only 12.2% of SemStyle captions were judged unrelated, the lowest fraction among the approaches.Source: §6 · p. 7, Figure 3 · p. 7
  • Human raters judged SemStyle captions story-like 41.9% of the time, versus 6.2% for CNN+RNN-coco; neural-storyteller reached 52.6% but had 44.2% unrelated captions.The two SemStyle variants had the lowest fraction of captions judged unrelated, below 25%.Source: §6 · p. 7, Figure 3 · p. 7

Example captions

  • Example outputs include four successes and two failures; the failures arise from incorrect cow detection by the term generator or a grammatically correct but commonsense-inconsistent word choice.Successes include story-like phrasing and more interesting verbs.Source: Figure 4 · p. 8, §6 · p. 8

Evaluating modeling choices of SemStyle

  • Randomly ordering semantic terms in SemStyle during training and testing reduced semantic relevance (SPICE 0.134 vs 0.144 for the full model) and target-style conformity (CLF 0.501 vs 0.589).SemStyle-words had SPICE 0.146 vs 0.144 and CLF 0.407; verb lemmas had SPICE 0.148 and CLF 0.533.Source: §6 · p. 8

Learning from both datasets

  • Joint training improves caption relevance: SemStyle-romonly, trained only on the romantic novel corpus, achieves CLF 0.770 but SPICE 0.138; joint training avoids this issue.Without joint training, some MSCOCO semantic terms are never seen by the language generator during training and are inaccessible at test time.Source: §6 · p. 8

7. Conclusion

SemStyle learns visually grounded style generation from text without paired images by separating semantic content from style. The conclusion identifies richer style coverage and better evaluation metrics as future work.

7. Conclusion

  • SemStyle learns visually grounded style generation from texts without paired images.It maps images to semantic terms that preserve visual content, then decodes those terms into styled descriptions.Source: §7 · p. 8
  • The method separates content and style through a semantic term representation and a styled-description decoding scheme.Source: §7 · p. 8
  • Future work includes learning from more styles and developing recognised automated and subjective metrics for styled captions.Source: §7 · p. 8

8.1. Baseline Implementation Details

The section describes the implementations of comparison systems and the JointEmbedding-based approach used to train SemStyle's sentence generator.

8.1. Baseline Implementation Details

  • CNN+RNN-coco implements Show+Tell [57] with a GRU and matches SemStyle's term-generator hyperparameters.It is the term-generator component trained to produce full sentences, using only MSCOCO.Source: §8.1 · p. 11
  • TermRetrieval queries the romance corpus with generated terms and ranks matches using BM25.It returns corpus sentences deterministically and cannot generate captions outside that corpus.Source: §8.1 · p. 11
  • StyleNet is reimplemented from Gan et al., with alternating MSCOCO and romance mini-batches to obtain convergence on these datasets.The authors report that alternating datasets each epoch failed to converge in their setting.Source: §8.1 · p. 11
  • Neural-storyteller style-shifts retrieved captions in skip-thought space and decodes them with a conditional RNN language model.Only the first generated sentence is evaluated; its decoder is trained on the romance subset of bookcorpus [64].Source: §8.1 · p. 11
  • JointEmbedding projects Inception-v3 image features and GRU caption states into a shared vector space, training paired embeddings to be more similar than unpaired ones.The noise-contrastive pairwise ranking loss follows Kiros et al. [27].Source: §8.1 · pp. 11–12, Figure 6 · p. 12
  • Training fixes the image and sentence embedders after learning them on MSCOCO pairs, then trains a GRU sentence generator to reconstruct styled sentences embedded in that space.The generator uses categorical cross entropy; the two-stage process is based on existing descriptive captioning techniques [27].Source: §8.1 · p. 12

8.2. Model Variants

The section defines SemStyle variants that isolate training-data, semantic-term representation, and term-ordering choices.

8.2. Model Variants

  • SemStyle-coco is jointly trained on MSCOCO and romance text but uses the MSCOCO indicator at test time to produce descriptive output.Source: §8.2 · p. 12
  • SemStyle-cocoonly is trained only on MSCOCO and is intended to produce purely descriptive output.Source: §8.2 · p. 12
  • SemStyle-unordered randomizes semantic-term ordering to assess the effect of ordering in the term space.Source: §8.2 · p. 12
  • SemStyle-words uses raw words without POS tags, lemmatization, or FrameNet mapping.Source: §8.2 · p. 12
  • SemStyle-lempos lemmatizes and POS-tags terms but does not map verbs to FrameNet frames, isolating verb abstraction's effect.Source: §8.2 · p. 12
  • SemStyle-romonly trains the language generator only on romance novels, without joint training, to assess joint training's effect.Source: §8.2 · p. 12

8.3. Human Evaluation

Human evaluation uses separate tasks to assess caption relevance and conformity to romantic-novel style. The evaluation process also applies worker-quality checks and aggregates multiple judgments per image-caption pair.

8.3.1 Crowd-sourcing Task Setup

  • Workers rate image-caption relevance on a four-point scale and classify style conformity as story-related, descriptive, or unrelated.Each image-caption pair is evaluated in both task types.Source: §8.3.1 · pp. 13–14

8.3.2 Crowd-sourcing Quality Control and Rating Aggregation

  • CrowdFlower requires workers to reach 70% accuracy on ground-truth questions, while each image-caption pair receives at least three ratings aggregated with weight 1/n.Descriptiveness ratings also receive an average score on a 1–4 scale.Source: §8.3.2 · p. 15

8.4. Results

The results examine human judgments and automatic measures, style attributes, semantic-term precision and recall, and the ranking of word classes by their contribution to sentence prediction.

8.4.1 BLEU, METEOR, and CIDEr for styled captions

  • The paper reports additional MSCOCO BLEU, METEOR, and CIDEr results, noting that these n-gram measures are less relevant for style generation.Human story and descriptiveness judgments were also tested pairwise with chi-squared tests and Bonferroni correction.Source: §8.4.1 · p. 15, §8.4.3 · p. 15, Table 8 · p. 16

8.4.2 Tabular Details for Human Evaluation

  • Tables 6 and 7 provide the full numerical results for human evaluations of caption accuracy and whether captions were descriptive or story-like.Their captions define accuracy ratings from unrelated to clear and accurate, and distinguish unrelated, basic-description, and story judgments.Source: §8.4.2 · p. 15, Table 6 · p. 16, Table 7 · p. 16

8.4.4 Attributes of the Generated Style

  • The style-attribute analysis sampled 4,000 sentences or captions per source and measured tense, first-person pronouns, and unique-verb counts.Past and present tense could both be counted in a sentence containing verbs of each tense.Source: §8.4.4 · pp. 16–17, Table 10 · p. 17
  • 75.0% of SemStyle captions used past-tense verbs, versus 72.0% for romance ground truth and 10.6% for CNN+RNN-coco.SemStyle used first-person pronouns in 24.4% of captions and had a verb vocabulary 92.3% larger than CNN+RNN-coco, although its verb diversity remained below both ground-truth datasets.Source: §8.4.4 · p. 17, Table 10 · p. 17

8.4.5 Precision and Recall in the Semantic Term Space

  • All four SemStyle variants using semantic terms performed better in term-space precision or recall than variants and baselines without that representation.SemStyle-cocoonly had the largest BLEU-1, while SemStyle-coco had the largest ROUGE-1; the caption defines Table 12 as reporting these measures.Source: §8.4.5 · p. 17, Table 12 · p. 18

8.4.6 Choosing Semantic Terms

  • Adjectives ranked least useful and nouns most useful for predicting ground-truth sentences; the final word-class order placed verbs before pronouns and nouns.The authors suggest adjectives may matter less because they have a local sentence effect and are often poorly detected.Source: §8.4.6 · p. 19

Related papers

Loading 1805.07030v1…