Source-linked AI summary
Visual Storytelling
Ting-Hao, Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, Margaret Mitchell
TL;DR
The paper addresses the gap between literal image captioning and context-sensitive visual storytelling. It introduces a tiered sequential vision-to-language dataset, establishes baselines and an automatic evaluation metric, and argues that figurative and social language can support more human-like grounded understanding.
Problem
Existing vision-to-language work focuses mainly on direct, literal image description, leaving context-dependent social, emotional, evaluative, and figurative language less explored.
Method
The paper constructs aligned DII, DIS, and SIS language tiers for sequential images, then develops crowdsourced stories, descriptions, and storytelling baselines.
Results
The paper establishes strong storytelling baselines and motivates METEOR as an automatic metric correlated with human judgments.
Takeaways & Limitations
Modeling concrete, figurative, and social language in sequential visual data is presented as a path toward more human-like understanding of grounded event structure and subjective expression.
Takeaways & Limitations
The reported heuristics are intended as clear baselines rather than the right approach to story generation, leaving training and decoding methods open for exploration.
Abstract
from arXiv · showhide
We introduce the first dataset for sequential vision-to-language, and explore how this data may be used for the task of visual storytelling. The first release of this dataset, SIND v.1, includes 81,743 unique photos in 20,211 sequences, aligned to both descriptive (caption) and story language. We establish several strong baselines for the storytelling task, and motivate an automatic metric to benchmark progress. Modelling concrete description as well as figurative and social language, as provided in this dataset and the storytelling task, has the potential to move artificial intelligence from basic understandings of typical visual scenes towards more and more human-like understanding of grounded event structure and subjective expression.
1 Introduction
The paper introduces sequential vision-to-language as a way to connect image sequences with descriptive and narrative language. Its tiered dataset targets the transition from literal scene understanding toward cohesive, context-sensitive storytelling.
- Sequential vision-to-language moves beyond static, context-free images toward reasoning about events as they occur and change.The language goal likewise extends from literal description toward evaluative, conversational, and abstract expression.
- The paper introduces the first dataset of sequential images with corresponding descriptions and advances visual storytelling.
- Three aligned language tiers—DII, DIS, and SIS—capture differences between isolated description, sequence-aware description, and narrative storytelling.Because the tiers use the same images, they support modeling relationships between literal and abstract visual concepts.
- The paper proposes an automatic metric correlated with human judgments and establishes strong baselines for visual storytelling.
2 Motivation and Related Work
Prior vision-to-language work largely describes image content literally, leaving context-dependent social, emotional, evaluative, and figurative language relatively unexplored. Visual storytelling is presented as a task that can engage these richer forms of grounded understanding.
- Existing vision-to-language research spans captioning, question answering, visual phrases, video understanding, and visual concepts, but focuses on direct literal description.
- Context can distinguish a literal scene description such as “sitting in a room” from a socially inferred interpretation such as “bonding”.The latter draws on social relations and emotions inferred from context while remaining grounded in visual input.
- Visually grounded stories support more evaluative and figurative language than prior vision-to-language research.
- Storytelling can educate, preserve culture, instill morals, and share advice, motivating its study as a route toward more human-like intelligence and understanding.
3 Dataset Construction
The dataset is built from event-oriented Flickr albums and a staged crowdsourcing process that aligns selected photo sequences with stories and descriptions. Separate language tiers preserve distinctions between isolated viewing, sequence context, and storytelling.
- Extracting Photos: The collection targets “storyable” event types by mining possessive 5-grams from Flickr titles and descriptions and retaining event heads classified in WordNet3.0.Manual winnowing and a small hand-constructed gazetteer supplement the data-driven selection.
- Extracting Photos: Albums are collected through the Flickr API and restricted to 10–50 photos, 48-hour spans, and Creative Commons licensing.
- Crowdsourcing Stories In Sequence: A two-stage workflow first has workers select and narrate photo subsets, then has other workers retell stories from the resulting sequences.
- Crowdsourcing Stories In Sequence: Workers arrange at least five photos on a storyboard and write a sentence or phrase for each selected image, preserving text–photo alignments.Albums skipped by two workers are discarded.
- Crowdsourcing Descriptions: DII and DIS descriptions are collected for sequences whose stories receive majority support, using MS COCO-style captioning instructions.DII presents images in isolation, whereas DIS presents photo sequences with adapted sequence instructions.
- Data Post-processing: The released data is tokenized, anonymized for people names and named entities, and split into training, validation, and test sets at 80%/10%/10% over story albums.
4 Data Analysis
The dataset spans 10,117 Flickr albums and 210,819 unique photos, while tier-specific language shifts from isolated literal descriptions toward context-sensitive and narrative expression.
- 10,117 Flickr albums contain 210,819 unique photos, averaging 20.8 photos per album across 7.9-hour spans.Album-level variability is σ = 9.0 photos and σ = 11.4 hours.
- Descriptions-in-isolation frequently use generic people labels such as “man” or “woman,” reflecting limited social specificity.
- Descriptions-in-isolation also favor posture verbs such as “standing” and “sitting” when single images provide little event information.
- Descriptions-in-sequence reduce relatively uninformative isolated-image words by incorporating temporal context.
- Stories-in-sequence contain more names, temporal references, and dynamic or abstract words such as “went” and “decided.”
5 Automatic Evaluation Metric
The study compares automatic story metrics with human judgments to identify a rapid benchmarking proxy for storytelling quality. METEOR correlates best among the evaluated metrics, apparently because it incorporates paraphrasing.
- Human judgment is treated as the most reliable evaluation for generated stories, while automatic metrics provide faster benchmarking.The correlation study uses 3,000 SIS training stories and five crowd judges per story.
- METEOR correlates best with human judgments among the evaluated automatic metrics.
- METEOR’s stronger correlation signals that paraphrase-sensitive metrics may better match human judgments for storytelling.The paper uses METEOR version 1.5 with hter weights.
- A more detailed study of automatic story evaluation remains future work.
6 Baseline Experiments
The baseline experiments compare sequence-to-sequence story generation with captioning and show that simple decoding constraints substantially improve storytelling quality. These results establish practical baselines while exposing challenges distinct from image caption generation.
- Experimental setup: The experiments use SIS for story training and compare against DII-based per-image caption generation, with example outputs reported in Table 5.The authors present these systems as clear baselines for challenges that differ fundamentally from caption generation.
- Model: A sequence-to-sequence GRU RNN encodes image sequences in reverse order and decodes stories word by word with softmax loss.The image encoder runs over fc7 vectors, and its final representation initializes the story decoder.
- Baseline decoding: The initial beam-search baseline produces generic, repetitive, high-level descriptions that are subjectively poor for story generation.The system uses beam search with size=10, a setting previously successful for image captioning.
- Decoding heuristics: A beam size of 1 improves storytelling by 4.6 METEOR points, whereas greedy decoding performs worse than beam search for caption generation.This contrast highlights a difference between story and caption generation.
- Decoding heuristics: Preventing repeated content words adds another 2.3 METEOR points after greedy decoding.The constraint addresses repeated words and phrases in greedy-generated stories.
- Caption-informed generation: Caption-model licensing of visually grounded words yields a further 1.3 METEOR improvement under greedy decoding without duplication.Words occurring in at least 10 sentences of the caption model’s 100-best list are licensed for generation.
7 Conclusion and Future Work
The paper introduces the first sequential vision-to-language dataset and advances visual storytelling from isolated image descriptions toward stories over image sequences. It argues that figurative and social language, together with strong baselines and METEOR-based evaluation, supports progress toward more human-like grounded understanding.
- Conclusion: The dataset moves incrementally from images-in-isolation to stories-in-sequence and supports sequential vision-to-language research.The conclusion presents this dataset as the paper’s central contribution.
- Future work: Modelling figurative and social language is presented as essential for evolving AI toward more human-like understanding.The stated target includes grounded event structure and subjective expression.
- Evaluation: The paper establishes strong visual-storytelling baselines and motivates METEOR as an automatic metric for evaluating future progress.The metric is proposed for benchmarking progress on the task.