Source-linked AI summary
Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
Nasrin Mostafazadeh, Chris Brockett, Bill Dolan, Michel Galley, Jianfeng Gao, Georgios P. Spithourakis, Lucy Vanderwende
TL;DR
Existing dialogue and vision-language benchmarks provide limited coverage of natural conversations grounded in images and multimodal context. The paper introduces IGC, a crowdsourced event-centric benchmark and generation framework, and finds that combining visual and textual context improves conversational-turn generation. Human performance remains ahead of the evaluated systems, leaving IGC an open dialogue challenge.
Problem
Existing corpora emphasize task-oriented dialogue, while meaningful open-ended chit-chat lacks a shared benchmark and image conversations often extend beyond visible content.
Method
The paper defines question and response generation from visual and textual conversational context and introduces 4,222 crowdsourced multi-turn, multi-reference conversations about event-centric images.
Results
Combining visual and textual context improves question and response generation quality, while human references remain favored over the evaluated model generations.
Takeaways & Limitations
IGC provides a benchmark for studying multimodal conversation between casual chit-chat and more topic-oriented dialogue.
Takeaways & Limitations
Question generation is more successful than response generation in both automatic and human evaluation, motivating richer long-context modeling and larger high-quality training datasets.
Abstract
from arXiv · showhide
The popularity of image sharing on social media and the engagement it creates between users reflects the important role that visual context plays in everyday conversations. We present a novel task, Image-Grounded Conversations (IGC), in which natural-sounding conversations are generated about a shared image. To benchmark progress, we introduce a new multiple-reference dataset of crowd-sourced, event-centric conversations on images. IGC falls on the continuum between chit-chat and goal-directed conversation models, where visual grounding constrains the topic of conversation to event-driven utterances. Experiments with models trained on social media data show that the combination of visual and textual context enhances the quality of generated conversational turns. In human evaluation, the gap between human performance and that of both neural and retrieval architectures suggests that multi-modal IGC presents an interesting challenge for dialogue research.
1 Introduction
Vision-and-language research has largely focused on image description or visible-content questions, while social-media conversations around images often discuss events and actions beyond what is visible. The paper introduces Image-Grounded Conversation as a multimodal dialogue task centered on shared images.
- Motivation: Most vision-and-language work describes images or answers questions about their visible content, whereas image-centered social conversations often extend beyond explicit visual content.The paper notes that 28% of tweets contained an image as of June 2015 and illustrates conversations grounded in events and actions.
- Task introduction: Image-Grounded Conversation requires a system to generate conversational turns that proactively move forward a discussion grounded in an image and textual context.The task places the image in the role of interaction context rather than treating it only as an artifact to describe.
- Contributions: The paper contributes a 4,222-conversation crowdsourced benchmark, generation and retrieval models trained on 250K Twitter conversations, and experiments combining visual and textual context.The dataset contains multi-turn, multi-reference conversations grounded on event-centric images.
2 Related Work
Prior vision-and-language tasks primarily describe visible image content, answer image-grounded questions, or generate engaging questions. The paper positions IGC as natural, multimodal conversation that can depend on accompanying language and need not mention literal image objects.
- Vision and language: Visual Question Answering constrains questions to answers directly available from the image, emphasizing its explicit content.This differs from IGC, whose conversations may rely on information beyond visible objects.
- Vision and language: Visual Dialog uses sequential questions based on an automatically generated caption and is characterized as one-sided question-answer exchange rather than natural conversation.IGC instead uses the image as grounding for natural conversations, even when literal objects are not mentioned.
- Related conversation modeling: Visual Question Generation asks for natural and engaging image-related questions, while IGC adds multimodal context and conversational turns.The paper also connects IGC to data-driven conversation modeling based on social-media message-response pairs and text-only conversation history.
3 Image-Grounded Conversations
IGC defines two consecutive generation tasks: produce a question from visual and textual context, then produce a response using the image, context, and question. Its benchmark uses eventful images and crowdsourced conversations, while Twitter supplies noisy training threads.
- Task Definition: Question Generation maps visual context I and textual context T to a coherent question Q, while Response Generation maps I, T, and Q to a response R.The tasks are separated to support feasible multi-reference evaluation, although models can consume their own generated questions.
- Task motivation: IGC occupies an intermediate position between goal-driven dialogue and open-ended chit-chat, using visual grounding to constrain conversation toward event-rich, contentful utterances.The paper identifies a lack of shared datasets for tracking progress in meaningful chit-chat modeling.
- Dataset construction: The IGCCrowd benchmark was built from eventful images sampled from the VQG dataset and collected through synchronous crowdsourcing interactions.Its examples are described as semantically rich and requiring commonsense reasoning, and the dataset is publicly released.
4 Task Characteristics
IGC conversations require multimodal context: visual and textual information jointly shape natural questions, while event-rich dialogue also involves commonsense frames and causal-temporal relations.
- The Effectiveness of Multimodal Context: Human judgments found that both visual and textual contexts were highly effective for making IGC questions sound natural.The crowd dataset more often required textual context than the Twitter dataset.
- Frame Semantic Analysis of Questions: A random sample of 330 image-text-question triplets was manually annotated using Minsky’s Frames to characterize commonsense knowledge in grounded questions.Frames represent stereotypical situations and associated questions, including causes, intentions, and side effects.
- Frame Semantic Analysis of Questions: Textual context frequently evoked a more contentful FrameNet frame than the image alone, with QFN=TFN in 47% of cases versus QFN=IFN in 32%.Only 14% of annotations had matching image and textual frames, and 17% lacked a corresponding question slot.
- Event Analysis of Conversations: IGC utterances average 0.71 event-entity mentions per utterance and contain commonsense semantic links between events.The analysis uses CaTeRS annotations on 20 conversations and indicates that causal-temporal relations are important for successful IGC systems.
5 Models
The paper compares visual-only, textual-only, multimodal neural, and retrieval architectures for generating questions and responses grounded in images and conversation context. Multimodal generation combines visual and textual representations, while decoding uses beam search and reranking to reduce generic outputs.
- Neural generation models: V-Gen transforms VGGNet fc7 image features into an initial state for a one-layer GRU decoder that generates words sequentially.The fc7 representation has 4096 dimensions, and the transformed recurrent state has 500 dimensions.
- Neural generation models: T-Gen uses a Seq2Seq encoder-decoder RNN, initializing the decoder with a 500-dimensional encoding of textual context.It uses the same vocabulary size and number of layers as V-Gen.
- Neural generation models: V&T-Gen concatenates 500-dimensional visual and textual vectors, using either bag-of-words or RNN text encoding before feed-forward decoding.The textual representation is V&T.BOW-Gen or V&T.RNN-Gen.
- Neural generation models: Response generation represents conversational history with an RNN that processes utterances word by word in temporal order.This treatment is used to represent the two preceding conversational turns.
- Decoding and retrieval: All generation models use left-to-right beam search with beamsize 25 and rerank completed hypotheses from the resulting N-best lists.The reranking score combines generation probability, verb counts, hypothesis length, and inverse document frequency; parameters are optimized for smoothed-BLEU.
- Decoding and retrieval: Retrieval models select candidates from visually nearest training images or from instances retrieved using combined visual and word2vec features.V-Ret ranks candidate annotations by textual similarity, whereas V&T-Ret uses a linear combination of fc7 and word2vec features.
6 Evaluation Setup
Evaluation combines human judgments and multireference BLEU on the IGCCrowd test set. Human raters score candidate quality on a three-point scale, while BLEU measures n-gram overlap with multiple references.
- Human evaluation: Seven crowd workers rate each generated question or response on a Likert-like scale from 1 to 3.Ratings are calibrated by discarding annotators whose scores differ from the mean by more than 2 standard deviations.
- Automatic evaluation: Automatic evaluation uses corpus-level BLEU with equal weights for n-grams up to 4-grams against multiple references.The metric is presented as a replicable complement to human evaluation for diverse open-domain outputs.
7 Experimental Results
Multimodal models generally outperform alternatives in human and automatic evaluation, although BLEU remains low and question generation is more successful than response generation. The results also expose a tradeoff between generic safe outputs and more contentful reranked candidates.
- Model selection: For questions, the experiments use V&T.BOW-Gen; for responses, they use V&T.RNN-Gen and omit V-Gen because visual-only response generation is unlikely to succeed.All models are trained on IGCTwitter except VQG, which uses 7,500 VQG questions.
- Human evaluation: The visual-and-textual model outperforms other systems in human evaluation, while human gold references remain consistently preferred.Judges preferred the top beam over the reranked beam, suggesting a tradeoff between safe generic utterances and riskier contentful ones.
- Automatic evaluation: BLEU scores are low, but the multimodal V&T model outperforms other models across test sets except the VQG model.The authors attribute VQG’s advantage partly to similar event-centric training images and to differences between crowdsourced and Twitter data.
- Training data: Controlled, high-quality crowdsourced training data can be more effective than noisy Twitter data, although collecting paired multi-turn conversations at scale is expensive.This cost motivates using readily available large-scale data from the wild.
- Overall comparison: Question generation is more successful than response generation in both automatic and human evaluation.The authors suggest richer modeling of long multi-turn contexts and larger high-quality datasets as possible ways to reduce the disparity.
8 Conclusions
The paper introduces multimodal Image-Grounded Conversation and releases a crowdsourced benchmark of event-centric multi-turn conversations. Experiments indicate that combining visual and textual context improves generation quality, while remaining performance gaps motivate further research.
- Contributions: IGC requires systems to generate meaningful conversational turns from an image and natural-language text.The released IGCCrowd dataset contains 4,222 high-quality multi-turn conversations with multiple references about eventful images.
- Conclusions: Capturing multimodal context improves the quality of generated questions and responses.The gap between the best models and humans leaves room for research spanning casual chit-chat and topic-oriented dialogue.
A IGCTwitter Training Dataset
IGCTwitter is a large-scale training resource extracted from Twitter image conversations, but substantial noise remains despite filtering and normalization. Its sentences tend to be longer on average than those in other datasets.
- Dataset construction: The authors sampled 250K visual-textual conversation quadruples from 1.4 million Twitter threads for IGCTwitter.The data was extracted from the Twitter Firehose over three years beginning in May 2015.
- Dataset construction: Filtering removed tweets with mid-tweet hashtags, excessive length, or URLs unrelated to the image.A pilot study motivated the 80-character limit as a way to retain concise, one-sentence utterances.
- Dataset limitations: 46% of sampled Twitter conversations were estimated to involve prior user history, complicating image- and context-grounded response generation.Screenshots and nonphotographic graphics were also identified as major potential noise sources.
- Dataset limitations: Despite these issues, IGCTwitter serves as the paper’s primary training dataset because its large scale and low extraction cost support open-ended conversational modeling.The authors note that it could be collected in substantially greater quantities.
- Dataset characteristics: IGCTwitter has longer sentences on average than the other compared datasets.
B Data Analysis
The analysis compares IGC datasets with existing vision-and-language benchmarks across sentence length, n-gram diversity, vocabulary, abstraction, and annotation similarity. IGCTwitter is especially diverse and linguistically broad, while IGCCrowd’s contextual questions are competitive with VQG in inter-annotation similarity.
- Dataset analysis: The analysis compares IGC datasets with existing vision-and-language datasets across several linguistic and annotation measures.
- Dataset analysis: IGCTwitter has the most diverse question n-gram distribution among VQG, IGCTwitter, and IGCCrowd.The lighter-colored circle regions represent sequences occurring in fewer than 0.1% of the dataset.
- Dataset analysis: IGCTwitter has by far the largest vocabulary size, making it more challenging for training than the compared datasets.
- Dataset analysis: IGCCrowd and IGCTwitter have the highest ratios of abstract to concrete terms, respectively, among the compared datasets.Abstract terms include concepts, qualities, and feelings, whereas concrete terms are sensory-experienced things.
- Dataset analysis: IGCCrowd’s contextually grounded questions are competitive with VQG in inter-annotation textual similarity.IGCTwitter is excluded from this comparison because it lacks multiple references.