PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Chat: Engaging Grounded Conversations

Kurt Shuster, Samuel Humeau, Antoine Bordes, Jason Weston

arXiv:1811.00945v2cs.CL

TL;DR

The paper addresses how machines can communicate naturally about perceived images while accounting for conversational mood and style. It collects styled image-grounded conversations and studies multimodal retrieval and generative models; its best retrieval model is preferred to human conversationalists 47.7% of the time.

  • Problem

    The paper targets natural-language communication about the world that reflects image meaning and the mood and style shaping conversation.

    Source: §1 · p. 1

  • Method

    The authors collect crowdworked image-and-style dialogues and study retrieval and generative models that combine image, dialogue, and style representations.

    Source: §1 · p. 1, §3 · p. 2, §4 · p. 3

  • Results

    The best fused retrieval models are preferred to human conversationalists 47.7% of the time, and the paper reports state-of-the-art results on IGC.

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    The authors conclude that engaging chit-chat models could contribute to social dialogue agents, while combining engagingness with world knowledge, personal interests, and task proficiency remains a challenge.

    Source: §6 · p. 8

    Human evaluations cover short conversations, and retrieval models outperform their generative versions.

    Source: §6 · p. 8

Abstract

from arXiv · show

To achieve the long-term goal of machines being able to engage humans in conversation, our models should captivate the interest of their speaking partners. Communication grounded in images, whereby a dialogue is conducted based on a given photo, is a setup naturally appealing to humans (Hu et al., 2014). In this work we study large-scale architectures and datasets for this goal. We test a set of neural architectures using state-of-the-art image and text representations, considering various ways to fuse the components. To test such models, we collect a dataset of grounded human-human conversations, where speakers are asked to play roles given a provided emotional mood or style, as the use of such traits is also a key factor in engagingness (Guo et al., 2019). Our dataset, Image-Chat, consists of 202k dialogues over 202k images using 215 possible style traits. Automatic metrics and human evaluations of engagingness show the efficacy of our approach; in particular, we obtain state-of-the-art performance on the existing IGC task, and our best performing model is almost on par with humans on the Image-Chat test set (preferred 47.7% of the time).

1 Introduction

The paper studies image-grounded conversation as a way for machines to communicate naturally about the world, including how mood and style shape dialogue. It develops multimodal conversational models and a large styled-dialogue dataset, reporting strong human and benchmark results.

1 Introduction

  • The motivating goal is natural-language communication about the world that reflects image meaning as well as conversational mood and style.Source: §1 · p. 1
  • The study fuses Transformer dialogue and response encoders with ResNet image encoders in generative and retrieval models, and evaluates them automatically and with crowdworkers.It also compares modality-fusion approaches and conducts ablations.Source: §1 · p. 1
  • Image-Chat provides 202k images and 401k utterances with 215 style traits for training and evaluating engaging grounded-dialogue models.The dataset is publicly available in ParlAI.Source: §1 · p. 1
  • The best fused retrieval models are preferred to human conversationalists 47.7% of the time, and the paper reports state-of-the-art results on IGC.The authors also find a significant gap between state-of-the-art retrieval and generative models on this task.Source: §1 · p. 1

2 Related Work

Prior work covers text-only dialogue, image captioning, and factual visual question-answering or dialogue, but these tasks do not address engaging open-ended conversation grounded in images. The paper positions its work among the smaller set of vision-grounded dialogue studies and contrasts its dataset with IGC.

2 Related Work

  • Image captioning produces a single descriptive utterance, while this work focuses on image-grounded dialogue rather than single-turn captions.Prior captioning work also incorporated style to make captions more engaging.Source: §2 · p. 2
  • Text-only dialogue models are not grounded in perception, and visual question-answering and visual dialogue emphasize factual questions rather than natural conversation.The cited tasks assess image perception through questions about image contents.Source: §2 · pp. 1–2
  • Earlier vision-grounded dialogue includes video-game dialogue and sentiment-based image dialogue; IGC uses 4k eventful images without providing training data.Unlike IGC, Image-Chat provides train, validation, and test sets over 202k non-overlapping images and studies general image dialogue rather than only event questions and responses.Source: §2 · p. 2
  • The paper measures engagement by asking people which of two speakers they find more engaging.This follows the approach used in other cited work.Source: §2 · p. 2

3 Image-Chat

Image-Chat is a crowdworked dataset of image-grounded conversations in which two speakers respond to the image and dialogue history while using assigned styles. It combines a large collection of varied style traits with monitored data collection and reported checks of image and style fit.

3 Image-Chat

  • Crowdworkers create consecutive-turn dialogue tuples containing an image, separate speaker styles, and the conversation, responding to the image and dialogue history engagingly.Instructions specify that style describes the speaker, not image content.Source: §3 · pp. 2–3
  • The dataset uses 215 positive, neutral, and negative style traits, assigning different traits to speakers A and B for each conversation.Source: §3 · p. 2
  • Images are randomly selected from the YFCC100M dataset.Source: §3 · p. 2
  • Figure 1 shows training-set samples of conversations about images, with each speaker assigned a provided style.Source: Figure 1 · p. 3
  • Data collection includes manual monitoring and banning poor performers; a subset check found 92.8% of utterances fit the image, 83.1% the style, and 80.5% both.The authors note that not every utterance is expected to directly reference an image property or invoke a style.Source: §3 · p. 3
  • Image-Chat contains over 400k utterances, compared with PersonaChat's 162k utterances and IGC's 4k validation and test examples.The dataset statistics are presented in Table 1.Source: Table 1 · p. 3, §3 · p. 3

4 Models

The paper presents retrieval and generative dialogue models that combine image, dialogue-history, and style representations. Retrieval scores candidate responses, while generation uses a Transformer decoder to produce a response.

4 Models

  • Both model types encode image, dialogue history, and style; retrieval combines these representations to score candidate responses, while generation uses a Transformer decoder to produce tokens.The retrieval architecture also encodes candidate responses; the generative architecture applies beam search.Source: §4 · p. 3
  • The models use pretrained image features and Transformer dialogue encoders, with Reddit next-utterance pretraining used to initialize the candidate encoder.The image representation is projected to 500 dimensions for retrieval and uses a linear layer for generation; the dialogue Transformer has four layers, 300 hidden units, and six attention heads.Source: §4 · p. 4

4.1 Retrieval Models

  • Retrieval compares a sum combiner with a self-attention combiner that reweights image, style, and dialogue representations before scoring candidate responses.Candidate-response encoder weights may be shared with or separate from the dialogue encoder; training uses other examples’ gold responses as negatives, and human evaluation searches 356k training-set candidates.Source: §4.1 · pp. 4–5

4.2 Generative Models

  • Generation projects image features into the decoder’s token-encoding dimension, appends them to the encoder output, and prepends style to the dialogue history.The model is trained with a batch size of 32 and learning rate .0001, then uses beam search with beam size 2 and trigram blocking at inference.Source: §4.2 · p. 5

5 Experiments

Experiments evaluate the models on IMAGE-CHAT and the IGC response task using automatic metrics and human judgments. They compare model configurations, modality ablations, and human-versus-model engagingness.

5.1 Automatic Evaluation on IMAGE-CHAT

  • On IMAGE-CHAT, the best ResNeXt-IG-3.5B model achieved 50.3% R@1 versus 40.6% for the best ResNet152 model, a 9.7% full-task advantage that rose to 19.5% on turn 1.MM-Sum outperformed MM-Att, which scored 49.3% on the full task; separate candidate and dialogue-history text encoders also worked better than shared weights.Source: §5.1 · p. 5
  • Across turns, ablations removed image, dialogue history, or style and reported retrieval R@1/100 and generative ROUGE-L, both per turn and averaged.The supplied passage describes the ablation design but does not give its results.Source: §5.1 · p. 5, Table 3 · p. 6

5.2 Human Evaluations on IMAGE-CHAT

  • Human evaluations used blind pairwise choices between a human and model utterance, with image, dialogue history, and style available to evaluators.The style was the same for both utterances, avoiding an advantage from different style conditioning.Source: §5.2 · pp. 5–6
  • 47.7% overall win rate against humans for TRANSRESNETRET with ResNeXt-IG-3.5B was significant; its turn-1 win rate was 49.4% and not significant.The other two tested models were significantly outperformed by humans on turn 1.Source: §5.2 · p. 6, Figure 4 · p. 8

5.3 Transfer to the IGC Task

  • The IGC response task supplies an image, context utterance, and question; the same trained models were applied without fine-tuning.IMAGE-CHAT contained 40,076 training questions, or 11.3% of its data, under the paper’s question heuristic.Source: §5.3 · pp. 6, 8
  • The prior IGC model’s predictions and human-evaluation code were unavailable, so the authors attempted to reproduce its evaluation setup.They planned to make their reproduced setup publicly available upon acceptance.Source: §5.3 · p. 8
  • On the IGC test set, the best TRANSRESNETGEN model achieved a BLEU-4 score of 2.30, outperforming the model from Mostafazadeh et al. (2017), which scored 1.49.Source: §5.3 · p. 8
  • 62.9% versus 54.2% of the human-authored response score: the model narrowed the human–model gap in IGC human evaluation.Annotators rated responses on a 1–3 scale; the reported percentage averaged about 2,000 questions.Source: Figure 5 · p. 8, §5.3 · p. 8

6 Conclusion

The paper presents image-grounded chit-chat as a route to engaging social dialogue, with its best model preferred over human conversations nearly half the time. The authors identify short human evaluations and the gap between retrieval and generative models as challenges for future work.

6 Conclusion

  • The best proposed model’s image-grounded dialogues were preferred over fellow humans’ dialogues 47.7% of the time.The conclusion attributes this result to the creation of the IMAGE-CHAT dataset.Source: §6 · p. 8
  • Retrieval models outperformed their generative versions, leaving the gap between them an important challenge.Source: §6 · p. 8
  • Human evaluations covered short conversations, while initial investigations suggest the model can extend to longer chats, which the authors say should be studied further.The authors also identify combining engagingness with world knowledge, personal interests, and task proficiency as a next challenge.Source: §6 · p. 8

A More Details of IGC Evaluations

The appendix describes IGC human-evaluation choices, including style-trait conditions, BLEU reporting, and the recoverable test-image subset; it also identifies instruction panes for the second and third dialogue rounds.

A More Details of IGC Evaluations

B IMAGE-CHAT Human Annotation Setup

C IMAGE-CHAT Human Evaluation Setup

Figure 8 identifies an instruction pane used when crowdworkers collected IMAGE-CHAT evaluations.

C IMAGE-CHAT Human Evaluation Setup

D IGC Human Evaluation Setup

The appendix identifies an IGC evaluation instruction pane and tables of highly and poorly rated IGC examples.

D IGC Human Evaluation Setup

  • Figure 9 shows an instructions pane for crowdworkers collecting IGC evaluations.Source: Figure 9 · p. 13

E Highly Rated Examples from IGC

  • Table 5 contains highly rated IGC test-split examples whose TRANSRESNETRET MM-Sum responses received the highest human rating, 3.Source: Table 5 · p. 14

F Poorly Rated Examples from IGC

  • Table 6 contains low-rated IGC test-split examples whose TRANSRESNETRET MM-Sum responses received the lowest human rating, 1.Source: Table 6 · p. 15

G Longer Chat Example

Figure 10 presents a long-form exchange between a style-conditioned model and an unstylized human, both discussing climbing a mountain. The conversation moves from admiration and climbing claims to skiing and whether the human could climb it.

G Longer Chat Example

  • The example labels speaker A as TransResNetRET MM-Sum with the “Extraordinary” style and speaker B as a human without style.Figure 10 describes the model as receiving a style while the human does not.Source: G Longer Chat Example · p. 16, Figure 10
  • A describes the mountain as beautiful and recounts climbing it, including a claim of completing the climb in 7 hours and 10 minutes.A also calls for hard will and determination to scale the peak.Source: G Longer Chat Example · p. 16
  • B admires the mountain, wonders about skiing, and says they do not think they could climb it after A challenges them.B also says they would need a lift to ski down.Source: G Longer Chat Example · p. 16

H Additional Ablation Results

Table 7 describes ablations of the best TransResNet generative model on Image-Chat. It compares removal of image, dialogue-history, and style-conditioning modalities using generation metrics across turns.

H Additional Ablation Results

  • Table 7 reports the ablation results for variants of the best TransResNet generative model using the ResNeXt-IG-3.5B image encoder.The compared variants remove image, dialogue history, or style conditioning.Source: Table 7 · p. 16
  • The table reports F1 and BLEU-4 for dialogue turns 1, 2, and 3 independently, along with averages over all turns.Source: Table 7 · p. 16
  • The caption specifies the ablation scope and reported metrics but does not provide the result values.Source: Table 7 · p. 16
Loading 1811.00945v2…