PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems

Hung Le, Doyen Sahoo, Nancy F. Chen, Steven C. H. Hoi

arXiv:1907.01166v1cs.CL

TL;DR

Video-grounded dialogue requires systems to relate dialogue context and human queries to information across video frames and modalities. The paper proposes MTN, combining multimodal attention with query-aware attention and simulated token-level decoding during training. The authors report strong DSTC7 performance and performance on a visual-grounded dialogue task.

  • Problem

    Video-grounded dialogue requires processing information across multiple frames and modalities, while prior RNN and sequence-to-sequence approaches are described as ineffective for complex long-term dependencies.

    Source: §1 · pp. 1–2

  • Method

    MTN uses Transformer attention to encode and combine text, visual, and audio inputs, with query-aware auto-encoder attention on non-text features and simulated token-level decoding during training.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 4

  • Results

    The authors report that MTN outperforms the reported DSTC7 baseline and other submission entries, and achieves strong performance after adaptation to visual dialogue.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that MTN can generate good conversational responses in multimodal settings and can be adapted to visual dialogue.

    Source: §6 · p. 9

    The visual-dialogue comparison details are not clearly stated.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Developing Video-Grounded Dialogue Systems (VGDS), where a dialogue is conducted based on visual and audio aspects of a given video, is significantly more challenging than traditional image or text-grounded dialogue systems because (1) feature space of videos span across multiple picture frames, making it difficult to obtain semantic information; and (2) a dialogue agent must perceive and process information from different modalities (audio, video, caption, etc.) to obtain a comprehensive understanding. Most existing work is based on RNNs and sequence-to-sequence architectures, which are not very effective for capturing complex long-term dependencies (like in videos). To overcome this, we propose Multimodal Transformer Networks (MTN) to encode videos and incorporate information from different modalities. We also propose query-aware attention through an auto-encoder to extract query-aware features from non-text modalities. We develop a training procedure to simulate token-level decoding to improve the quality of generated responses during inference. We get state of the art performance on Dialogue System Technology Challenge 7 (DSTC7). Our model also generalizes to another multimodal visual-grounded dialogue task, and obtains promising performance. We implemented our models using PyTorch and the code is released at https://github.com/henryhungle/MTN.

1 Introduction

Video-grounded dialogue systems must connect dialogue context and queries to information distributed across video frames and modalities. The paper introduces MTN and evaluates it on DSTC7 and visual-grounded dialogue.

1 Introduction

  • Video-grounded dialogue requires responses to reflect both relevant dialogue context and a query’s relevance to the video.Videos combine information across frames, including background noise, speech, and actions, while systems must process text, visual, and audio inputs.Source: §1 · p. 1
  • Prior multimodal dialogue approaches use RNNs and sequence-to-sequence architectures, which are described as ineffective or inefficient for complex long-term information across modalities.Some prior work also used query-aware attention to focus on relevant features.Source: §1 · p. 1
  • MTN applies multi-head attention across modalities, adds query-aware auto-encoder attention to non-text features, and simulates token-level decoding during training.The paper presents these components as ways to model video sequences, improve reasoning over non-text features, and improve generated responses.Source: §1 · p. 2
  • The authors evaluate MTN on DSTC7 and report that it generalizes to the visual-grounded dialogue task.Their evaluation includes automatic metrics, ablations, and qualitative analysis.Source: §1 · p. 2

2 Related Work

Related work covers dialogue systems grounded in text or external knowledge, multimodal tasks such as visual question answering, and visual dialogue grounded in images. The paper distinguishes VGDS by grounding generated dialogue responses in videos while tracking dialogue context.

2 Related Work

  • Text-grounded dialogue systems use external sources such as encyclopedias, social networks, or recommendation sites, while Transformer-based dialogue work cited here considers text-grounded knowledge [Vaswani et al., 2017].The paper contrasts these settings with video-grounded knowledge that includes multimodal information.Source: §2 · p. 2
  • Image and video captioning and visual question answering process visual features, but the cited methods are restricted to specific queries and do not maintain dialogue context.The paper focuses on generating dialogue responses rather than selecting from candidate answers.Source: §2 · p. 2
  • Visual dialogue tracks conversational context but grounds knowledge in images, whereas VGDS grounds dialogue in videos spanning multiple frames and modalities [Das et al., 2017a,b; Kottur et al., 2018].The paper identifies the larger feature space and multiple modalities as added complexity.Source: §2 · pp. 2–3

3 Multimodal Transformer Networks

MTN encodes text and video inputs, then uses decoder attention and query-aware auto-encoder layers to generate dialogue responses. Its components separately handle sequential encoding, reasoning over encoded inputs, query-aware attention to non-text video features, and simulated token-level decoding during training.

3 Multimodal Transformer Networks

  • MTN combines encoder, decoder, and auto-encoder layers to fuse text and video representations and generate a response to a video-grounded query.The model uses positional encodings for text tokens and video frames and is trained end-to-end.Source: §3 · p. 3

3.1 Encoder Layers

  • Video encoders transform sliding-window audio and visual features into d-dimensional sequences and add positional encodings for video-frame order.The feature extractors are pre-trained and kept fixed during training.Source: §3.1 · p. 3

3.2 Decoder Layers

  • Decoder layers attend separately to the offset target, dialogue history, caption, query, and non-text video features, allowing attention to differ across inputs.The decoder uses multi-head attention on encoded inputs; the paper also experiments with concatenating inputs into one attention block.Source: §3.2 · pp. 3–4

3.3 Auto-Encoder Layers

  • Because decoder attention may not fully capture query–video interaction, the auto-encoder adds query-aware attention to visual and audio features.The authors report that Table 4 rows C-5 and D show the auto-encoder is more effective at capturing relevant video features.Source: §3.3 · p. 4, Table 4 · p. 5

3.4 Generative Network

  • During training, the model sometimes crops the target sequence at a random position to simulate token-level decoding and reduce train–test input mismatch.The procedure applies to decoder target sequences, not to the query auto-encoder.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate MTN on DSTC7 video-grounded dialogues and VisDial, with tests of token-level decoding simulation and model ablations. The reported results include strong benchmark performance and generalization to visual dialogue.

4.1 Data

  • The DSTC7 experiments use multimodal dialogues grounded on Charades videos, with audio and visual features and video captions.Visual features are 2048-dimensional I3D flow features and audio features are 128-dimensional Audio Set VGGish.Source: §4.1 · p. 5

4.2 Training

  • Training uses a target-sequence log-likelihood objective conditioned on dialogue history, query, video features, and caption, with regenerated-query likelihood added when QAE is used.Base and Large configurations use 6 and 10 layers, respectively; the simulation probability is 0.5.Source: §4.2 · pp. 5–6

4.3 Video-Grounded Dialogues

  • Large MTN outperforms the best previously reported DSTC7 challenge models across all four evaluation metrics.Both Base and Large MTN outperform the baseline on all metrics; Base outperforms most prior results.Source: §4.3 · p. 6
  • Token-level decoding simulation improves performance for most tested probabilities between 0 and 1, while p = 1 degrades performance.The figure caption reports more significant BLEU4 improvement at p = 0.4 to 0.6.Source: §4.3 · p. 6, Figure 4 · p. 6
  • The query-aware auto-encoder improves BLEU4 and CIDEr with audio features, and improves all metrics with caption and video features.The auto-encoder structure is also reported as superior to stacked encoder layers.Source: §4.3 · pp. 6–7

4.4 Visual Dialogues

  • On VisDial, MTN generalizes to candidate ranking and outperforms retrieval-based approaches in NDCG without task-specific fine-tuning.Some leaderboard submissions have higher NDCG, but their approaches are not clearly detailed for comparison.Source: §4.4 · p. 7, Table 2 · p. 7

5 Qualitative Analysis

MTN outperforms the Hori et al. (2018) baseline across dialogue turns, while qualitative examples show more accurate responses describing individual and sequential actions. The supplied captions also identify test-set objective comparisons and an audio/visual ablation, but do not provide their cell values.

5 Qualitative Analysis

  • MTN outperforms Hori et al. (2018) at every dialogue turn position, with stronger performance generally at one-, two-, and three-turn lengths.The passage notes that performance could be further improved for longer dialogues.Source: §5 · p. 7, Figure 5 · p. 7
  • MTN responses are described as more accurate than the baseline, including descriptions of single actions and sequences of actions.Examples include “cleaning the table” and walking to a closet before removing a jacket.Source: §5 · p. 7, Figure 6 · p. 8
  • Table 3 reports better test-set objective measures for the proposed approach than the baselines and DSTC7 submissions.The supplied caption does not specify metric values.Source: Table 3 · p. 8
  • The ablation table compares Base models using audio features, visual features, or both, but its caption gives no outcome values.Audio features are identified as VGGish and visual features as I3D-Flow.Source: Table 4 · p. 8

6 Conclusion

The conclusion presents MTN as a multi-head attention-based network for multimodal conversational responses, reporting strong DSTC7 results and performance on a visual dialogue task. It identifies pretrained embeddings as a possible improvement.

6 Conclusion

  • MTN, a multi-head attention-based neural network, generates conversational responses in multimodal settings.The conclusion reports that it outperforms the reported baseline and other DSTC7 submissions.Source: §6 · p. 9
  • The authors report that MTN outperforms the reported baseline and other DSTC7 submission entries.Source: §6 · p. 9
  • The authors adapted MTN to a visual dialogue task and report excellent performance.Source: §6 · p. 9
  • Adding pretrained embeddings such as BERT or image-grounded word embeddings is proposed as a possible improvement to semantic understanding.Source: §6 · p. 9

A.1 Video-Grounded Dialogues

The video-grounded dialogue setup uses a 6,175-token vocabulary and batches examples by approximate sequence lengths. Training uses batches of 32.

A.1 Video-Grounded Dialogues

  • The video-grounded dialogue vocabulary contains 6,175 unique tokens, including four special tokens, after retaining training tokens occurring more than once.The special tokens are ⟨eos⟩, ⟨sos⟩, ⟨pad⟩, and ⟨unk⟩.Source: §A.1 · p. 11
  • Examples are batched by approximate sequence lengths, ordered by dialogue history, video caption, question, and target sequence lengths.Source: §A.1 · p. 11
  • Training uses a batch size of 32.Source: §A.1 · p. 11

A.2 Visual-Grounded Dialogues

The visual-grounded dialogue setup uses about 4,000 COCO-like image dialogues from Flickr, a frequency-filtered vocabulary, and bottom-up image features.

A.2 Visual-Grounded Dialogues

  • The test-std v1.0 set contains about 4,000 dialogues grounded in COCO-like images collected from Flickr.Source: §A.2 · p. 11
  • The vocabulary contains 13,832 unique tokens after retaining training tokens occurring at least three times.Source: §A.2 · p. 11
  • Each image is represented by 36 2,048-dimensional bottom-up attention feature vectors from Faster R-CNN pretrained on Visual Genome.Source: §A.2 · p. 11

B Additional Experiment Results

Additional experiments compare text-only MTN variants with different dialogue-history lengths and with either separate or concatenated source-sequence attention. Separate attention improves response quality, while using all possible dialogue turns achieves the best results among the tested history lengths.

B Additional Experiment Results

Related papers

Loading 1907.01166v1…