Source-linked AI summary
Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog
Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, Cheng Niu, Jie Zhou
TL;DR
AVSD asks systems to generate correct, fluent responses grounded in video, audio, and text, while existing methods may not fully benefit from joint multimodal representations. The paper uses a GPT-2-initialized universal multimodal transformer with multitask learning, reports state-of-the-art AVSD performance, and notes a limitation in reconstructing masked video features from dialogue history alone.
Problem
AVSD requires correct, fluent responses grounded in text, video, and audio, while existing independent encoders may not fully benefit from joint multimodal representations.
Source: Introduction · p. 1Method
A GPT-2-initialized universal multimodal transformer uses response language modeling, video-audio sequence modeling, and caption language modeling to learn joint representations and generate responses.
Source: Introduction · p. 2, Model Overview · p. 3, Multi-task Learning · p. 3Results
The system achieves state-of-the-art AVSD performance, outperforming existing methods and other DSTC8-AVSD teams by a large margin.
Source: Introduction · p. 2Takeaways & Limitations
The approach is presented as learning more accurate cross-modal representations while generating informative responses.
Source: Conclusion and Future Work · p. 6Feature regression performs worse on text + video without caption or summary, where dialogue history alone is insufficient to rebuild masked video features.
Source: Analysis and Discussion · p. 5
Abstract
from arXiv · showhide
Audio-Visual Scene-Aware Dialog (AVSD) is a task to generate responses when chatting about a given video, which is organized as a track of the 8th Dialog System Technology Challenge (DSTC8). To solve the task, we propose a universal multimodal transformer and introduce the multi-task learning method to learn joint representations among different modalities as well as generate informative and fluent responses. Our method extends the natural language generation pre-trained model to multimodal dialogue generation task. Our system achieves the best performance in both objective and subjective evaluations in the challenge.
Introduction
AVSD requires correct, fluent responses grounded in video, audio, and text, while prior methods’ independent encoders may not fully benefit from joint multimodal representations. The paper proposes a GPT-2-initialized universal multimodal transformer with multitask learning and reports state-of-the-art AVSD performance.
Introduction
- Earlier scene-aware dialogue work includes text-grounded conversations, text-based adventure agents, and the introduction of AVSD.The cited examples are Zhou, Prabhumoye, and Black (2018), Jack Urbanek (2019), and Alamri et al. (2018).Source: Introduction · p. 1
- AVSD aims to generate correct, fluent responses by understanding text, video, and audio, making it more challenging than image-based or text-grounded dialogue.The task is described as generating responses grounded in the given scene.Source: Introduction · p. 1
- Prior approaches use independent modality encoders followed by attention-based fusion, which may not fully benefit from joint multimodal representations.The paper proposes a universal transformer to encode modalities and generate responses together.Source: Introduction · pp. 1–2
- The proposed model initializes a universal multimodal transformer with pre-trained GPT-2 and uses multitask learning for joint representations and informative, fluent responses.Its tasks are response language modeling, video-audio sequence modeling, and caption language modeling.Source: Introduction · p. 2, Figure 2 · p. 2
- The authors identify using pre-trained natural language generation models for multimodal dialogue generation as a first contribution.They also describe integrating multimodal features in one encoder and introducing multitask learning.Source: Introduction · p. 2
- The system achieves state-of-the-art AVSD results, outperforming existing methods and other DSTC8-AVSD teams by a large margin.The contribution statement reports this outcome without specifying metric values.Source: Introduction · p. 2
Related Work
Related work includes dialogue grounded in documents or knowledge graphs, image-based visual tasks, and multimodal systems. The paper extends pre-trained language-model transfer learning to multimodal generation and proposes self-supervised learning for video representation.
Related Work
- Prior systems ground dialogue in documents or knowledge graphs, while AVSD uses video and audio as background knowledge.Visual QA concerns image questions, and visual dialogue generates responses from an image and dialogue context.Source: Related Work · p. 2
- The paper extends transfer learning with pre-trained language models from generative dialogue to multimodal language generation.It also proposes self-supervised learning for better video representation.Source: Related Work · p. 2
Methodology
The methodology formulates response generation around video, audio, caption, and dialogue context, using a GPT-2-based multimodal Transformer. It trains with response, video-audio sequence, and caption language-modeling tasks across several input settings.
Task Formulation
- Responses are modeled conditioned on video, audio, a caption combining the summary and video caption, dialogue history, and the current question.The response probability is factorized over its words.Source: Task Formulation · pp. 2–3
Model Overview
- A 12-layer decoder-only Transformer based on GPT-2 combines text, video-audio features, positional encodings, and segment embeddings.Word tokens use WordPieces; segment embeddings distinguish video, caption, and the two speakers.Source: Model Overview · p. 3
- Video segments use I3D-RGB and I3D-flow features, while synchronized audio segments use VGGish features projected into the text embedding space.The video and audio features are concatenated before projection.Source: Model Overview · p. 3, Figure 3 · p. 3
Multi-task Learning
- Training combines response language modeling, video-audio sequence modeling, and caption language modeling as three fine-tuning tasks.Video-audio sequence modeling regresses the Transformer output to the next continuous video-audio feature using an L2 loss.Source: Multi-task Learning · p. 3
Different Settings
- The text-only setting uses response language modeling, while text plus video uses all three tasks; caption-free tests either omit captions or generate one before responding.The recaptioning variant trains with all three tasks and conditions responses on the generated caption, video-audio input, and dialogue history.Source: Different Settings · p. 4
Experiments
Experiments evaluate AVSD data, baseline systems, objective and human ratings, and multiple input and training settings. The model outperforms prior systems in reported text-only and text-plus-video comparisons, while its results depend on available video information and training setup.
Datasets
- The AVSD dataset contains dialogues of questions and answers about videos between two speakers, with a video description for each video.The experiments use I3D Mixed 5c features of 2048 dimensions, VGGish audio embeddings of 128 dimensions, and 7,659 training, 1,787 validation, and 1,710 test dialogues.Source: Datasets · p. 4
Baselines
- The compared baselines are Naive Fusion, Hierarchical Attention, and MTN, with MTN described as the pre-challenge state-of-the-art system.Hierarchical Attention was used by the DSTC7-AVSD winner; Naive Fusion is the organizers' multimodal baseline.Source: Baselines · p. 4
Metrics
- Objective evaluation uses BLEU, METEOR, ROUGE-L, and CIDEr, while crowd-sourced human ratings assess correctness, naturalness, informativeness, and appropriateness on five levels..Source: Metrics · p. 5
Experiment Settings
- Experiments initialize from GPT2-base, use up to three dialogue-history turns, and decode with beam search of size five, maximum length 20, and length penalty 0.3.The model uses hidden size 768, batch size 32, and Adam with learning rate 6.25e-5.Source: Experiment Settings · p. 5
Experimental Results
- Text-only results improve BLEU-4 by 0.069 and CIDEr by 0.185 over Hierarchical Attention; the model scores 3.934 in human ratings, the highest among DSTC8 submissions.The ground-truth references score 4.000.Source: Experimental Results · p. 5
- Text + video improves BLEU4 by 0.056 and CIDEr by 0.216 over MTN; video-audio sequence modeling adds 0.011 BLEU-4 and 0.026 CIDEr.In DSTC8 results, this method has a slightly lower BLEU4 value but improves CIDEr by 0.014.Source: Experimental Results · p. 5
- Without captions or summaries, performance is lower than with text + video, but the model beats DSTC7-AVSD Team 9, while multi-task learning lowers performance on almost all metrics.This setting uses only video-audio information and dialogue history and is described as most similar to actual scene-aware dialogue.Source: Experimental Results · p. 5
Analysis and Discussion
- Feature regression improves text + video results but lowers results without captions or summaries; the authors attribute this to difficulty rebuilding masked video features from dialogue history.They also identify limited video-information extraction as a reason the method does not work in the no-caption setting.Source: Analysis and Discussion · p. 5
Conclusion and Future Work
The paper concludes that it proposes a multimodal dialogue-generation model based on a pre-trained language model and multi-task learning. It plans to investigate additional video features and training tasks, and to extend the methods to other visual-language tasks.
Conclusion and Future Work
- The proposed model combines a pre-trained language model with multi-task learning to learn joint multimodal representations and generate informative responses.The authors plan to use more video features, including ResNet features, and explore additional training tasks to improve joint video-text understanding.Source: Conclusion and Future Work · p. 6
- The authors hope to extend these methods to video captioning, image captioning, and visual dialog.Source: Conclusion and Future Work · p. 6
- The authors identify improving joint video-text understanding as a future direction through more video features and training tasks.Source: Conclusion and Future Work · p. 6