Source-linked AI summary
Learning Video Context as Interleaved Multimodal Sequences
Kevin Qinghong Lin, Pengchuan Zhang, Difei Gao, Xide Xia, Joya Chen, Ziteng Gao, Jinheng Xie, Xuhong Xiao, Mike Zheng Shou
TL;DR
Narrative videos require understanding characters, situations, and events over time, while many video models prioritize concise clips and atomic objects or actions. MovieSeq represents videos as interleaved multimodal sequences and demonstrates effectiveness and flexibility across six diverse datasets.
Problem
Narrative-video understanding must handle character identity, situations, and event dependencies, while top-performing video models typically prioritize concise clips and atomic objects or actions.
Source: §1 · p. 2Method
MovieSeq represents narrative videos as interleaved multimodal sequences and trains a decoder-only multimodal language model on corresponding instruction data.
Source: §1 · pp. 2–3Results
MovieSeq demonstrates effectiveness and flexibility across six diverse datasets; on Movienet, it surpasses TDM Decoder by 2.3 in F1-score.
Source: §1 · p. 3, §5.2 · p. 10, §6 · p. 14Takeaways & Limitations
The Movienet result indicates that a generative language model can also perform well as a discriminative model.
Source: §5.2 · p. 10MovieSeq does not incorporate novel architectural designs and instead restricts visual branches, for example by using CLS tokens.
Source: §6 · pp. 14–15
Abstract
from arXiv · showhide
Narrative videos, such as movies, pose significant challenges in video understanding due to their rich contexts (characters, dialogues, storylines) and diverse demands (identify who, relationship, and reason). In this paper, we introduce MovieSeq, a multimodal language model developed to address the wide range of challenges in understanding video contexts. Our core idea is to represent videos as interleaved multimodal sequences (including images, plots, videos, and subtitles), either by linking external knowledge databases or using offline models (such as whisper for subtitles). Through instruction-tuning, this approach empowers the language model to interact with videos using interleaved multimodal instructions. For example, instead of solely relying on video as input, we jointly provide character photos alongside their names and dialogues, allowing the model to associate these elements and generate more comprehensive responses. To demonstrate its effectiveness, we validate MovieSeq's performance on six datasets (LVU, MAD, Movienet, CMD, TVC, MovieQA) across five settings (video classification, audio description, video-text retrieval, video captioning, and video question-answering). The code will be public at https://github.com/showlab/MovieSeq.
1 Introduction
Narrative videos require understanding characters, situations, and events over time, but existing approaches are often task-specific or inflexible. MovieSeq represents video context as interleaved multimodal sequences and trains a decoder-only multimodal language model on instruction-following data to support diverse tasks.
1 Introduction
- Narrative videos challenge models to identify characters [87], understand situations, and track event dependencies [53] over time.Many top-performing video perception models instead prioritize concise videos and atomic objects or actions [83] [?,61,65] [31].Source: §1 · p. 2
- Prior work advances video recognition, retrieval, and question-answering [53], but remains task-specific and may require tailored designs such as external-knowledge modules.The paper identifies the diversity of video contexts and tasks as a broader challenge for general-purpose understanding.Source: §1 · p. 2
- Single-visual-input methods [38] [39] [49] [95] and structured few-shot approaches [37] [79] remain inflexible for narrative videos with rich contexts.The paper motivates a more flexible way to handle multimodal contexts within videos.Source: §1 · p. 2
- MovieSeq unifies images, subtitles, plots, and video history as interleaved multimodal sequences that a language model processes for narrative video understanding.The approach also provides solutions for converting existing video datasets into interleaved multimodal instruction-following data and trains a decoder-only multimodal language model on that data.Source: §1 · pp. 2–3
- MovieSeq is evaluated on six benchmarks across video classification, audio description, character identification, retrieval, captioning, and question-answering, and enables free-form interleaved multimodal video instructions.The benchmarks are LVU, MAD, Movienet, CMD, TVC [35], and MovieQA.Source: §1 · p. 3
2 Related Work
Narrative video understanding extends beyond object and action recognition, and prior work has addressed its contextual challenges through specialized methods. MovieSeq aims to unify modalities and tasks in a generative model using interleaved multimodal instruction tuning.
2 Related Work
- Prior narrative-video work targets characters, relationships, emotions, dialogue, and storylines, using temporal alignment, memory, sound, and external character images for task-specific improvements [8] [34].Visual-only work has also improved architectures for efficient long-form recognition, while benchmarks cover contextual aspects [35] [89].Source: §2 · p. 3
- Large multimodal models commonly project images or videos into textual embedding spaces and align them with LLMs through visual instruction tuning [17] [38] [39] [49] [95], while other work explores interleaved vision-text input [37] [79].Related efforts also target spatial understanding [14] [93] and memory-efficient processing of long videos [42] [69].Source: §2 · p. 4
- Because most methods focus on individual tasks with tailored architectures, MovieSeq seeks one generative model for diverse modalities and tasks, using interleaved multimodal instruction tuning with an open-source model such as Llama2 [76].Prior work [44] [92] used GPT-4V [51] to convert video streams into textual documents; narrative videos also combine character photos, dialogue, and external metadata.Source: §2 · p. 4
3 Challenges in Video Contexts
Narrative video understanding requires models to connect dialogue with visuals, link events across time, and integrate external knowledge. Although movies illustrate these challenges, they also apply to other video types.
Situational Dialogue
- Models must interpret dialogue alongside visuals, associating what is said with the scene to produce contextually informative narration.For example, a girl saying she has arrived at her grandmother’s ranch should be narrated as her arrival there.Source: §3 · p. 4
Event Dependency
- Models must link fine-grained events across separate clips, using causal dependencies to connect past, present, and future in a storyline.Recognizing an old man retrieving a flashlight as preparation to illuminate something later illustrates this need.Source: §3 · pp. 4–5
External Knowledge
- Video understanding can benefit from integrating external knowledge, including character photos and textual plot documents, alongside movie metadata.Helpful context may include a movie’s title, genre, plot summary, or character glimpses from trailers.Source: §3 · p. 5
Broader Applicability
- These contextual challenges also arise beyond movies, including in social media videos with titles, descriptions, user avatars, and ASR transcripts.Movies serve as a representative example because of their descriptive storylines, informative character metadata, and extended durations.Source: §3 · pp. 4–5
4 Methodology
MovieSeq turns narrative-video understanding into context-conditioned multimodal instruction following, combining video with additional context and training the model to generate answers from that context and a question.
4.1 Overview
- MovieSeq uses Llama-2 with LoRA in all linear layers, freezes other parameters except input embeddings, and adds 0.12% trainable parameters; CLIP ViT-B/16 frame features pass through a temporal transformer.It extracts frame-wise [CLS] tokens rather than patch tokens, acknowledging a performance constraint, and uses temporal position encoding to capture relationships.Source: §4.1 · pp. 5–6
4.2 Interleaved Multi-Modal Instructions
- The instruction format pairs multimodal context Xctx with question Xq and answer Xa, allowing the same question to elicit different responses under different contexts.The model predicts Xa followed by <EOS> to mark termination.Source: §4.2 · p. 6, Table 1 · p. 6
- For character identification [24,35,67], instructions identify who appears before using character photos; long plots are sampled by query retrieval or clip-time ratio.Query retrieval uses sentence embeddings and cosine similarity to select the top-1 relevant paragraph; general queries use the clip’s normalized timestamp to locate a plot sentence and extract a window.Source: Table 1 · p. 6, §4.2 · p. 7
- Subtitle instructions combine video with temporally aligned subtitles from CMD and TVC [35], while history instructions pair past clips with narrations from MAD for current or future event understanding.Past narrations can come from annotations or the model’s previous predictions in recurrent inference.Source: §4.2 · p. 7
- For videos with multiple contexts, MovieSeq concatenates the contexts and uses ChatGPT-3.5 to rephrase templates for diversity and robustness.Rephrasing can turn a character-name query into a yes/no appearance question.Source: §4.2 · p. 8
4.3 Training Objectives
- Instructions train the model to predict answer Xa autoregressively, conditioned on multimodal context Xctx, question Xq, and preceding answer tokens.The loss is computed on Xa, whose sequence length is L.Source: §4.3 · p. 8
5 Experiments
Experiments test MovieSeq’s flexibility across benchmarks, the effects of multimodal context, multi-task instruction-tuning, and data-construction choices. The evaluations span six datasets and five video-understanding settings.
5.1 Datasets and Settings
- The six datasets cover video classification, character identification, audio description, video-text retrieval, video captioning, and question answering, with task-specific adaptations and metrics.For example, MovieSeq generates category or character names for classification, captions for CMD retrieval, and answers to multiple-choice MovieQA questions; LVU uses top-1 accuracy and MovieNet uses Recall, Precision, and F1-score.Source: §5.1 · p. 9
5.2 Main Results (Q1&Q2).
- LVU subtitles improve relation and speech assessment but reduce scene understanding, MovieNet actor-identification F1 exceeds TDM Decoder by 2.3, and MAD CIDEr is 4.9% higher than the best baseline.The LVU changes are +14.6% for relations, +25.4% for speech, and -10.9% for scene understanding. MAD reuses the MovieNet-trained model to predict character names and requires no additional pretraining.Source: Table 3 · p. 10, §5.2 · p. 10, Table 4 · p. 10
- On CMD, dialogue raises Geo. Mean by 29.74; MovieSeq also beats several TVC baselines, while MovieQA plot context outperforms the visual-only result.For MovieQA, MovieSeq (vid) scores 27.32 Acc., compared with 63.54 for (plot) and 66.41 for (vid, plot). The paper notes that plots provide distilled information for questions requiring reasoning beyond the video.Source: Table 6 · p. 11, Table 7 · p. 11, §5.2 · p. 12
5.3 Ablation Studies
- For MAD audio description, plots add 4.7 CIDEr, reliable oracle history improves metrics, and combining context designs improves metrics while reducing repetition.Past predicted narrations avoid repetition without changing the original audio-description metrics, whereas oracle narrations boost them; character identification from additional instructions is also necessary.Source: §5.3 · p. 12, Table 9 · p. 12
- Multi-task co-training improves average performance by 1.5 across MAD, MovieNet, and TVC, which were selected to balance dataset size across different tasks.The development set contains 1K samples from each dataset; MAD was about 200 times larger than LVU.Source: §5.3 · pp. 12–13, Table 10 · p. 13
- On MAD, adding narrations to history clips beats captions alone, more input clips improve both variants, and reliable plot sampling and longer context improve performance.Video alone changes little as history length increases; the plot experiment contrasts temporal estimated sampling with CLIP vision-text similarity filtering.Source: §5.3 · p. 13, Figure 4 · p. 13
5.4 Visualization.
- Visualizations show interleaved images, subtitles, plots, and video history helping MovieSeq associate characters with names, dialogue with relationships, plots with reasons, and prior events with later narration.In one example, dialogue links the name Max to the relationship “friends”; retrieved plot context supplies why Miley attends a birthday party as Hannah Montana.Source: §5.4 · pp. 13–14, Figure 5 · p. 14
6 Conclusion and Limitations
MovieSeq models narrative videos as interleaved multimodal sequences with external knowledge and additional modalities, supporting flexible instruction and addressing several video-context challenges. The paper reports effectiveness across six datasets while noting architectural and long-context limitations.
Conclusion
- MovieSeq uses interleaved multimodal sequences with external knowledge and additional modalities to support flexible instructions and address dialogue understanding, character identification, and event dependency.The paper also illustrates how to collect tuning data for each type of interleaved prompt.Source: §6 · p. 14
- The authors demonstrate MovieSeq’s effectiveness and flexibility across six diverse datasets.Source: §6 · p. 14
Limitations
- MovieSeq introduces no novel architectural designs and restricts visual branches, such as using CLS tokens instead of patch tokens, to save token length.Effectively modeling long context without affecting visual input remains future work.Source: §6 · p. 14