Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization

1. Introduction

• Video adaptation remains less explored than image-text adaptation, although learning from dynamic video is described as important for real-world visual understanding.Video requires modeling spatiotemporal clues such as changing actions and scenes.Source: §1 · p. 1
• Frame-wise 2D encoding can miss temporal motion needed to identify video behaviors and events, while long clips can require unacceptable LLM token and compute budgets.The cited example uses 1280 tokens for a 2.2-second VideoPoet clip.Source: §1 · pp. 1–2
• The representation alternates keyframes, which carry visual semantics, with motion vectors that describe their evolution, reducing tokens relative to consecutive-frame 3D encoding.This design also reuses visual knowledge from an image-only LLM while focusing on temporal information.Source: §1 · p. 2
• Video-LaVIT tokenizes keyframes and motion, detokenizes generated tokens to pixels, and trains the resulting sequence with text under a unified next-token objective.The method uses an established image tokenizer for keyframes and a spatiotemporal motion encoder for temporal context.Source: §1 · p. 2
• Video-LaVIT achieves very competitive performance across 13 multimodal benchmarks spanning image and video comprehension and zero-shot text-to-image and text-to-video generation.The reported evaluation covers both understanding and generation tasks.Source: §1 · p. 2

2. Related Work

• Image-text generative pre-training adapts visual inputs to LLMs and uses discrete visual tokenizers, but these methods cannot directly extend to video because of its higher computational cost.Adaptation modules include cross-attention, Q-Former, and linear projection; CM3Leon and LaVIT use discrete visual tokens.Source: §2 · p. 2
• Video comprehension has used masked and autoregressive language models, while mainstream video generation uses diffusion; LLM-based video methods face temporal-encoding and context-resource constraints.Prior language-model-based video generation and understanding approaches include Yan et al. (2021), Hong et al. (2023), and Kondratyuk et al. (2023).Source: §2 · p. 2
• Video-LaVIT leverages motion vectors and decomposes video into keyframes and motion, then concatenates encoded tokens with text tokens for unified generative pre-training.The tokenizers are learned by maximizing reconstruction of original inputs.Source: Figure 2 · p. 3, §2 · p. 3

3.1. Video Tokenization

• Video-LaVIT decomposes video into keyframes and motion vectors, tokenizing them as alternating visual and motion codes that reduce inter-frame redundancy while capturing temporal information.It uses MPEG-4 I-frames as keyframes and a motion-specific VQ-VAE tokenizer for motion vectors.Source: §3.1 · pp. 3–4

3.2. Video Detokenization

• The detokenizer first recovers a keyframe from visual tokens, then decodes subsequent frames conditioned on that keyframe and motion tokens.Its video denoising U-Net adds temporal convolution and attention; motion and keyframe features are supplied both as input conditions and through cross-attention.Source: §3.2 · pp. 4–5, Figure 3 · p. 4
• For long videos, Video-LaVIT progressively decodes clips and uses the previous clip’s last frame to initialize noise for the next keyframe, improving temporal consistency between clips.The last frame is reversed through DDIM sampling to form the next keyframe’s initial noise state.Source: §3.2 · p. 5

3.3. Unified Generative Modeling

• Video-LaVIT autoregressively models video, image, and text as discrete multimodal sequences, and its three-stage training proceeds from tokenizer and detokenizer training to generative pre-training and instruction tuning.After pre-training, it can perform multimodal comprehension and generation across modalities.Source: §3.3 · p. 5

4.1. Multimodal Understanding

• Video-LaVIT achieves the best results on all image question-answering datasets and leads on three of four comprehensive multimodal benchmarks.On ScienceQA-IMG it surpasses LLaVA-1.5 by 3.2%, despite that model using higher input resolution.Source: Table 1 · p. 6, §4.1 · p. 6
• Video-LaVIT achieves state-of-the-art accuracy on all three zero-shot video question-answering benchmarks, including a 2.5% gain over Video-LLaVA on MSVD-QA.Its relative score on MSRVTT-QA is second to Video-LLaVA by 0.2.Source: Table 2 · p. 6, §4.1 · p. 6
• On Perception Test Video-LaVIT has the highest zero-shot performance, and on EgoSchema it exceeds InternVideo by 5.2% in zero-shot QA accuracy.It evaluates 16 keyframes and intervening motion vectors spanning 64 seconds, compared with InternVideo’s use of up to 90 frames.Source: §4.1 · pp. 6–7, Table 3 · p. 7

4.2. Multimodal Generation

• Video-LaVIT significantly outperforms most text-to-video baselines using similar public datasets, leads MSR-VTT FVD, and outscores CogVideo and VideoPoet.VideoPoet is described as using a much larger dataset.Source: Table 4 · p. 7, §4.2 · p. 7

4.3. Qualitative Results

• Qualitative comparisons show text-to-video quality not far from Gen-2 and image-to-video results comparable to SVD, with examples of inferred motion and vivid movement.Figure 4 compares text-to-video and image-to-video generations; image-to-video generation is conditioned on the leftmost frame.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

4.2. Multimodal Generation

• In long-video generation, Video-LaVIT surpasses all baselines on KVD among specialists evaluated on 2,048 generated 64-frame videos.KVD compares the first 16 frames of short videos with subsets of the last 16 frames of long videos.Source: §4.2 · p. 7, Table 5 · p. 8

4.4. Ablation Study

• Without explicit motion tokenization, video question-answering accuracy decreases and text-to-video generation lacks temporal motion guidance.The ablation compares against independently tokenized sampled frames for understanding and a keyframe-conditioned image-to-video pipeline for synthesis.Source: §4.4 · p. 8, Table 6 · p. 9
• A small number of motion tokens suffices for high understanding and generation performance, while more tokens may add redundancy and duplicate IDs.Fewer motion tokens also allow more video clips as input under the same LLM context length; the noise constraint improves temporal consistency between decoded clips.Source: §4.4 · p. 9, Figure 5 · p. 9

• Video-LaVIT is a multimodal generative pre-training method for unified comprehension and generation of videos, images, and language.Source: §5 · p. 9
• The method decomposes videos into keyframes and motion vectors for more effective temporal modeling while reusing visual knowledge from image-only multimodal LLMs.Source: §5 · p. 9
• The paper reports quantitative and qualitative results verifying Video-LaVIT’s understanding and generative capabilities.Source: §5 · p. 9

Impact Statement

• The authors caution that the model’s reasoning and generative capabilities should be treated carefully, citing hallucination, misinformation, bias, and lack of alignment procedures as concerns.They specifically mention misinformation through personalized generation and harmful responses arising from data bias and missing alignment procedures.Source: Impact Statement · p. 9
• Table 7 examines the effect of motion-token count on zero-shot video understanding and generation.Source: Table 7 · p. 9
• Using public datasets for training may allow the research community to correct bias and harmful content in the training data.The authors hope this practice will help address safety and alignment issues in multimodal models.Source: Impact Statement · p. 9

A.1. Model Implementation Details

• The video tokenizer maps each keyframe to an average of 90 visual tokens using the off-the-shelf LaVIT tokenizer and ViT-G/14 of EVA-CLIP, with images and keyframes resized to 224×224.The visual codebook contains 16,384 entries.Source: §A.1 · p. 14
• Motion tokenization downsamples video to 6 fps, uses 24-frame clips, and quantizes motion into 135 discrete tokens.The normalized motion input is resized to 20×36; the learned motion codebook has 1,024 entries.Source: §A.1 · p. 14
• The detokenizer uses a motion-conditioning encoder to guide frame denoising in a 3D U-Net, with training followed by higher-resolution fine-tuning.It is trained first at 384×384 for 50k steps, then at 768×768 or 1024×576 for 10k more steps.Source: §A.1 · p. 14
• Generative pre-training uses Llama 2 7B initialized from LaVIT, with image-text, video-text, and text data mixed into multimodal input sequences.Source: §A.1 · p. 14

A.2. Pre-training Data

• Stage 1 trains the video tokenizer and detokenizer on WebVid-10M; a filtered 300k-clip InterVid subset is additionally used to improve detokenizer aesthetics, but not for the reported experimental checkpoint.The tokenizer and detokenizer do not rely on textual data.Source: §A.2 · p. 14
• Stage 2 mixes video, image, and text data for language-model pre-training, while Stage 3 instruction-tunes on 665k image-text and 100k video-text examples.The English text corpus is included to preserve language understanding, and understanding results use the Stage 3 model.Source: §A.2 · pp. 14–15

A.3. Training Settings

• Video-LaVIT adopts the same instruction-tuning setting as LLaVA v1.5.Source: §A.3 · p. 15
• Table 8 reports detailed training hyperparameters for the video tokenizer, detokenizer, and language model.Source: Table 8 · p. 15, §A.3 · p. 15
• The provided passages identify the hyperparameter table’s scope but do not state its individual settings.Source: Table 8 · p. 15, §A.3 · p. 15

A.4. Evaluation

• Image understanding uses eight question-answering and multimodal benchmarks, with VQA accuracy as the metric for question-answering datasets.The listed benchmarks are VQA v2, GQA, VizWiz, ScienceQA-IMG, MME, MMBench, SEED, and MM-Vet.Source: §A.4 · p. 15
• Video question answering is evaluated on MSVD-QA, MSRVTT-QA, and ActivityNet-QA using GPT-3.5-assistant accuracy and relative scores from 0 to 5.Source: §A.4 · p. 15
• Text-to-image generation is tested on 30,000 randomly selected MS-COCO validation samples using FID in Inception V3 feature space.Source: §A.4 · p. 15
• Text-to-video generation is measured on MSR-VTT and UCF-101, with 2,990 and 2,020 video-text pairs respectively.The passage identifies the datasets and sample construction; the accompanying metric descriptions include CLIPSIM, FVD, FID, and IS.Source: §A.4 · pp. 15–16
• Evaluation protocols differ slightly across papers, and the authors sought to use protocols matching or resembling those of most top-ranked methods.Source: §A.4 · p. 16

B.1. Multimodal Generation

• In text-to-image examples, Video-LaVIT shows competitive visual quality and better language understanding, including correctly rendering two green apples.The passage also describes capturing “sharp focus” in a portrait prompt.Source: §B.1 · p. 16
• Against Gen-2, Video-LaVIT produces generally comparable video quality and follows prompt details such as running and multiple pirate ships.The examples also show prompt details including moss, flowers, and autumn.Source: §B.1 · p. 16
• On unseen image prompts, Video-LaVIT produces more natural animal and motorcycle motions than SVD in the reported examples.The comparison attributes motion transfer to the decomposed video representation; the motorcycle example also avoids motion inconsistent with its tire direction.Source: §B.1 · p. 16
• Explicit noise constraints support temporal consistency across decoded clips, including maintained identities and coherent trajectories in the long-video examples.The examples include a dog and jeep retaining identity across clips and a consistent trajectory during large camera movement.Source: §B.1 · p. 16
• Without enhanced motion conditioning, the 3D U-Net receives only motion vectors and fails to recover motion such as the train and fish moving.The comparison presents enhanced motion conditioning as improving reconstruction fidelity.Source: §B.1 · pp. 16–17

B.2. Multimodal Understanding

• In image question answering, Video-LaVIT gives a reasonable answer with correct vehicle detail, identifying an SUV where LLaVA says van or minivan.The passage notes that the answer also includes a safety warning.Source: §B.2 · p. 17
• Video question-answering examples show concise salient explanations, fewer unsupported action details than competing models, and an instruction-following fairy tale with a moral.The section describes these examples as consistent with prior quantitative comparisons across multiple benchmarks.Source: §B.2 · p. 17

B.3. Ablation Studies

• Including motion tokens hardly affects image comprehension, supporting the reported compatibility of decoupled visual-motional tokenization with unified image, video, and text modeling.The impact is reported in Table 9.Source: §B.3 · p. 17
• Training the detokenizer from scratch on WebVid-10M without SVD image-to-video initialization has little impact on final text-to-video generation results.The detailed results are reported in Table 10.Source: §B.3 · p. 17

C. Limitations

• The model cannot generate very long videos with its 4096-token context window and current dataset; WebVid-10M averages about 15 seconds and has little scene change.These data characteristics lead to similar keyframes across clips; scaling training to web-scale video remains costly and may require better use of spatial and temporal redundancies.Source: C. Limitations · p. 17