PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Long-form Video Prior via Generative Pre-Training

Jinheng Xie, Jiajun Feng, Zhaoxu Tian, Kevin Qinghong Lin, Yawen Huang, Xi Xia, Nanxu Gong, Xu Zuo, Jiaqi Yang, Yefeng Zheng, Mike Zheng Shou

arXiv:2404.15909v1cs.CV

TL;DR

Long-form videos contain complex people, objects, and interactions that remain challenging to learn over extended time and varied frames. This paper tokenizes visual locations and text in movie storyboards, using Storyboard20K to train a generative pre-trained model. The experiments report potential for learning long-form video prior, while the dataset release is limited to annotations because of movie-copyright concerns.

  • Problem

    Long-form video prior is difficult to learn because videos involve complex people, objects, and interactions across extended temporal coverage, frame variation, and consistent characters.

    Source: §1 · pp. 1–2

  • Method

    The paper tokenizes bounding boxes, keypoints, and textual descriptions from movie storyboards and learns their sequences with generative pre-training on Storyboard20K.

    Source: §1 · pp. 2–3, §6 · p. 15

  • Results

    Experiments indicate that long-form video prior can be learned via generative pre-training, and the models outperform GPT-3.5 on Rouge-L and BertScore.

    Source: §5.2 · p. 12, §6 · p. 15

  • Takeaways & Limitations

    The results suggest potential for modeling long-form video prior for vision generation tasks.

    Source: §1 · p. 3

    Only annotations will be released to avoid potential copyright concerns involving movie content.

    Source: §3 · p. 8

Abstract

from arXiv · show

Concepts involved in long-form videos such as people, objects, and their interactions, can be viewed as following an implicit prior. They are notably complex and continue to pose challenges to be comprehensively learned. In recent years, generative pre-training (GPT) has exhibited versatile capacities in modeling any kind of text content even visual locations. Can this manner work for learning long-form video prior? Instead of operating on pixel space, it is efficient to employ visual locations like bounding boxes and keypoints to represent key information in videos, which can be simply discretized and then tokenized for consumption by GPT. Due to the scarcity of suitable data, we create a new dataset called \textbf{Storyboard20K} from movies to serve as a representative. It includes synopses, shot-by-shot keyframes, and fine-grained annotations of film sets and characters with consistent IDs, bounding boxes, and whole body keypoints. In this way, long-form videos can be represented by a set of tokens and be learned via generative pre-training. Experimental results validate that our approach has great potential for learning long-form video prior. Code and data will be released at \url{https://github.com/showlab/Long-form-Video-Prior}.

1 Introduction

Long-form video concepts form a complex implicit prior that remains difficult to learn, particularly across extended temporal coverage, frame variation, and interactions among consistent characters. The paper explores generative pre-training over tokenized visual locations and text, supported by the Storyboard20K dataset.

1 Introduction

  • Long-form video prior is difficult to learn because it involves complex people, objects, and interactions over time.Short-video generation methods face challenges with extended temporal coverage, frame variation, and interactions among consistent characters.Source: §1 · pp. 1–2
  • The approach represents long-form videos as token sequences and learns them with generative pre-training alongside textual descriptions.Continuous visual locations are discretized and tokenized; instruction tuning can support sampling videos from user instructions.Source: §1 · pp. 2–3
  • Storyboard20K provides movie storyboards with synopses, keyframes, and fine-grained annotations for film sets and characters.Character annotations include consistent IDs, bounding boxes, and whole-body keypoints across shots.Source: Figure 1 · p. 2, §1 · p. 3
  • The paper frames long-form video concepts as an implicit prior and investigates whether generative pre-training can learn it.The motivation draws on GPT's demonstrated capacity for text and visual-location modeling.Source: §1 · pp. 1–3
  • Experiments on Storyboard20K indicate that generative pre-training has potential to model long-form video prior for vision generation tasks.The paper represents videos as token sequences rather than modeling pixel space.Source: §1 · p. 3

2 Related Works

Prior work spans short-form video generation, movie understanding and storyboard datasets, and language modeling. This paper builds on those areas to investigate generative pre-training for long-form video prior.

2.1 Video Generation

  • Video-generation approaches cover unconditional generation, text-to-video, and text-guided editing, but largely target short videos and still face temporal-consistency challenges.The related-work discussion notes that increased interaction complexity and view changes make these methods less suited to long-form scenarios.Source: §2.1 · pp. 3–4

2.2 Learning from Movies

  • Movie datasets and methods support movie understanding and storyboard creation, while Storyboard20K adds fine-grained annotations for long-form video-prior learning.LSMDC includes clips with transcribed audio descriptions, MovieNet provides multimodal movie data, and prior storyboard methods retrieve relevant frames.Source: §2.2 · p. 4

2.3 Language Modeling

  • Language-modeling research includes masked bidirectional pre-training and GPT's sequential next-token prediction, which this work applies to movie storyboards.GPT and its successors, including GPT-2 and GPT-3, are described as generative pre-training models.Source: §2.3 · pp. 4–5

3 Storyboard20K

Storyboard20K is a movie-storyboard dataset designed to represent narrative and visual information across multiple shots. Its annotations cover characters, film sets, and summative descriptions, with scale and coverage characterized through dataset analysis.

3 Storyboard20K

  • Storyboard20K contains 20,310 movie storyboards and approximately 150K shots from around 400 movies; movie clips typically average 33.9 seconds, providing long-form time coverage.Storyboards were condensed into representative shots, with algorithms and manual screening used to preserve story coherence and remove outliers.Source: §3 · pp. 5, 7
  • Character annotations pair bounding boxes and whole-body keypoints with consistent IDs across shots, using automatic tagging followed by manual calibration.The annotations target character blocking, actions, and facial expressions that textual descriptions alone cannot fully represent.Source: §3 · pp. 6–7, Figure 2 · p. 6
  • Film-set annotations add bounding boxes for objects related to the storyboard, including objects inferred from descriptions or associated scenes.The examples include a railway associated with a train even when the railway is not mentioned in the script.Source: Figure 2 · p. 6, §3 · p. 7
  • Each storyboard includes summative information such as title, genre, emotion, scene, and a rewritten short summary.These annotations can be used for classification or as human instructions for fine-tuning.Source: Figure 2 · p. 6, §3 · p. 7
  • The dataset has around 500K bounding boxes: 319K for characters and 180K for film sets, spanning around 6K film-set categories.The analysis reports about 24.6 boxes per storyboard and describes coverage of common genres, emotions, and everyday objects.Source: §3 · pp. 7–8, Figure 3 · p. 8
  • Only Storyboard20K annotations will be released; source movie clips and frames are available through the MovieNet and LSMDC websites.The release boundary is stated as a measure to avoid potential movie-copyright concerns.Source: §3 · p. 8

4 Method

The method converts movie storyboards into structured token sequences and learns their prior with a transformer decoder. It then samples storyboards unconditionally or from textual descriptions and instructions.

4.1 Problem Definitions

  • The method approximates the storyboard token prior by learning a neural network through likelihood maximization, from which novel outputs can be sampled.The modeled tokens encode bounding boxes, keypoints, and text rather than pixel information.Source: §4.1 · p. 9, §4 · p. 9

4.2 Prompt Designs for Long-form Videos

  • Continuous visual locations are quantized into discrete words and organized with synopses, per-shot objects, and main-character information in a JSON prompt.Character keypoint detail is adjusted by character size to reduce token use and emphasize main characters.Source: §4.2 · pp. 9–10

4.3 Modeling Prior via Generative Pre-Training

  • A GPT-2 base transformer decoder learns the storyboard sequences with a standard next-token likelihood objective.Training uses a context window and stochastic gradient descent.Source: §4.3 · p. 10

4.4 Sampling from long-form video prior

  • The learned prior supports unconditional sampling and conditional sampling from synopses or instructions.Instruction-storyboard pairs based on summative annotations support more free-form instruction-based sampling.Source: §4.4 · p. 10

5 Experiments

Experiments evaluate Storyboard20K-based learning with textual and visual-layout metrics, decoding success, and storyboard samples. The results report advantages over GPT-3.5 and show script-aligned shot variation; the paper also describes potential applications.

5.1 Experimental Setup

  • The setup divides Storyboard20K into train, testA with condensed descriptions, and testB with shot-by-shot descriptions.Evaluation uses Rouge-L, BertScore, and Perplexity for generated sequences and FID for visual layouts.Source: §5.1 · p. 11

5.2 Quantitative Results

  • Our models outperform GPT-3.5 on the textual metrics Rouge-L and BertScore.GPT-3.5 could not be fine-tuned on Storyboard20K because the model was unavailable.Source: §5.2 · p. 12
  • Our models outperform GPT-3.5-turbo on visual-layout FID for both test sets; GPT-3.5-turbo scores 23.08 on testA and 35.14 on testB.Pre-training GPT-2 on the large-scale VisorGPT dataset also significantly improves over training only on Storyboard20K.Source: §5.2 · p. 12, Table 3 · p. 12
  • The proposed model achieves a 92.5% decoding success rate, compared with 53.5% for GPT-3.5.The comparison counts valid sequences decoded into storyboards across 1,000 samplings.Source: §5.2 · pp. 12–13, Table 4 · p. 12

5.3 Visualization Results

  • The proposed model generates script-aligned storyboards with substantial shot-to-shot view changes, unlike the described GPT-3.5 and GPT-4 examples.Examples follow narrative sequences across shots, including a boy holding a knife, blood on the floor, and the boy staring at his hands.Source: Figure 5 · p. 13, §5.3 · p. 14

5.4 Applications

  • Storyboards are proposed as spatiotemporal guidance for controllable image/video diffusion models and as blocking guidance for amateur filmmakers.The paper illustrates the diffusion-guidance application with synthesized pixel frames.Source: §5.4 · p. 14, Figure 7 · p. 14

6 Conclusion

The conclusion presents generative pre-training over tokenized scripts and visual locations as a feasible approach to learning long-form video prior, supported by the Storyboard20K dataset.

6 Conclusion

  • Storyboard20K provides scripts, shot-by-shot keyframes, and fine-grained annotations of film sets and characters with consistent IDs, boxes, and whole-body keypoints.The approach discretizes and tokenizes scripts and visual elements, then learns their sequences through generative pre-training by maximizing sequence likelihood.Source: §6 · p. 15

1 Additional Data Samples

The additional samples illustrate the dataset's temporal and visual variety, including changing views, scenes, and characters, with related film sets annotated.

1 Additional Data Samples

  • Storyboard samples show extensive temporal coverage and view changes, with film-set annotations such as speedboat, freighter, magazine, and notebook.The illustrated “loading drugs” example includes diverse views, scenes, and characters.Source: §1 · p. 18
  • Figure 2 lists the indices of sampled keypoints.Source: Figure 2 · p. 18

2 More implementation Details

The implementation samples body keypoints selectively based on character bounding-box area and uses a discretized 512-by-512 coordinate representation.

2 More implementation Details

  • Characters with bounding-box areas from 322 to 962 use 17 keypoints; characters outside that range use only the bounding box.The method samples 93 keypoints rather than all 136 whole-body keypoints.Source: §2 · p. 19

3 Impact of Model Sizes

The section reports a consistent trend of improved performance as GPT model size increases. Figure 3 presents prompts for GPT-3.5 and GPT-4.

3 Impact of Model Sizes

  • Performance improves consistently as GPT model size scales up.The section describes this as the impact of increasing GPT scale.Source: Table 5 · p. 19
  • Figure 3 presents prompts for GPT-3.5 and GPT-4.Source: Figure 3 · p. 19

Related papers

Loading 2404.15909v1…