Source-linked AI summary
Vlogger: Make Your Dream A Vlog
Shaobin Zhuang, Kunchang Li, Xinyuan Chen, Yaohui Wang, Ziwei Liu, Yu Qiao, Yali Wang
TL;DR
Vlogger asks how to generate long-form vlogs, which the paper argues require systematic planning and shooting rather than only a generative model. It combines an LLM director with scene-level planning and video snippets, and reports state-of-the-art zero-shot T2V generation and prediction alongside coherent vlogs longer than five minutes.
Problem
The paper asks how to generate vlogs automatically, arguing that long-form video requires systematic planning and shooting rather than only a generative model.
Source: §1 · pp. 1–2Method
Vlogger uses an LLM director to plan scenes and durations, then combines actor images and scene descriptions with ShowMaker-generated snippets to assemble a vlog.
Source: §1 · p. 2Results
Vlogger reports state-of-the-art performance on zero-shot T2V generation and prediction and generates over 5-minute vlogs without loss of script and actor coherence.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that Vlogger can generate long vlogs from open-world descriptions while preserving video coherence on script and actor.
Source: §5 · p. 9
Abstract
from arXiv · showhide
In this work, we present Vlogger, a generic AI system for generating a minute-level video blog (i.e., vlog) of user descriptions. Different from short videos with a few seconds, vlog often contains a complex storyline with diversified scenes, which is challenging for most existing video generation approaches. To break through this bottleneck, our Vlogger smartly leverages Large Language Model (LLM) as Director and decomposes a long video generation task of vlog into four key stages, where we invoke various foundation models to play the critical roles of vlog professionals, including (1) Script, (2) Actor, (3) ShowMaker, and (4) Voicer. With such a design of mimicking human beings, our Vlogger can generate vlogs through explainable cooperation of top-down planning and bottom-up shooting. Moreover, we introduce a novel video diffusion model, ShowMaker, which serves as a videographer in our Vlogger for generating the video snippet of each shooting scene. By incorporating Script and Actor attentively as textual and visual prompts, it can effectively enhance spatial-temporal coherence in the snippet. Besides, we design a concise mixed training paradigm for ShowMaker, boosting its capacity for both T2V generation and prediction. Finally, the extensive experiments show that our method achieves state-of-the-art performance on zero-shot T2V generation and prediction tasks. More importantly, Vlogger can generate over 5-minute vlogs from open-world descriptions, without loss of video coherence on script and actor. The code and model is all available at https://github.com/zhuangshaobin/Vlogger.
1. Introduction
Minute-level vlogs with complex narratives remain challenging because existing approaches either rely on extensive long-video training or produce incoherent shot changes. Vlogger uses an LLM-led, four-stage workflow and a scene-level video diffusion model to generate coherent vlogs from open-world stories.
1. Introduction
- Most current video diffusion approaches generate only a few seconds, while earlier long-video methods require extensive training data or exhibit incoherent shot changes, making minute-level vlogs challenging.The authors motivate systematic planning and shooting, rather than relying on a generative model alone.Source: §1 · pp. 1–2
- An LLM Director converts a user story into a scene-based script, designs actor reference images, guides ShowMaker to create a snippet per scene, and invokes Voicer to dub subtitles.The script specifies shooting scenes and their durations.Source: §1 · p. 2
- Top-down scene planning and actor design provide explicit guidance for individual snippets, which Vlogger integrates into a cohesive vlog without large-scale long-video training.The authors say this can reduce incoherence from abrupt shot changes.Source: §1 · p. 2
- ShowMaker uses scene descriptions and actor images as prompts to enhance script and actor coherence, while mixed T2V-generation and prediction training supports controllable-duration snippets.The LLM Director plans each scene’s duration.Source: §1 · p. 2
- Vlogger reports state-of-the-art performance on zero-shot T2V generation and prediction, and generates over 5-minute vlogs without loss of script and actor coherence.It also outperforms Phenaki [68] while using only 66.7% of the training videos.Source: §1 · p. 2
2. Related Works
Prior text-to-video approaches mainly produce short clips, while long-video methods face training-data or coherence challenges. Vlogger combines LLM-based planning with scene-level shooting; its planning process turns a story into a script and actor assignments.
2. Related Works
- Most text-to-video diffusion approaches generate short videos from simple descriptions, while parallel long-video generation requires extensive annotated data and autoregressive generation can suffer shot-change incoherence.The discussion frames these limitations as problematic for vlogs with complex narratives and multiple scenes.Source: §2 · p. 2
- Four rounds of LLM dialogue convert a user story into a final script, from which actor reference images and scene-specific actor assignments are determined.The figure describes this as top-down planning.Source: Figure 2 · p. 3
- Vlogger introduces an LLM Director to coordinate top-down planning and bottom-up shooting as an approach to training burden and content incoherence in earlier long-video methods.The cited discussion connects LLM reasoning to visual tasks [6, 26, 36, 40, 43, 80, 89, 91].Source: §2 · p. 3
3. Method
Vlogger plans a vlog with an LLM Director, then shoots and combines scene-level snippets. ShowMaker uses scene descriptions and actor images, with spatial-temporal attention and mixed generation/prediction training.
3.1. Overall Framework of Vlogger
- The LLM Director creates a script and actors, ShowMaker generates a snippet for each scene, and Voicer dubs it before the snippets are combined.Source: §3.1 · p. 3
3.1.1 Top-Down Planning
- The Director refines a user story through rough, detailed, completed, and scheduled script stages, then aligns actor reference images and a protagonist with each scene.Source: §3.1.1 · pp. 3–4
3.1.2 Bottom-Up Shooting
- ShowMaker uses scene descriptions and actor images as prompts, generates a first clip, then predicts further clips until the scene’s allocated duration is reached; Voicer adds speech before scenes are combined.Source: §3.1.2 · pp. 4–5
3.2. ShowMaker
- ShowMaker is a video diffusion model that reconstructs snippet latents through a denoising U-Net, with spatial-temporal enhanced blocks and mode selection as its two design changes.Source: §3.2 · p. 5
3.2.1 Spatial-Temporal Enhanced Block (STEB)
- STEB adds actor-image spatial cross-attention and scene-text temporal cross-attention to address actor and text incoherence in snippets.Source: §3.2.1 · pp. 5–6
3.2.2 Mixed Training Paradigm with Mode Selection
- Mode selection probabilistically chooses preserved clean frames: k=0 trains generation, while k>0 trains prediction of the remaining frames.Source: §3.2.2 · p. 6
- On Kinetics-400, Vlogger’s zero-shot FID is better than Phenaki’s while using only 66.7% pre-training videos.Source: Table 2 · p. 6
4. Experiments
Experiments report strong zero-shot video-generation and long-video results, then examine Vlogger’s generation process and ShowMaker components. Visualizations compare long-vlog generation and illustrate ablation effects.
4. Experiments
- The evaluation includes zero-shot UCF-101, Kinetics-400, and MSR-VTT benchmarks, plus Vimeo11k, a collected open-world-caption benchmark of 11,293 videos from 10 categories.The paper describes Vimeo11k as an evaluation benchmark for ablation studies.Source: §4 · p. 6
4.1. Comparison with state-of-the-art
- Vlogger surpasses TATS on 1,000-frame UCF-101 generation without UCF-101 finetuning, and its video quality does not continuously decline as frame count increases.The comparison uses FVD, for which lower values indicate better performance; no specific values are supplied here.Source: Figure 5 · p. 7, §4.1 · p. 7
- Zero-shot, Vlogger achieves the best UCF-101 FVD with either class labels or hand-crafted prompts, better Kinetics-400 FID than Phenaki using 66.7% of its training videos, and competitive MSR-VTT FID and CLIPSIM.Its generated videos are 320×512, compared with Phenaki’s 256×256 resolution.Source: §4.1 · p. 7
4.2. Ablation Study
- The generation-process comparison holds scripts and ShowMaker constant; the reported results favor Vlogger over no-autoregression and fully autoregressive alternatives for preferable vlogs.It evaluates CLIP-I, CLIP-T, and video duration across five scripts and 20 random seeds per script.Source: §4.2 · p. 7, Table 4 · p. 7
- Spatial image cross-attention improves CLIP-I as β increases, while CLIP-T peaks at β=0.5; temporal text cross-attention improves T2V generation and prediction on Vimeo11k.The mixed training paradigm is reported to enhance both generation and prediction capabilities.Source: §4.2 · pp. 7–8, Figure 6 · p. 7, Table 5 · p. 7
4.3. Visualization
- In qualitative comparison with Phenaki, Vlogger produces more diverse video content and illustrates the story over 7min59s according to its LLM-created script.Phenaki’s comparison is based on its official website demo because its code was not open-sourced.Source: §4.3 · p. 8, Figure 7 · p. 8
- ShowMaker’s visual ablation depicts improvements in generation and prediction, and visual prompts help distinguish the “Teddy” concept in text prompts.Figure 8 compares temporal text cross-attention and mixed training settings, including a model without a visual prompt.Source: §4.3 · p. 8, Figure 8 · p. 8
5. Conclusion
The conclusion presents Vlogger as a system for generating long vlogs from open-world descriptions, alongside ShowMaker for video generation and prediction.
5. Conclusion
- Vlogger generates over 5-minute vlogs from open-world descriptions without loss of script and actor coherence, while ShowMaker is presented for boosting state-of-the-art T2V generation and prediction.The authors state that models, data, and code will be released to support further designs for long-video generation in the open world.Source: §5 · p. 9
A. Top-Down Planning
The appendix presents prompt examples as guidance for producing structured outputs and qualitatively emphasizes progressive script generation.
A. Top-Down Planning
- Examples in Tables 6–9 guide users toward JSON-formatted LLM outputs and qualitatively illustrate the value of progressive script generation.Source: A. Top-Down Planning · p. 9
B. ShowMaker Implementation Details
ShowMaker implementation uses two training stages, with distinct parameter-training choices and a shared video-and-image training approach.
Stage-1 Settings
- Stage 1 expands the input channels from 4 to 9 and trains the U-Net except spatial image cross-attention components, using α=0.6, m=6, 54,000 steps, and batch size 384.Training takes approximately five days; the batch contains a 16-frame video and six images.Source: B. ShowMaker Implementation Details · p. 9
Stage-2 Settings
- Stage 2 freezes the U-Net and trains 22M spatial image cross-attention parameters, using β=1 and 62,000 steps with batch size 256.It trains only the video-generation pattern and takes approximately five days.Source: B. ShowMaker Implementation Details · p. 9
Shared training details
- Both stages use AdamW and jointly train on images and videos; WebVid10M videos are sampled at random six-frame intervals and data are resized to 320×520.The Stage 1 and Stage 2 settings both specify a learning rate of 1e-4.Source: B. ShowMaker Implementation Details · p. 9
C. Comparison with State-of-the-art
The passage describes dataset-specific sampling, resizing, prompting, and metric procedures used for video-generation evaluation, but gives no state-of-the-art comparison results.
C. Comparison with State-of-the-art
- For UCF-101, 100 videos per class or hand-crafted prompt are generated at 320×512, resized to 240×320, and used for FVD computation.The sampling strategy follows [77], and FVD uses code derived from [59].Source: C. Comparison with State-of-the-art · p. 9
- For Kinetics-400 T2V evaluation, GPT-4 creates a hand-crafted prompt per class, and ground-truth videos are resized to 320×512.This matches the resolution of generated videos.Source: C. Comparison with State-of-the-art · p. 9
- For MSR-VTT, one randomly selected description generates each of 2,990 test videos, with 16 frames at 320×512; CLIPSIM and FID are computed over 47,840 frames.The metrics follow [57].Source: C. Comparison with State-of-the-art · p. 9
- For UCF-101 1000 Frames, generated long videos are split into 16-frame clips, and FVD is computed against ground truth.The prompts follow [25], aligning with TATS.Source: C. Comparison with State-of-the-art · p. 9
D.1. Vlog Generation Process
GPT-4-generated scripts are processed through Vlogger’s planning process; the passage also notes that two comparison systems cannot use visual prompts.
D.1. Vlog Generation Process
- GPT-4 automatically generates five scripts—Teddy Travel, Alfie Work, Tiger Moon, Pete Skiing, and Anna Cooking—for Vlogger’s planning process.The same planning process is used for all five scripts.Source: §D.1 · p. 9
- MovieFactory and Phenaki use only textual prompts because their framework designs do not support visual prompts.Both systems receive the same Script as Vlogger.Source: §D.1 · p. 9
- Table 6 marks the Rough Script with a spiro disco ball and the Story with orange.The table describes Vlogger’s rough-script planning process.Source: Table 6 · p. 10
D.2. Temporal Text Cross Attention (TTCA) and Mixed Training Paradigm (MTP)
The passage reports Vimeo11K dataset composition and points readers to supplementary visualizations of Vlogger’s script-planning outputs.
D.2. Temporal Text Cross Attention (TTCA) and Mixed Training Paradigm (MTP)
- Vimeo11K contains 11,293 videos across ten categories, with category counts ranging from 905 Sports videos to 1,288 Travel videos.The dataset is captioned using VideoChat, and test-content data and code are to be released afterward.Source: §D.2 · p. 10
E. Visualization
- Vlogger’s visualization results are placed in the supplementary material and linked from the Project Page.Source: E. Visualization · p. 10
- Table 7 marks Detail Script in cyan, Rough Script with a spiro disco ball, and Story in orange.The table presents Vlogger’s detail-script planning process.Source: Table 7 · p. 11